Commit 0fac0295 authored by Janek Kozicki's avatar Janek Kozicki Committed by Bruno Chareyre

Update documentation about simpler writing of yade --checks

parent 53e835b6
......@@ -336,6 +336,8 @@ Note that the cmake ``PYTHON_VERSION`` option can be set to force any python ver
Also see notes about :ref:`converting python 2 scripts into python 3<convert-python2-to3>`.
.. _speed-up:
Speed-up compilation
^^^^^^^^^^^^^^^^^^^^^
......
......@@ -164,20 +164,29 @@ The purpose of unit testing is to make sure that the behaviour of the most impor
To add a new test, the following steps must be performed:
#. Place a new file such as :ysrc:`py/tests/dummyTest.py` using instructions in `python unittest documentation <https://docs.python.org/3/library/unittest.html>`__,
#. Add the file name such as ``dummyTest`` to the :ysrc:`py/tests/__init__.py` file,
#. Place a new file such as :ysrc:`py/tests/dummyTest.py`.
#. Add the file name such as ``dummyTest`` to the :ysrc:`py/tests/__init__.py` file.
#. If necessary modify the ``import`` and ``allModules`` lines in :ysrc:`py/tests/__init__.py`.
#. According to instructions in `python unittest documentation <https://docs.python.org/3/library/unittest.html>`__ use commands such as ``self.assertTrue(…)``, ``self.assertFalse(…)`` or ``self.assertRaises(…,…)`` to report possible errors.
.. note:: It is important that all variables used in the test are stored inside the class (using the ``self.`` accessor), and that all preparations are done inside the function ``setUp()``.
Check tests
-----------
:ysrc:`Check tests <scripts/checks-and-tests/checks>` (also see :ysrc:`README<scripts/checks-and-tests/checks/README>`) perform comparisons of simulation results between different versions of yade, as discussed `here <http://www.mail-archive.com/[email protected]/msg05784.html>`__. They differ with regression tests in the sense that they simulate more complex situations and combinations of different engines, and usually don't have a mathematical proof (though there is no restriction on the latest). They compare the values obtained in version N with values obtained in a previous version or any other "expected" results. The reference values must be hardcoded in the script itself or in data files provided with the script. Check tests are based on regular yade scripts, so that users can easily commit their own scripts to trunk in order to get some automatized testing after commits from other developers.
:ysrc:`Check tests <scripts/checks-and-tests/checks>` (also see :ysrc:`README<scripts/checks-and-tests/checks/README.md>`) perform comparisons of simulation results between different versions of yade, as discussed `here <http://www.mail-archive.com/[email protected]/msg05784.html>`__. They differ with regression tests in the sense that they simulate more complex situations and combinations of different engines, and usually don't have a mathematical proof (though there is no restriction on the latest). They compare the values obtained in version N with values obtained in a previous version or any other "expected" results. The reference values must be hardcoded in the script itself or in data files provided with the script. Check tests are based on regular yade scripts, so that users can easily commit their own scripts to trunk in order to get some automatized testing after commits from other developers.
Since the check tests history will be mostly based on standard output generated by ``yade --check``, a meaningfull checkTest should include some "print" command telling if something went wrong. If the script itself fails for some reason and can't generate an output, the log will contain "scriptName failure". If the script defines differences on obtained and awaited data, it should print some useful information about the problem and increase the value of global variable resultStatus. After this occurs, the automatic test will stop the execution with error message.
When check fails the script should return an error message via python command ``raise YadeCheckError(messageString)`` telling what went wrong. If the script itself fails for some reason and can't generate an output, the log will contain only "scriptName failure". If the script defines differences on obtained and awaited data, it should print some useful information about the problem. After this occurs, the automatic test will stop the execution with error message.
An example dummy check test :ysrc:`scripts/checks-and-tests/checks/checkTestDummy.py` demonstrates a minimal empty test. A little more functional example check test can be found in :ysrc:`scripts/checks-and-tests/checks/checkTestTriax.py`. It shows results comparison, output, and how to define the path to data files using "checksPath".
Users are encouraged to add their own scripts into the :ysrc:`scripts/checks-and-tests/checks/` folder. Discussion of some specific checktests design in `questions and answers <https://answers.launchpad.net/yade/>`__ is welcome. Note that re-compiling is required before that added scripts can be launched by ``yade --check`` (or direct changes have to be performed in "lib" subfolders).
An example dummy check test :ysrc:`scripts/checks-and-tests/checks/checkTestDummy.py` demonstrates a minimal empty test. A little more functional example check test can be found in :ysrc:`scripts/checks-and-tests/checks/checkTestTriax.py`. It shows results comparison, output, and how to define the path to data files using ``checksPath``.
Users are encouraged to add their own scripts into the :ysrc:`scripts/checks-and-tests/checks/` folder. Discussion of some specific checktests design in `questions and answers <https://answers.launchpad.net/yade/>`__ is welcome. Note that :ref:`re-compiling <speed-up>` is required before the newly added scripts can be launched by ``yade --check`` (or direct changes have to be performed in "lib" subfolders).
A check test should never need more than a few seconds to run. If your typical script needs more, try to reduce the number of elements or the number of steps.
To add a new check, the following steps must be performed:
1. Place a new file such as :ysrc:`scripts/checks-and-tests/checks/checkTestDummy.py`,
2. Inside the new script use ``checksPath`` when it is necessary to load some data file, like :ysrc:`scripts/checks-and-tests/checks/data/100spheres`
3. When error occurs raise exception with command ``raise YadeCheckError(messageString)``
Conventions
============
......
=== Checktesting ===
Checktesting
============
1. Check tests perform comparisons of simulation results between different versions of yade, as discussed in http://www.mail-archive.com/[email protected]/msg05784.html and the whole thread. They differ with regression tests in the sense that they simulate more complex situations and combinations of different engines, and usually don't have a mathematical proof (though there is no restriction on the latest).
......@@ -16,4 +18,5 @@
8. A check test should never need more than a few seconds to run. If your typical script needs more, try and reduce the number of element or the number of steps.
9. Failures are reported via a global variable "resultStatus", which should be ONLY INCREMENTED in a checkTests, never assigned to (resultStatus+=1 is ok, resultStatus=1 is bad), else the script would erase the history of the checkTests coming before it.
9. Failures are reported via exception using python command: raise YadeCheckError(stringMessage)
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment