Skip to content

Non deterministic failure of grouping test step and result into correct test action pair.

Class TestActionDirective generates action_id, which identifies test step and result pair of test action, as int(time.time() * 1000) which is not good enough: sometimes test actions could be missed or grouped together in wrong way.

Version

0.1.0

Steps to Reproduce

Non deterministic, try to run unit tests until it fails eventually:

while tox --sitepackages -e py35 -- -vv; do if [[ $? != 0 ]]; then break; fi; done

Example of failures caused by this issue:

>       assert _publish_html(rst_input_full_example, use_plain=False) == exp_result
E       assert '<div class="...le>\n</div>\n' == '<div class="d...le>\n</div>\n'
E           <div class="document">
E           <table border="1" class="docutils">
E           <colgroup>
E           <col width="2%" />
E           <col width="49%" />
E           <col width="49%" />
E           </colgroup>
E           <thead valign="bottom">
E           <tr><th class="head">&nbsp;</th>
E           <th class="head">Step</th>
E           <th class="head">Expected Result</th>
E           </tr>
E           </thead>
E           <tbody valign="top">
E           <tr><td>1</td>
E           <td>List files in the volume: <tt class="docutils literal">ls <span class="pre">-a</span> /mnt/helloworld</tt></td>
E           <td>There are no files, output should be empty.</td>
E           </tr>
E           <tr><td>2</td>
E           <td><p class="first">Donec et mollis dolor:</p>
E           <pre class="last literal-block">
E           $ foo --extra sth
E           $ bar -vvv
E           </pre>
E           </td>
E           <td>Maecenas congue ligula ac quam viverra nec
E           consectetur ante hendrerit.</td>
E           </tr>
E           <tr><td>3</td>
E           <td>This one has no matching test result.</td>
E         + <td>&nbsp;</td>
E         + </tr>
E         + <tr><td>4</td>
E         + <td>&nbsp;</td>
E           <td>And this result has no test step.</td>
E           </tr>
E           </tbody>
E           </table>
E           </div>

or:

>       assert _publish_html(rst_input, use_plain=True) == exp_result
E       assert '<div class="...iv>\n</div>\n' == '<div class="d...iv>\n</div>\n'
E           <div class="document">
E           <div action_id="1" action_name="test_step" class="pylatest_action">
E           Ceterum censeo Carthaginem esse delendam.
E           </div>
E           <div action_id="2" action_name="test_result" class="pylatest_action">
E           This city is no more ... it has ceased to be ...
E           </div>
E         - <div action_id="2" action_name="test_step" class="pylatest_action">
E         ?                 ^
E         + <div action_id="3" action_name="test_step" class="pylatest_action">
E         ?                 ^
E           Step foo.
E           </div>
E           <div action_id="3" action_name="test_result" class="pylatest_action">
E           Result bar.
E           </div>
E           </div>
Edited by mbukatov