You need to sign in or sign up before continuing.
Non deterministic failure of grouping test step and result into correct test action pair.
Class TestActionDirective
generates action_id
, which identifies test step and result pair of test action, as int(time.time() * 1000)
which is not good enough: sometimes test actions could be missed or grouped together in wrong way.
Version
0.1.0
Steps to Reproduce
Non deterministic, try to run unit tests until it fails eventually:
while tox --sitepackages -e py35 -- -vv; do if [[ $? != 0 ]]; then break; fi; done
Example of failures caused by this issue:
> assert _publish_html(rst_input_full_example, use_plain=False) == exp_result
E assert '<div class="...le>\n</div>\n' == '<div class="d...le>\n</div>\n'
E <div class="document">
E <table border="1" class="docutils">
E <colgroup>
E <col width="2%" />
E <col width="49%" />
E <col width="49%" />
E </colgroup>
E <thead valign="bottom">
E <tr><th class="head"> </th>
E <th class="head">Step</th>
E <th class="head">Expected Result</th>
E </tr>
E </thead>
E <tbody valign="top">
E <tr><td>1</td>
E <td>List files in the volume: <tt class="docutils literal">ls <span class="pre">-a</span> /mnt/helloworld</tt></td>
E <td>There are no files, output should be empty.</td>
E </tr>
E <tr><td>2</td>
E <td><p class="first">Donec et mollis dolor:</p>
E <pre class="last literal-block">
E $ foo --extra sth
E $ bar -vvv
E </pre>
E </td>
E <td>Maecenas congue ligula ac quam viverra nec
E consectetur ante hendrerit.</td>
E </tr>
E <tr><td>3</td>
E <td>This one has no matching test result.</td>
E + <td> </td>
E + </tr>
E + <tr><td>4</td>
E + <td> </td>
E <td>And this result has no test step.</td>
E </tr>
E </tbody>
E </table>
E </div>
or:
> assert _publish_html(rst_input, use_plain=True) == exp_result
E assert '<div class="...iv>\n</div>\n' == '<div class="d...iv>\n</div>\n'
E <div class="document">
E <div action_id="1" action_name="test_step" class="pylatest_action">
E Ceterum censeo Carthaginem esse delendam.
E </div>
E <div action_id="2" action_name="test_result" class="pylatest_action">
E This city is no more ... it has ceased to be ...
E </div>
E - <div action_id="2" action_name="test_step" class="pylatest_action">
E ? ^
E + <div action_id="3" action_name="test_step" class="pylatest_action">
E ? ^
E Step foo.
E </div>
E <div action_id="3" action_name="test_result" class="pylatest_action">
E Result bar.
E </div>
E </div>
Edited by mbukatov