CIP-2003 enhance system tests to handle multiple delay models
marking as draft since the speedrun test is also failing at the last two histograms - looking into this
Note: the delay_models_formatter and the delay_models_publisher are temporarily included in this MR because they have some changes that are not in the latest ec version (there is a separate MR on the ec side), so they do not be to be reviewed as part of this MR
Description
- delay models publisher (from CIP-1620) called asynchronously, so that the test can continue with other tasks while delay models are being published
- new system test added for multiple delay models
-
TODO: switch the talon used to talon3 (temporarily using whichever one is available for testing)
-
- delay models used for testing updated as per design in SP-4061, including:
- delay_model_package.json converted to delay_model_config.json (system-tests and internal-schemas)
- delay models moved from delay_model_package.json into standalone files (system-tests and internal-schemas)
- system-tests will need to save the test parameters. The modified delay models should be saved, instead of just the files prior to modifications.
-
TODO: filter out the unused receptors on the system-tests side (this happens on the ec side and is not saved to a file)- sample delay models package file saved to test_parameters
-
- Visibilities checkpoint criteria pass (zip_file_exists, fraction_of_data, number_of_timestamps (depends on
CIP-2479), number_of_expected_baselines)
Testing
most recent signal chain test results:
- last two histograms failed
- last two histograms failed
- speedrun1 passed
- speedrun3 passed
- speedrun3 wideband state count check failed, everything else passed
- speedrun3 wb state count check failed, everything else passed
Test_14 (signal chain verification with multiple delay models) results:
- last two histograms failed
- Test_14 all checkpoints and visibilities passed
- Test_14 all checkpoints and visibilities passed
- Test_14 after first round of code review comments - all checkpoints and visibilities passed
- Test_14 after updating branch with latest changes, including MAP-59 and MAP-60 changes - all checkpoints and visibilities passed
- Test_14 after re-ordering the lstv replay step - all checkpoints except wideband state count passed
- Test_14 after updating branch with latest changes, including logger changes and fix for CIP-2506 - all checkpoints and visibilities passed
- Test_14 with latest changes from main and code review updates (and ec hash) - all checkpoints and visibilities passed
other related tests (write delay models to the TMC emulator):
-
two configure scans in succession - the second fraction of data checkpoint failed- only tests 1-7 and 11 are run as part of the nightly
- two scans - the second one failed (same result as yesterday's nightly test. could this be related to CIP-2377 or CIP-2565?)
other tests:
TODO: update changelog, update branch with latest main, verify other tests pass off test and signal chain tests pass. two scans does not pass (has 2 bug tracking this on JIRA)
Edited by Edna Sanchez