CI sometimes fails in a flaky way

This makes it hard to trust it. My hunch is that it has something to do with filesystem timestamp resoution under Docker, but that's just a hunch.

Failed: https://gitlab.com/subplot/subplot/-/jobs/1409106580

Worked: https://gitlab.com/subplot/subplot/-/jobs/1409141144

No difference in code. I just pressed "retry" after the failure.

changed the description

mentioned in merge request !187 (merged)

changed milestone to %Iteration ending 2021-07-31

assigned to @larswirzenius

Looking at the log of the failed build, there's a two-second time delta on the metadata of the file. There should be none, so something is certainly amiss.

Also, I note that the file timestamps are whole seconds, so the file system used inside Docker is suspicious.

Docker uses the overlay file system internally. This seems to have issues with timestamp resolution.

It's possible we can't rely on mtime comparison, when deciding whether to generate docgen output, at least not under Docker.

Possible solutions for this dilemma:

• Don't run tests that need filesystem mtimes under Docker.
• Don't use mtime, and use something like checksum of inputs instead. This would require storing the checksums somewhere between runs. Not sure where that would be.
• Always generate new output, drop the "don't genereate output unless needed" functionality.

The test (in the code) in question which is likely causing the pain is here:

https://gitlab.com/subplot/subplot/-/blob/main/src/bin/subplot.rs#L341

We did it that way (source >= output) in order to cope with low precision timestamps and rapid scripted input changes.

If we insert a sleep before the running of docgen the first time, that'll ensure the timestamp of the output document is larger than the input timestamps, which might make things more reliable perhaps?

I do not to like inserting sleeps in test suites, but it may be necessary.

mentioned in merge request !194 (merged)

changed milestone to %Iteration ending 2021-08-14

mentioned in commit 31170280

closed with merge request !194 (merged)