I disabled EULA seeds because I actually have no need for it since it's
so easy to generate them in testing (and I do not need to preload one to
bootstrap the application).
I added the DatabaseCleaner gem to combat problems with how Rspec uses
a single transaction for an entire test, and configured it to clean and
seed the database properly, and allow for changing to truncation when
needed.
Basically:
In my tests for EULA searching, I test bad regex. This is regex which
is valid Ruby regex but invalid Postgres regex, causing a Postgres
exception.
Normally (i.e. in dev AND prod) this is not a problem - Rails will
automatically rollback a failed transaction, and I rescue the error
generated.
But tests combine all transactions into one database. When you trigger a
Postgres exception, it poisons all future transactions until a rollback
occurs (per https://stackoverflow.com/a/31146267). However, tests ignore
any attempt to rollback the transaction because doing so would destroy
the ability of the rest of the test to operate. So that effectively
makes it impossible for the test to pass.
The reason for this discrepancy is speed - using a transaction means all
of the statements will be correct with respect to the others in the
transaction, meaning it is consistent and you can easily clean up
everything at the end by cancelling the transaction. I'm not sure what
isolation level this requires off the top of my head, but it could
probably use a less strict isolation level since I think PG guarantees
relative consistency within all isolation levels.
This also had the side effect of effectively resetting auto-increment
counters at the end of each test (though not really, it just never
committed the transaction so it didn't update auto-increment counters to
begin with), which is something that had puzzled me for some time.
Resetting auto-increment counters without transactions takes more time,
so I can see why the Rails developers went this route - it is fast, it
never actually modifies the database, and thus keeps it in a completely
predictable state.
Unfortuantely, due to the limitations which were made all too obvious
today, I had to install the DatabaseCleaner gem and set it up in the
RailsHelper.
DatabaseCleaner allows me to switch to truncation, so I use transactions
as a default and just use truncation whereever I will be causing
Postgres exceptions.
I believe that DatabaseCleaner can be a bit slow in CI environments
(https://stackoverflow.com/a/11423886), but the deletion method does not
allow you to reset auto-increment counters, and I will accept the
slight slowdown if it means a more predictable database state in my test
runs.
I will put all this into #87 as a comment but I felt like writing this
wall of text up here for posterity as well.