There is no way to disable synchronous deposits for deployments
Background
With normal processing, we have two servers, svc1a and svc1b, that accept deposits. These are stored locally until ready to be submitted to the queue.
During deployments, the queue is disable, along with the submitting of new deposits to the queue.
Once the queue is operational again, each of the svc1 machines will then read and submit the deposits to the database and queue.
With synchronous deposits, all deposit processing happens immediately, and reference processing is deferred by sending those to the submissions queue.
It is likely that the initial processing of a deposit will not be interrupted by a shutdown, and the generation of reference deposit will also happen quickly enough to go through, there is a window of data loss.
Observed behavior
synchronous deposit processing happens whether queue is down or update notifications are down.
Expected behavior
synchronous deposits should have a failsafe. A shutdown of the synchronous deposit controllers should disable uploads to the machine first, and a small delay (a second or two) should occur before the machine is shut down.
There is also the concern of database failure, where normal processing stops accepting deposits but the synchronous deposits keep going.
How urgent
Not very, more like technical debt. comment: # (There are myriad factors that go into prioritizing and scheduling development work, but any information you can provide to help us understand severity, urgency, relative priority, or deadlines, is much appreciated.)
Definition of ready
-
Product owner: @SaraBowman -
Tech lead: @myalter -
Service:: label applied -
Definition of done updated -
Acceptance testing plan: -
Weight applied
Definition of done
-
Unit tests identified, implemented, and passing -
Code reviewed -
Available for acceptance testing via a staging URL, or otherwise -
Consider any impacts to current or future architecture/infrastructure, and update specifications and documentation as needed -
Knowledge base reviewed and updated -
Public documentation reviewed and updated -
Acceptance criteria met -
Deployment scripts should disable deposits to synchronous deposit servers before shutting down the server.
-
-
Acceptance testing passed -
Deployed to production