Outdated observer updates
Workers and observers could potentially modify the status of a resource at the same time which would lead to an invalid reconciliation.
Problem
The following figure shows the problem:
The user updates a resource (specification changed from version A to B). This triggers the worker of a controller. The worker detects a mismatch between the real world state (status is A) and the desired state (spec is B) and will perform actions to reconcile both states.
At this point in time, the observer could still be running. If a status change is detected, an update is sent to the API with the new detected state. But the status update is for a specification of version A which is already outdated. Update would trigger the worker with an invalid reconciliation target: the specification is updated version B and the status is C but referring to specification version A.
Solution
- Observers are coroutines controlled by workers.
- Observers are terminated when a worker starts modifying a resource and recreated afterwards with a state given by the worker.
- The API rejects outdated status updates.
Update (on 19.11.19)
The solution 3 is discarded, as it would break quite a few mechanisms. For instance, the Kubernetes Controller gets an Application, does some work, then update it. But if any update is sent in the meantime, the update will be refused. This is absolutely not optimal.
In the end, the solution 2 has been chosen, see #278 (closed).