It is not possible to do security releases that require coordinated changes across multiple components
Some security issues require coordinated changes across multiple components across our stack. A prime example of this is the case where we need to amend existing Protobuf definitions e.g. in Gitaly and make the client use them in order to fix a vulnerability. With the current workflow wwe have no way of doing this without leaking details to the public before the security release happens.
Usually, the workflow is the following in this case:
- We add a new optional field to Gitaly's Protobuf definitions and implement its logic. The default behaviour without this field being set is the same as before so that there is no client-visible change.
- When the changes get merged we release a new Gem for the updated Protobuf definitions.
- We update the Gem version in Rails to pull in the changed Protobuf definitions.
- We implement the Rails-side logic to populate the field as required.
During a security release we cannot release the Gem though and thus have to stop after step 1 already. So ultimately, we have no other option than to publish the mitigation code on the Gitaly-side without actually mitigating the vulnerability yet because the client doesn't know how to do that.
- We cannot publish the Protobuf Gem in the first place because the definitions are likely to reveal important information to an adversary.
- We cannot create the Protobuf Gem because the changes have not yet been merged to the target branch, and the MR might still change e.g. to merge conflicts. So ultimately we may end up releasing a Gem with code that never made it into our codebase.