Smooth RPC API evolution
This project was devised by @Saroupille, @romain.nl, @yrg, @sir4ur0n, and @smelc. François and Romain had the ideas presented in this document in the back of their minds for a long time.
Problem
- Changes to the RPC API have no official workflow.
- Reviewers don't have a resource to know what a breaking change is and how it should be mitigated.
Offering a smooth evolution of the RPC API is important for clients.
Solution
Definitions and choices
There is fundational work to perform. We need to define what is a breaking change:
- Removing a RPC is a breaking change
- What about adding a field to an RPC output? In most serialization library this is fine (parsers ignore fields they don't know), but we believe data-encoding is strict in this regard and will fail. We need to clarify that and take a decison (do we stick to the current behavior or change data-encoding?)
- Adding a RPC is fine
- Handle other cases, in particular detailing the impact of changing inputs and/or ouputs:
- Adding a field to the input of an RPC is a breaking change, unless the field is optional
- Adding a field to the output of an RPC may not be a breaking change (depending on strictness of parser, see above)
- Consider the case of a change to both input and output.
When we do a breaking change, how do actually perform it? When changing the type of a field, do we keep the old field as it is (until the end date support, see below) and introduce a new field with the desired type? Handle other cases.
- Do we want to have a
/internal
directory whose RPCs can be changed without further notice? - Do we want to have
/internal/
segments whose tails can be changed without further notice?
Process
With the definition of what a breaking change is, we can define a process for handling such a change. Here is what we have in mind for the list of things to do when there is a breaking change:
- Mark the RPC service/field with a
deprecated
tag in theOCaml
source code. - Mark the deprecation in the OpenAPI version of the API too (see Tooling support below for automating this)
- If possible, mark in the tag the end date of official support, for example 3 months or 6 months from now.
- what does "3 months" mean also needs to be clarified
- after this date, "it will be removed" or "we may remove it if we feel like it"?
- is it really 3 months or rather "the release that happens 3 months after the release where the deprecation was announced"?
- what does "3 months" mean also needs to be clarified
- Create an issue to keep track of the removal and link to the issue in a standardized format (like for TODO/FIXME) in the
OCaml
source code. Ideally we want to perform the removal when the end date of support is attained, taking opportunity from changes to the concerned code. - Explain the change in
CHANGELOG
, mentioning the end date of official support.
This process should be ultimately included in the developer documentation and be added to the list of checks done by #merge-team
members when reviewing.
Tooling support
We would like to have support for keeping track of changes. We would like the deprecated
tag to be automatically integrated to our export to OpenAPI, because OpenAPI supports declaring parts of the API as deprecated.
In addition, we would like to be able to detect changes in the CI. The basic version would consist of versioning the export to OpenAPI in the repo (as this export is fat (1.5M ish), we would maybe store only its hash) and have the CI check that the versioned version is correct. This means that a MR changing the API would have to update the versioned version, hereby making the change visible.
A fancier version is possible: use OpenAPI-diff to compute the change between the versioned version and the version of the MR.