Testing GitLab / KAS / agentk combinations of different versions
In Updates and Version Compatibility in the GitLab Docs we make a rather bold statement about what we support:
For the best experience, the version of the agent installed in your cluster should match the GitLab major and minor version. The previous and next minor versions are also supported. For example, if your GitLab version is v14.9.4 (major version 14, minor version 9), then versions v14.9.0 and v14.9.1 of the agent are ideal, but any v14.8.x or v14.10.x version of the agent is also supported. See the release page of the GitLab agent.
While historically there have not been a lot of issues with that because the interface is rather stable and with protobuf + gRPC not too difficult to keep compatible.
However, there have been issues (e.g. #580 (closed)).
There are NO tests to actually guarantee that we support what we claim in the docs.
Proposal
A few possible ideas to implement some tests:
- As Mikhail mentioned in https://gitlab.slack.com/archives/C015U1TKV4M/p1716445328043789 (internal) fuzzy testing could help here
- Contract tests
- Actually have integration tests with those versions