in order to reduce backwards compatibility and give us more flexibility, i am proposing every X.0 we bump the analyzer versions by a major
i do not believe we have to agree to this as a section for it to be effective?
we do add the risk that if a security bug happens in X.0-X.1 we need to backport to the prior X-1 image (because major security bugs get backported 3 releases)
We are more likely to release minor versions that major ones (based on past experience), which means the evolution of our tools will add less cost to our maintenance as we just maintain one major version, most of the time. And in the rare scenario where we absolutely need to backport something to older versions (e.g. log4j), we are likely to have less major versions that minor ones to update.
I also think that doing it at least once a year and synchronously with the rails app makes it simpler to understand for our customers and developers. This is also good for hygiene as it gives us the opportunity to clean up old code.
@gonzoyumo would this be something we need to announce as a deprecation / removal? or is it more an FYI? (since we would be updating the templates etc)
@rdickenson do you reckon the reasoning behind that move? The maintenance and update policy of our scanners and vulnerability databases is not directly related to the vulnerability page so I'd rather move that section back to the main application security doc. WDYT?
There would publish with an image tag that corresponds to a major release, like 14. In the past, the image tags corresponded to minor releases, like 13-0-stable.
We would no longer back port to older versions of GitLab by default. Instead, we would push changes to the git branch that corresponds to the old version, like v14, and run a pipeline for that branch. Put another way, a pipeline wouldn't target multiple major releases of GitLab.
To me this proposal is similar to pinning the MAJOR version of the analyzers in the CI templates – what we currently do. The main difference is that the MAJOR would match the major release of GitLab. (Also, as a consequence the analyzers would all share the same MAJOR version, but that's already the case in the existing CI template.)
Pros
Users can predict and interpret the image tag. For instance, they now that gemnasium:15 works with GitLab 15.x.
Developers clearly know the maintenance policy of each major version, based on the release and maintenance policy of GitLab.
Developers are no longer put in a situation where it's tempting to deploy new features to older versions of GitLab (because it's easier) BUT they can't be totally sure that there'll be no side-effects.
Cons
Back ports (sec fixes and important bug fixes) are necessary, even if the latest version of the analyzer is backward compatible with all maintained versions GitLab. Developers maintain multiple git branches only to deploy fixes to older versions of GitLab.
The major version doesn't reflect changes to the analyzer. It's predictable (see pros), but it's no longe meaningful.
The major version of the analyzer is aligned with the major version of GitLab, but the minor version doesn't. This is totally confusing, in my opinion.
(Context: We are are deprecating bundler-audit and retire.js, and merging the Gemnasium projects, so eventually we should end up with a single project, but multiple images.)
To me the pros don't clearly outweigh the cons; if this is a mixed bag then I'd rather keep things as-is.
In any case, this is a great conversation, and it might help us see yet another option.
Be more agile and publish a new MAJOR version whenever we need one. AFAIR we haven't published a new major since we removed the Docker-in-Docker orchestrator.
Make the CI template make flexible, so that the analyzers of the same product category no longer share the same MAJOR version. We should get rid of DS_MAJOR_VERSION then.
Maintain a map that gives the major release of GitLab for any major version of the analyzer.
Refer to this map when we have to back port sec fixes and important bug fixes.
WDYT?
EDIT: It looks like this is what you were suggesting @NicoleSchwartz, right?
Back ports (sec fixes and important bug fixes) are necessary, even if the latest version of the analyzer is backward compatible with all maintained versions GitLab.
this is for 3 releases only, how much extra lift is it to maintain a branch? or a way to backport for 3-4 months? (like annoying? large, PITA?)
The major version doesn't reflect changes to the analyzer.
no, but it does reflect the GitLab version which is changed so i'm ok with that
The major version of the analyzer is aligned with the major version of GitLab, but the minor version doesn't. This is totally confusing, in my opinion.
interesting point! but we discourage minor pinning, hmmmm
Make the CI template make flexible, so that the analyzers of the same product category no longer share the same MAJOR version.
when you get a chance, tell me more, as in gemnasium-python and gemnasium-maven wouldnt have the same major? possible? why? is that just b/c we are tying them to the template and getting more chill with release often!
Maintain a map that gives the major release of GitLab for any major version of the analyzer.
Refer to this map when we have to back port sec fixes and important bug fixes.
Also i'm not 100% on matching the analyzer to the gitlab version, more just bumping the major every major (at least) in order to limit new features/fixes to new versions and encourage upgrading and reduce backward compatability bug risk
Initial setup. We need to change CI pipelines so that they have the same behavior for any version branch. Right now we can only deploy for the default branch. See .release job for instance. I think this is not too difficult, but there's probably trial and error involved in getting this right.
Maintenance. We'll have to create back ports for security fixes and important bug fixes.
I'm not sure about the initial setup. This is probably worth a PoC.
About maintenance, I see nothing complicated in creating a backport, and I suspect that each backport is like doing an easy issue, with a weight of 1 – just guessing.
tell me more, as in gemnasium-python and gemnasium-maven wouldnt have the same major? possible? why?
Well, I was thinking about retire.js, bundler-audit, and Gemnasium. But it's probably irrelevant now that we're deprecating retire.js and bundler-audit.
As for Gemnasium, the plan is to merge the 3 projects, so they would share the same versions, changelog, and release process.
Also i'm not 100% on matching the analyzer to the gitlab version,
Great!
more just bumping the major every major (at least) in order to limit new features/fixes to new versions and encourage upgrading and reduce backward compatability bug risk
I would argue that we don't need to do that if we know how the major versions of an analyzer map to the major releases of GitLab. Again, I'd rather not be in the scenario where we have to publish a major simply because there's a new major release of GitLab. I see the benefit the though.
In any case, I believe we should be ready for version branches and backports, and that we should then do the "initial setup" I talked about earlier. My understanding is that we already need these version branches because of the GitLab release and maintenance policy. So far we've managed to delay the release of a new major version, but maybe it has prevented us from deploying breaking changes when this was the right thing to do. cc @gonzoyumo WDYT?
Also i'm not 100% on matching the analyzer to the gitlab version
@fcatteau Yeah this is a no-go for me. We can't ensure we won't need to bump the major of the analyzer before the next yearly major update of GitLab. This is a difficult constraint similar to what we were suffuring from when we had to keep it in sync between the analyzers and the orchestrator.
I agree that we will have to adapt to support multiple major releases, but this is to a very limited extent here to me.
I already discussed with Nicole about that maintenance risk and she's very onboard with having a strict line of conduct when it comes to backporting. There should be only rare instances (e.g. log4j stuff) where we would have to backport to old major versions of the analyzer matching previous major releases of GitLab. The only period of time when the mainteance cost might increase is around the x.0:
when shipping 15.0, we still have to backport security fixes to 14.10 and 14.11, which will be on the previous major
when shipping 15.1, we still have to backport security fixes to 14.11, which will be on the previous major
This is not for bug fixes, only security fixes, so this is still very limited use case. I'm not even sure it is worth it to automate this today (will depend on complexity). I mean, in case of backporting, we could simply fallback to manually creating a tag/release, right?
so I need to make an issue to setup being able to deploy from branches it sounds like
if there was a major customer bug i could see releasing in 1-2 max 3 stable versions - but it would have to be major (think that vuln db not updating one) so that we could keep manual until it became a frequent problem (hopefully never), but yes we only have to do security patches
@gonzoyumo - I don't recall why the maintenance doc was moved to the Vulnerability Pages page. I agree with moving it to the main Security page, though I'd prefer to see it as a child page. The main Security page is quite long.
Should we announce deprecation of the current Major of our analyzer images?
As we decide to go that route, it also means we no longer want to maintain the previous major version. This could be implicit while publishing a new major but we should rather be explicit about it.
Granted we agree on that approach for the whole Sec section, I'd suggest a single deprecation announcement for all our images.