Inform in the issue or MR when the change landed in one of the environments

changed title from Consider pushing our auto-deploy tag into the ee repository to Consider pushing our auto-deploy tag into the ee repositories

The problem we are trying to resolve is increase the visibility of what was deployed. Alternatively to tagging in EE, we might want to expose https://ops.gitlab.net/gitlab-com/gl-infra/deployer/blob/master/releases/releases.json better

Right now we only push branches, is this good enough? Presumably not as there's a bit of disconnect between a branch and figuring out where we are in the timeline of the repo when we tag.

I thought the reasoning behind not tagging is because we have the EE sha in the semver anyway?

I think we should be resolving this on another level. The problem that we are trying to resolve is increase the visibility but that is not sufficient, people appreciate more to have a notification when something landed in one of the environments.

changed title from Consider pushing our auto-deploy tag into the ee repositories to Inform in the issue or MR when the change landed in one of the environments

As seen in gitlab-com/Product#199 (comment 186185603), there is a lot of confusion with developers and others related to when a change is landing in one of the environments. This is not limited to devs only, Delivery is also getting confused with larger number of deployments we are doing.

Based on the comment in gitlab-com/Product#199 (comment 187055752), I wonder if we could do an iteration of our QA issue given that we have all the info we need there:

When QA task issue is opened, the release bot applies a label workflow::In staging in addition to the QA issue
We run the QA task again when we deploy to canary, but without creating an issue. Bot would apply workflow:: In canary label. Same for production deploy

This should give us enough visibility across the board on when a change lands in an environment and should remove the general confusion. Based on that, developers can then go and close the issue that was assigned to them.

@marin this sounds great!

One problematic detail is that the QA issue lists the MRs, while the workflow label should be applied to the issue. Not sure if we can reliably determine the correct issue from the MR?

@toupeira We could use related issues feature if we are exposing it in the API (I really hope we do). The only problem I see there is when there is more than one related issue.

Doesn't look like we do ATM, see e.g. https://gitlab.com/api/v4/projects/13083/merge_requests/29598 or https://gitlab.com/api/v4/projects/13083/issues/59023

Also the explicit "Related issues" feature seems to be only available on issues, on MRs only references in the description are supported.

@marin I had another thought on this: GitLab already has a concept of environments and deployments to track this, but I'm not sure how much of that is tied to Kubernetes / using environment: in CI. It is possible to manually create environments, so maybe we can extend the Deployments API to be able to POST a new deployment with the commit SHA, and have it automatically show up on all the deployed MRs?

This would be very nice because it means we're dogfooding more of that functionality, and could also be useful for other GitLab users who currently don't use CI/K8s for deployment.

Some more complications with my wild idea:

Not sure if the deployed environment shows up on issues in the "Related merge requests" box. This might make sense to add, though.
As long as we still have the two codebases we would need to add the environments/deployments to both the gitlab-ce and gitlab-ee projects. If it's easy to determine the CE commit from the deployed EE commit this might be straightforward, otherwise we would have to wait for the merged codebase.

GitLab already has a concept of environments and deployments to track this, but I'm not sure how much of that is tied to Kubernetes / using environment: in CI. It is possible to manually create environments, so maybe we can extend the Deployments API to be able to POST a new deployment with the commit SHA, and have it automatically show up on all the deployed MRs?

This would be very nice because it means we're dogfooding more of that functionality, and could also be useful for other GitLab users who currently don't use CI/K8s for deployment.

I think there would be a disconnect in that the "deployment" we'd create isn't running as a review app or environment, at least in the context of GitLab (the application). We'd basically be faking a deployment pointing to staging (or canary, or production). Beyond general confusion, I'm not sure what technical issues that may cause.

@rspeicher exactly, I'm wondering how large/flexible this disconnect really is. My intent is that we could reuse this UI for staging/canary/production, which probably wouldn't be confusing and exactly the information we need (although visible only on MRs currently):

But I'm not sure where else deployments are exposed in the UI, and if we have code which needs deployments to be tied to a "live" environment (e.g. the stop button in the screenshot above is already optional on the frontend side). So this would definitely require some more investigation, let me know if I should open a separate issue for that.

mentioned in issue #345 (closed)

mentioned in merge request gitlab-org/release-tools!660 (merged)

@jarv pointed me to this issue as I suggested this to him in Slack.

Having notes on MRs when the MR lands in each environment (staging, canary, production) etc would be incredibly useful.

In particular, when an urgent MR is merged, many parties (who many not be cc'ed on the QA issue) will be concerned about when it arrives in production. Having to find the QA issue, figure out its status etc is quite an arduous process. Having notes on the MR stating the time it landed would be much better

I want :-)

Me too! We will prioritise this issue as soon as we can, we are tackling some higher prio items.

marked this issue as related to #416

gitlab-com/chatops!70 (merged) adds an auto_deploy status ChatOps command. Here's an example output:

The next step will be to allow the command to take a SHA argument, allowing developers to see which auto-deploy branch(es) contain that SHA, and if/where it's currently deployed.

marked this issue as related to #441 (closed)

marked this issue as related to #141 (closed)

marked this issue as related to #371 (closed)

I've been thinking about this (and related) problems for a bit and I believe that we need to tackle them at (nearly) the same time.

Problem description

There are a number of requirements in various places (see related issues), so summarising them per overview level:

High level: Show all changes (MR's of all projects) included in the package being deployed.
Medium level: Individual MR's should show not only which release they ended up in, but also when they have been deployed to various GitLab.com environments.
Lower level: Transparent information of the version currently running on GitLab.com and environments for the whole community.

High level overview

The overview of what is being deployed is interesting to everyone using GitLab.com as it gives people a quick overview of changes. This is basically the CHANGELOG of deploys. This is currently exists in the form of manual QA task issues. At the moment of tagging a deploy, we collect all MR's using the release-tools project in order to create an overview and also mention people to ensure they can check their changes in a few environments. The QA task is insufficient as it currently stands. Ideally, we would could be looking into using something like https://gitlab.com/gitlab-org/release/framework/issues/141 but we have the following requirements:

The deploy overview needs to show not only changes in GitLab CE/EE but also satellite repositories and packaging projects (omnibus/charts)
The overview can't be public immediately since we do security releases

Medium level overview

Each individual MR that is a part of the deploy has a milestone assigned. This milestone shows when the MR is executed on and also the targeted self-managed release. However, that is not sufficient information in order to deduce when was the change rolled out to GitLab.com environments given that the targeted self-managed release comes after the deploy.

Requirements for this level are:

Each MR should automatically get a label that describes the lifecycle of the MR. We currently use workflow labels so something like:
workflow::verification - when the MR has been merged but not yet deployed / released
workflow::staging - when the MR landed on a staging environment
workflow::canary - when the MR to a portion of users
workflow::production - when MR is on GitLab.com
workflow::release-12.2.2 - when MR has been released to the self-managed customers

Lower level overview

Right now we have gitlab.com/help showing the version and the revision of currently running GitLab instance. The version is incorrect for GitLab.com and revision only shows CE/EE revision.

We also have the full information of component version in the /admin page of the instance. I do not remember why we had limited this to admins only.

Requirements for this level of overview I can think of are:

Ensure that the version is not shown if it is not a tagged (semver) release
Show the revision of GitLab Rails but also other satellite repositories

Option 1

As a common ground for all three overview levels, ideal solution would involved dogfooding our own releases feature :

Select a project (not CE/EE, maybe next.gitlab.com?) for the releases page
The releases page would publicly show only items that are ready to be public
- In the mean time, it would only show all content to project/group members
The releases page would be able to have specific labels applied to the the specific releases to show the progression of the release through environments
We would be able to parse the releases page for links to MRs included and at the same time update all MR's with the label we are about to apply to the specific version
Expand the help page to allow adding external links. This would allow replacing the revision with a link to the releases page

This is great moonshot but I am afraid that our releases feature is not mature enough to accomplish this, nor I am sure we want that feature to go that way.

Option 2

More realistic and iterative option might be to:

Change the QA task issue to be machine parseable
QA task automation that already updates the task issue, could parse the issue and based on the environment deployed, would apply applicable labels to parsed MRs
Expand the help page to contain revisions of other components deployed

@marin

Populating the release page is probably the easiest way to clearly communicate the state of a release. However, I don't think this is a good long term solution. The releases page is basically a display of annotated tags. This works because we currently generate a tag for a release, but I feel that those tags should not stay around forever: they are not useful to anybody but us, and they only further increase the repository size.

Setting the tags aside, the releases page feels like something meant for actual releases; not deploys to environments that we control. This is mostly just a matter of pedantics though, so we might just ignore it.

An entirely different approach would be to commit a Markdown file of sorts to a separate repository. This repo would then produce a website on GitLab Pages that displays the list of changes. I think this would be a bit more accessible and flexible, but it's also a bit more tricky to set up (though not that much).

For labelling MRs: as long as we have merge requests coming from dev we won't be able to reliably tag them, or at least will have to go through trouble to get this done. Even then, since nobody really checks dev outside security releases having the info there is not super useful.

Label wise I am a bit uncertain about using labels like workflow::release-X-Y-Z. We will over time end up with a lot of labels, and I feel labels are the wrong approach (a release is closer to a milestone than a label). Of course we don't really have an alternative for now, so we likely just have to set this aside.

I think an important question to ask here is "Who is our target audience?". Blog posts and changelogs should be sufficient to customers/users, and I feel that tagging individual MRs does not add much value on top of that. For our developers it would be useful to see what version first introduced and MR, but I think GitLab can already do this when you use the environments feature. Perhaps we can find a way to reuse that?