Ensure group webhooks are always enabled

Spinning off from #1182 (comment 1486937016)

It's been an issue that webhooks in https://gitlab.com/gitlab-org we're using for the service can be auto-disabled. In the original issue we're talking about raising visibility when that happened. This issue focuses on making sure it's always enabled without human intervention.

Potential solutions

• Product feature to make sure it'll self-heal: gitlab-org/gitlab#396577
• Figuring out if there's any bugs to auto-disable webhooks, because it doesn't seem to add up that it should be auto-disabling it so often: gitlab-org/gitlab#396577 (comment 1433229017)
  • Improve observability of the feature by adding logging: gitlab-org/gitlab!128212 (merged)
• Use an owner token to automate re-enabling the webhooks:
  • Periodically, or
  • When we detected that it's likely going down
  • We can do both!
• Change the product so enabling group webhooks doesn't need to have owner permission, but maintainer permission, so we can use a maintainer token for this
• Make it able to disable auto-disabling for this case

See also gitlab-org&8083

added Engineering Productivity bugavailability labels

added typebug label

mentioned in issue #1182 (closed)

marked this issue as related to #1182 (closed)

marked this issue as related to gitlab-org/gitlab#396577

marked this issue as related to #1178 (closed)

marked this issue as related to gitlab-org/quality/engineering-productivity/team#149 (closed)

marked this issue as related to #1352 (closed)

changed the description

• Figuring out if there's any bugs to auto-disable webhooks, because it doesn't seem to add up that it should be auto-disabling it so often: gitlab-org/gitlab#396577 (comment 1433229017)

I'm working on adding logging to the auto-disabling behavior (I'm surprised there is no logging at all already)!

This is done in gitlab-org/gitlab!128212 (merged) and we identified the issue: https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure/-/issues/96 and fixed it: https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure/-/merge_requests/437

I would guess this is the root culprit because this can explain the webhook being disabled "a bit randomly".

Let's wait and see if this will ever happen again. If this is stable enough, I think we can stop here

@rymai Thank you so much for getting to this. I think my lesson here is what you said on Slack:

Example of why logging is useful, and why the EP team can contribute in improving observability in the product in order to improve our own tools.

And:

Yes, I think we shouldn’t be afraid to contribute to the product sometimes, as it can unblock us as well as allow improve long-term quality of features.

/cc @gl-quality/eng-prod

Thanks! I found another thing we can probably improve thanks to the logs: https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure/-/issues/97

Use an owner token to automate re-enabling the webhooks:

I agree, we might have to do that to ensure availability. We could have a regular check and automatic re-enablement if needed.

• Change the product so enabling group webhooks doesn't need to have owner permission, but maintainer permission, so we can use a maintainer token for this

That would definitely make sense!

changed the description

mentioned in merge request gitlab-org/gitlab!128212 (merged)

changed the description

mentioned in issue #1367 (closed)

changed milestone to %Backlog

added priority2 label

Webhooks for gitlab-org is likely disabled again and tested at #1353 (comment 1616242437)

There might have some other issues other than https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure/-/issues/96 last time we solved.

I created an incident: #1417 (closed)

mentioned in issue #1417 (closed)

marked this issue as related to #1417 (closed)

Expanding on this point:

• Use an owner token to automate re-enabling the webhooks:

  • Periodically, or

  • When we detected that it's likely going down

  • We can do both!

In the team meeting, @splattael mentioned that we can hook into when Kubernetes decides to restart or move pods around, trigger a CI job to re-enable the webhooks again. Specifically, that's:

When we detected that it's likely going down

I am not sure if we can hook into that, but we should be able to trigger the CI job when the application is starting, which we can run whatever we want. If Kubernetes is restarting or moving the application pods, which should start the application then it can trigger the CI job. This should help because that's likely when the webhook can be disabled, based on the last incident: #1417 (comment 1616398921)

One item above the idea in the Google docs agenda, @rymai mentioned that we can create a private project to use a group token, enabling the webhooks periodically. So we can do both!

This should be a cheap API call if there's an API, so I think we can do both and that would be the most reliable workaround.

I created https://gitlab.com/gitlab-org/quality/engineering-productivity/triage-ops-webhooks-lifeguard, but realized there's no API to test a webhook, so we'll need to contribute it first.

marked the checklist item Improve observability of the feature by adding logging: gitlab-org/gitlab!128212 (merged) as completed

mentioned in issue gitlab-org/quality/engineering-productivity/team#312 (closed)

The webhook for gitlab-org was disabled again: gitlab-org/gitlab!147164 (comment 1816807946)

Looking at #1182 (comment 1486937016) we did get a Slack notification like 5 days ago, which was off.

I think it really shouldn't disable the webhook so fast: (Screenshot from @stanhu)

Quoting myself:

I think this is really the problem. Even with 99.992% uptime the webhook still thought the service was broken and disabled it. I don't think this is reasonable at all. How many services have 99.992% uptime?

Might be worth to mention that we had a similar incident at gitlab-org/gitlab#396577 (comment 1433229017) where it did look like it's disabled way too aggressively.

I think another evidence that there's some bugs is that, gitlab-com webhooks didn't get disabled so often. The same service, one got disabled so much more often than the other, doesn't feel right.

gitlab-com has much lower traffic but still.

Relevant epic: gitlab-org&8083 (Can't relate to an epic, so linking it here)

changed the description

mentioned in issue gitlab-org/gitlab#396577

Is gitlab-org webhook still up?

It was indeed disabled. Stan saw this and activated again:

removed typebug label

added typefeature label

removed bugavailability label

added bugavailability label

added typebug label and removed typefeature label

added severity2 label

severity2 with bugavailability can only have priority1

Please see the Availability prioritization guidelines for more detail.

This message was generated automatically. You're welcome to improve it.

I suppose I'll downgrade then.

added priority1 label and removed priority2 label

added priority2 label and removed priority1 severity2 labels

We discussed this in the team meeting today: https://docs.google.com/document/d/1Q18REIcGhmSaKxuQ5ER15q-Vj31iYOl6fjAj6sMHKlw/edit#bookmark=id.mnx0km907ln9

We'll first see if we can disable the feature for auto-disable webhooks for gitlab-org according to: gitlab-org/gitlab#396577 (comment 1347622605) (Thanks @jennli for this!)

And if that doesn't work, we'll ask for getting more owners for gitlab-org (Thanks @yanguo1 for this!)

Not sure who we should grant owner permission, but maybe 2 AMER timezone and at least one more EMEA timezone, so we have 2 for both timezone. (I am more early EMEA timezone than APAC)

Thanks @splattael for pointing out the feature flag we're looking for as well: auto_disabling_web_hooks: gitlab-org/gitlab#390157 (closed)

I haven't looked into this, and I hope this can be group based instead of instance based. (And we're not going to disable it if it's instance based)

We can consider making this group based as well, if that's feasible.

@godfat-gitlab I hope it's as simple as

diff --git a/app/models/concerns/web_hooks/auto_disabling.rb b/app/models/concerns/web_hooks/auto_disabling.rb
index 72812f35f724..78a9ec25ab6f 100644
--- a/app/models/concerns/web_hooks/auto_disabling.rb
+++ b/app/models/concerns/web_hooks/auto_disabling.rb
@@ -17,7 +17,7 @@ module AutoDisabling
       def auto_disabling_enabled?
         enabled_hook_types.include?(name) &&
           Gitlab::SafeRequestStore.fetch(:auto_disabling_web_hooks) do
-            Feature.enabled?(:auto_disabling_web_hooks, type: :ops)
+            Feature.enabled?(:auto_disabling_web_hooks, parent, type: :ops)
           end
       end
 

See gitlab-org/gitlab!148089

Edit: Nope, it's more involved

@splattael Thank you! Would you mind if I pass this to you? I don't think we need to rush this at all, and if we can disable that for gitlab-org, let's do that because that's completely solving the problem for us, and I think that's way way better than any alternatives for us. (The best is still finding and fixing the bug, but I am not feeling like digging into that )

I am asking help from Luke at gitlab-org/gitlab#390157 (comment 1835902066)

@godfat-gitlab

Would you mind if I pass this to you?

Not at all! Happy to work on this

assigned to @godfat-gitlab

changed milestone to %16.11

mentioned in merge request gitlab-org/gitlab!148089

mentioned in issue gitlab-org/gitlab#390157 (closed)

assigned to @splattael

I also just thought we could leverage custom roles https://docs.gitlab.com/ee/user/custom_roles.html and add a custom ability to "manage group webhooks".

That's great! Sadly we need to add that custom permission for group webhooks as well.

Just to keep you in the loop- we successfully built this during community pairing last week: https://youtube.com/live/drODUf5cthY

The MR needs splitting up because there are a few oddities in there at the moment. Here's the first MR: Standardize webhook abilities (gitlab-org/gitlab!149864 - merged)

Hopefully it'll land next week and i'll get the final MR in to add the ability to custom roles

gitlab-org/gitlab!151551 (merged) Just hit canary so we can now create a custom role which will allow “us” to manage webhooks for GitLab-org without being group owners

Who wants to set it up? (You need to be group owner to create the custom role)

Happy to walk you through it if needed @gl-quality/eng-prod

@leetickett-gitlab thanks so much for this great work @leetickett-gitlab ! we can reach out to one of the gitlab-org group owners to help:

And by watching the recorded video I believe the steps needed to set it up is the following:

1. Navigate to https://gitlab.com/groups/gitlab-org/-/settings/roles_and_permissions
2. Create a custom role called Maintainer + Web hook admin, choose Maintainer as the base role and check the Admin web hook checkbox
3. Then go to https://gitlab.com/groups/gitlab-org/-/group_members, grant this Maintainer + Web hook admin role to the appropriate team members

I think to start with, we can get everyone on EP to have this custom role? @leetickett-gitlab should members from contributor success also receive this custom role?

I created a new Developer + Web hook admin + CI/CD variables admin for EP team members!

Those steps look spot on @jennli

The only thing I wonder if we should consider using group shares instead of individuals? That way when people join/leave teams we won't have additional admin steps of adding/removing permissions?

should members from contributor success also receive this custom role?

I would certainly appreciate it- especially as we would see the banner when the webhook is disabled (actually, we wouldn't because the permission check for the banner is wrong... i'll spin up an MR for that)

Ooops- I didn't submit this in time!

@rymai I think you updated each individual's role, wdyt of switching to use group shares?

I think you updated each individual's role, wdyt of switching to use group shares?

I don't see group share at the gitlab-org group level?

I don't know why it wouldn't be there- but obviously I don't have permission to check

Conveniently we have a catch up in the morning @rymai so we could take a look?

@rymai Is it possible to create a group access token with this new custom role so that we can use the access token to constantly try to activate the webhook from triage-ops?

If we can, perhaps that would be the quickest way to mitigate this?

I don't believe you can apply custom roles to group access tokens, but I do believe you can apply them to service accounts

I am creating a custom role for https://gitlab.com/groups/gitlab-com as well.

• Name: Developer + Web hook admin + CI/CD variables admin
• Description: For engineering productivity to manage web hooks and CI/CD
• Permissions:
  • Developer
  • Admin cicd variables
  • Admin web hook

I would like to mirror this role as https://gitlab.com/groups/gitlab-org but I could seem to easily find who has custom roles from the members page. I think it's worth to give this also to Contributor Success but I'll try to give this to Engineering Productivity first.

Not sure if we can give this to a group but I'll see. Edited: No, I can't. When I tried to share to a group, there's no custom roles I can pick. That option only shows up when inviting an individual.

I can't share to a group with a custom role, so I updated the roles for each individual based on the members of https://gitlab.com/gl-quality/eng-prod except for whom are already an owner. I didn't mean to downgrade even though the very reason that I was given owner permission is now not needed. (It's convenient to be honest)

Additionally, I also set this custom role for @leetickett-gitlab because even if we don't give this role to Contributor Success, I think he should also have this permission given that he's often on top of this and the contributor to this great feature! I am sure it'll be useful.

Thank you @leetickett-gitlab again for all your awesome contributions!

gitlab-org/gitlab!151551 (merged) Just hit canary so we can now create a custom role which will allow “us” to manage webhooks for GitLab-org without being group owners

I wanted to propose to expand gitlab-org/gitlab!151551 (merged) to also ensure the "new group owners" also receive emails when those hooks are disabled, but I think we don't even send emails to "full" group owners when webhooks are disabled

For your information, there's an UI bug to show the webhook setting: gitlab-org/gitlab#466682

unassigned @godfat-gitlab

added severity1 label