Incident Review: Unintended participants mentioned in MRs

INC-6084: Unintended participants mentioned in MRs

Generated by Cameron McFarland on 4 Dec 2025 19:46. All timestamps are local to Etc/UTC

Key Information

Metric	Value
Customers Affected	8 reported via support tickets. List of tickets (internal link)
Requests Affected	Any requests accessing the participants API endpoints
Incident Severity	Severity 2
Impact Start Time	Tue, 02 Dec 2025 01:00:00 UTC
Impact End Time	Thu, 04 Dec 2025 04:40:00 UTC
Total Duration	11 hours, 3 minutes
Link to Incident Issue	https://app.incident.io/gitlab/incidents/6084 and https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20925

Summary

Background As part of discussions in this issue, and with approval from the AppSec team, we removed per-source and per-participant permission checks from the participants API endpoints to improve performance. These changes ensured that all participants were listed regardless of their current access level to the issuable's parent (project or group) or the participatable source (such as a note). We also updated the documentation to reflect that this is now the expected behaviour, although the documentation did not provide many details.

Problem: Previously hidden participants became visible in merge requests or issues when certain GitLab groups were mentioned. These participants were legitimate, but were previously kept hidden based on permission checks. The change revealed all participants regardless of viewer permissions, and without clear context about why these participants appeared, this was perceived as a security issue by customers.

Impact: The visibility of previously hidden participants generated a high volume of support tickets and customer dissatisfaction. This raised security concerns, but it has been confirmed that these participants did not gain unauthorized access to any private issues or merge requests.

Causes: Recent changes removed permission checks that previously hid certain participants from users who lacked access to the resources those participants had referenced. This made all participants visible to all viewers, regardless of their permissions.

Response strategy: We merged a revert (gitlab-org/gitlab!215092 (merged)) to undo both problematic changes and included the fix in the auto-deploy pipeline. The fix has now been rolled out to production environments. No backports are required, as the problematic change and the revert both occurred within the same milestone (%18.7).

What went well?

Summary based on discussion in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20930.

1. Effective coordination - EOCs successfully facilitated the incident, connected relevant team members, and assisted with managing the rollout/revert process.
2. Collaborative review - Team members openly shared perspectives and identified process gaps.
3. Successful resolution - The revert was implemented and deployed to production.

What was difficult?

Summary based on discussion in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20930.

1. Unclear who should lead - EOCs performed coordination work that may have been better suited for IMOC or the affected team.
2. Finding the right people - No automated routing meant manually looking up team members to handle application-level issues.
3. Missing verification criteria - Had to ask how to verify the revert was successful after deployment.
4. Knowing when to hand off - Unclear who should take over once the fix was identified.
5. Scope question - Should EOCs have been involved at all for an incident with no infrastructure impact?

Investigation Details

Timeline

Incident Timeline

2025-12-02

01:00:00 Impact started at

Custom timestamp "Impact started at" occurred

2025-12-03

17:55:19 Incident reported in triage by Bethany McGrew

Bethany McGrew reported the incident

Severity: Severity 2

Status: Triage

17:57:27 Incident accepted

Cameron McFarland shared an update

Status: Triage → Investigating

17:57:27 Escalated to GitLab.com Production

Your workflow manually escalated the incident to the escalation path GitLab.com Production

17:57:27 Escalated to Incident Manager Oncall (IMOC)

Your workflow manually escalated the incident to the escalation path Incident Manager Oncall (IMOC)

18:02:30 Message from Ryan Kelly

Ryan Kelly's message was pinned by Cameron McFarland

Some additional context can be found here (https://gitlab.slack.com/archives/C03EZMMCPQX/p1764782742952759). It appears that this may be expected behavior; however, Support is seeking further clarification due to the high volume of tickets reporting potential security concerns. The behavior still seems unintended, as external users are being listed as participants on Merge Requests, which has raised security concerns among customers.

18:03:10 Message from John Skarbek

John Skarbek's message was pinned by Cameron McFarland

We cannot rollback. We must move forward with a some other fix and deploy.

18:03:38 Message from Ryan Kelly

Ryan Kelly's message was pinned by Cameron McFarland

We suspect these:

• gitlab-org/gitlab!214345 (merged)
• gitlab-org/gitlab!213623 (merged)

18:11:29 Update shared

Cameron McFarland shared an update

Support is seeing an influx of tickets regarding external participants being added to MRs. Reviewing further, this occurs when the MR @ mentions a GitLab.com group such as @app or @criticcal. These users should not be listed as participants and is resulting in security concerns.

18:25:40 Message from Mario Sebastián Celi Calderón

Mario Sebastián Celi Calderón's message was pinned by Cameron McFarland

@gabe Weaver @nick Leonard adding you here to continue the discussion. But from what I can tell, I think we might need to go ahead and revert both MRs for this change and perhaps think of a better way to roll-out this change as it is already impacting customers in a negative way

18:38:28 Message from Agnes Slota

Agnes Slota's message was pinned by Cameron McFarland

As mentioned here (https://gitlab.slack.com/archives/C03EZMMCPQX/p1764783272198909?thread_ts=1764782742.952759&cid=C03EZMMCPQX), the decision to remove per-source (gitlab-org/gitlab!213623 (merged)) and per-participant (gitlab-org/gitlab!214345 (merged)) permission checks from the participants API endpoints was approved by the AppSec team (https://gitlab.com/gitlab-org/gitlab/-/work_items/577825#note_2867015374).

We've also updated the documentation to reflect that this is now the expected behaviour (but further documentation updates might be required to add more details).

• https://docs.gitlab.com/user/project/merge_requests/#participants
• https://docs.gitlab.com/user/project/issues/managing_issues/#participants cc @Mario Celi

18:44:06 Update shared

Cameron McFarland shared an update

A revert for the recent change suspected to cause unintended external participants in merge requests has been prepared. The proposed fix is available as MR #215080.

This change was expected, approved, and documented. It may just be a question of wether to revert for customer response, or let the intended design remain in production.

There is no indication of performance or service inoperability.

19:14:14 Message from Agnes Slota

Agnes Slota pinned their own message

Thanks everyone! I'll restart the discussion in this issue (https://gitlab.com/gitlab-org/gitlab/-/work_items/577825) tomorrow morning, so we can decide on the next steps.

cc @alexc @heinrich

19:55:00 Identified at

Custom timestamp "Identified at" occurred

19:55:51 Status changed from Investigating → Fixing

Cameron McFarland shared an update

Status: Investigating → Fixing

The team agreed to revert the recent changes that caused external users to be added as merge request participants when GitLab groups are mentioned. This decision was made due to customer dissatisfaction and concerns about misalignment with user expectations.

A new revert is prepared as MR #215092, which undoes both problematic changes. The team is moving forward with merging this revert to resolve the issue.

20:43:47 Message from Mario Sebastián Celi Calderón

Mario Sebastián Celi Calderón's message was pinned by Cameron McFarland

The revert was merged and already picked into auto-deploy

21:02:44 Update shared

Cameron McFarland shared an update

The revert to address unintended external participants being added to merge requests has been merged as MR #215092 and included in the auto-deploy pipeline.

The deployment is not yet complete. Handover instructions have been given to ensure the APAC team notifies stakeholders when the fix is fully deployed.

22:05:53 Message from Heinrich Lee Yu

Heinrich Lee Yu pinned their own message

Reposting here so we can record it in the incident timeline:

Just to clarify, even if these users show up as participants, they do not get access to the private issue or MR

So this is not a security vulnerability

2025-12-04

04:39:09 Update shared

Nick Duff shared an update

The revert to address unintended external participants being added to merge requests was merged as MR #215092 and included in the auto-deploy pipeline.

The fix has been rolled out to production environments.

It's also been clarified that while external users appeared as participants in some merge requests, they did not gain access to any private issues or merge requests. This was not a security vulnerability, but customer concerns and support volume prompted the revert.

04:40:00 Monitoring at

Custom timestamp "Monitoring at" occurred

04:40:00 Fixed at

Custom timestamp "Fixed at" occurred

04:40:48 Status changed from Fixing → Monitoring

Nick Duff shared an update

Status: Fixing → Monitoring

04:58:19 Incident resolved and entered the post-incident flow

Nick Duff shared an update

Status: Monitoring → Documenting

We've confirmed that the revert successfully remediated the issue. Testing in a private work item by mentioning an unauthorized user resulted in no new participants being added, as expected.

2h later

07:41:48 Message from Agnes Slota

Agnes Slota pinned their own message

I'll restart the discussion in this issue (https://gitlab.com/gitlab-org/gitlab/-/work_items/577825) tomorrow morning, so we can decide on the next steps. I've shared a list of proposed next steps to discuss and decide on here (https://gitlab.com/gitlab-org/gitlab/-/issues/577825#note_2932191009). Feel free to share your thoughts there.

Investigation Notes

Any details you may want to add about the investigation can go here.

Follow-ups

Follow-up

• Review: Incident-6084 EOC involvement

• Discuss how we move forward and decide whether we want to reintroduce these changes

Review Guidelines

This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI.

For the person opening the Incident Review

• Set the title to Incident Review: (Incident issue name)
• Assign a Service::* label (most likely matching the one on the incident issue)
• Set a Severity::* label which matches the incident
• In the Key Information section, make sure to include a link to the incident issue
• Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) The DRI for the incident review is the issue assignee.

For the assigned DRI

• Fill in the remaining fields in the Key Information section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find.
• If there are metrics showing Customers Affected or Requests Affected, link those metrics in those fields
• Create a few short sentences in the Summary section summarizing what happened (TL;DR)
• Link any corrective actions and describe any other actions or outcomes from the incident
• Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported?
• Once discussion wraps up in the comments, summarize any takeaways in the details section
• If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue.
• Close the review before the due date
• Go back to the incident channel or page and close out the remaining post-incident tasks