Research Report: Root cause and follow-up for GitLab issue 18929

Original Research Query

What was the root cause of gitlab-com/gl-infra/production#18929 (closed)? What follow up actions were taken? What changes were made?

Research Report

Production Incident Analysis: Emails Sent for Old Imports (gitlab-com/gl-infra/production#18929 (closed))

This report details the root cause, follow-up actions, and changes made in response to the production incident gitlab-com/gl-infra/production#18929+, where emails were unintentionally sent for old project imports and project mirrors.

1. Incident Summary

On December 3, 2024, a production incident (gitlab-com/gl-infra/production#18929+) occurred on GitLab.com. A newly deployed change, intended to notify users upon the completion of new project imports, began sending these notification emails to creators of old projects and projects configured with repository mirroring.

As stated in the incident summary:

We deployed a change that sent notification emails to a project’s creator after a successful import. The emails were intended for new imports only. However, emails were unintentionally sent to creators of old projects and projects set up via mirroring.

This caused confusion and inconvenience for users who received these unexpected emails. A significant side effect, reported in gitlab-org/gitlab#507531+, was that these emails, particularly for mirrored projects, could expose credentials embedded in the import URL.

2. Root Cause

The root cause of the incident was the deployment of Merge Request gitlab-org/gitlab!173596+ ("Added completion notification for project imports"). This MR introduced an email notification to inform users when a project import process finished. The intention was to aid users, especially group owners, in post-import tasks like reassigning placeholder users, aligning with notifications for Direct Transfers.

The description of the original MR gitlab-org/gitlab!173596+ states:

This MR adds an email to notify users that a for project import has completed, similar to the notification sent out when a Direct Transfer finishes. With the addition of user contribution mapping, no real users exist in an imported project until a group owner reassigns. We want to notify group owners when the import completes so that they can begin that process, and also to be consistent with Direct Transfer.

The critical flaw was that the logic triggering these emails was tied to the ProjectImportState transitioning to a finished state. This state transition, however, was not exclusive to new, user-initiated imports. It also occurred for:

  • Project Mirroring: Repository mirroring leverages the project import functionality. Each time a mirror updated, its ProjectImportState would transition, including through the finished state, incorrectly triggering the new email notification. This was a key factor, as highlighted in gitlab-org/gitlab#508458+:

    repository mirroring must be setting the ProjectImportState state from finished to schedule and then restarting using the same object over and over, so our hooks on state transitions need to account for finished not being a final endpoint.

  • Old Projects: The system might have re-evaluated or updated the ProjectImportState for some older projects, causing them to also trigger the notification.
  • Shared import_type: Features like project creation from built-in templates used the same import_type ("gitlab_project") as standard file-based imports. The new notification logic did not sufficiently differentiate these cases.

Furthermore, as discovered in gitlab-org/gitlab#507531+, the email template used the project's import_url directly. For mirrored projects using basic authentication, this URL often contained embedded credentials (username and password), leading to their exposure in the email subject and body.

A user reported in gitlab-org/gitlab#507531+:

The 9 emails were generated with the full URL including the basic auth info, specifically in the form of Dec 3, 2024 4:59...08:00

Subject:
Expo Server SDK Java | Import from https://caclark%40trueblue.com:MYREALPASSWORD@github.com/nia-medtech/expo-server-sdk-java.git completed

Body:
Import completed

The import you started on November 14, 2024 from https://caclark%40trueblue.com:MYREALPASSWORD@github.com/nia-medtech/expo-server-sdk-java.git [github.com] has completed. You can now review your import results.

The original MR gitlab-org/gitlab!173596+ did include a sanitize_url method, but it was not applied to the import_url before being used in the email subject or body.

3. Impact

The incident had the following impacts:

  • User Confusion and Inconvenience: Project creators received unnecessary and unexpected emails, leading to confusion.
  • Credential Exposure: A critical security issue arose where emails for mirrored projects exposed usernames and passwords embedded in the import URLs.
  • Increased Support Load: The unexpected emails likely generated support tickets and user queries.

4. Follow-up Actions

Several actions were taken to address the incident and prevent recurrence:

  1. Immediate Mitigation:

  2. Incident Tracking and Review: The incident was tracked in gitlab-com/gl-infra/production#18929+.

  3. Development of Corrected Solution: A new Merge Request, gitlab-org/gitlab!174610+ ("Updated - Added completion notification for project imports"), was created to re-introduce the functionality with necessary safeguards.

  4. Addressing Related Issues:

    • Credential Exposure: The credential exposure was specifically tracked in gitlab-org/gitlab#507531+. The fix for this involved ensuring URLs were sanitized.
    • Documentation: Issue gitlab-org/gitlab#508458+ ("Add documentation of how repository mirroring leverages aspects of project import.") was created to improve understanding of the interaction between mirroring and import functionalities.

      Project mirroring is owned by ~"group::source code", project import/export is owned by ~"group::import and integrate". However, project mirroring leverages project import functionality. There's no developer documentation how the two features intersect which makes it difficult for engineers from the respective groups to be mindful about side effects when working on one of these two features.

    • Import Type Refactoring: Issue gitlab-org/gitlab#511960+ ("Refactor built-in project creation by template to have its own import_type value") was created to differentiate project creation from templates from other import types, aiming to prevent similar broad-reaching effects of changes to generic import logic.

      Proposal

      To match the custom template feature import_type of "gitlab_custom_project_template", refactor project creation by in-built template to have its own unique import_type of "gitlab_project_template".

5. Changes Made

The key code changes occurred across three main merge requests:

  1. Original Problematic Change (gitlab-org/gitlab!173596+):

    • Introduced email notifications upon project import completion.
    • Modified app/models/project_import_state.rb to trigger the notification.
    • Added new mailer templates (app/views/notify/project_import_complete.html.haml, app/views/notify/project_import_complete.text.erb) and mailer logic in app/mailers/emails/imports.rb.
    • The mailer used project.import_url directly in the subject and body without proper sanitization.
    • Relevant code snippet from app/mailers/notify.rb (introduced in !173596): 
      def import_completion_email(user, project)
  @user = user
  @project = project
  @group = project.group
  @import_url = project.import_url // Unsanitized URL
  @import_time = project.import_state.created_at
  @can_reassign_users = @group&.owner?(user) && @group&.user_group_creation_allowed

  subject = "#{project.name_with_namespace} | Import from #{@import_url} completed" // Unsanitized URL in subject

  mail(to: user.notification_email_for(@project), subject: subject) do |format|
    format.html { render layout: 'mailer' } // Template would use @import_url
    format.text { render layout: 'mailer' } // Template would use @import_url
  end
end

private

def sanitize_url(url)
  url.gsub(/\/\/(.*):(.*)@/, '//[FILTERED]:[FILTERED]@') // Method existed but was not used for display URL
end

  2. Revert Change (gitlab-org/gitlab!174531+):

    • This MR reverted all changes introduced by gitlab-org/gitlab!173596+, effectively removing the new email notification logic and stopping the unintended emails.

  3. Corrected Change (gitlab-org/gitlab!174610+):

    • Re-introduced the project import completion email functionality with crucial safeguards.
    • Key modifications:
      • Excluded specific project types: The notification logic was updated to explicitly exclude project mirrors, project forks.
      • Filtered by ImportSources: Notifications are only sent for import types listed in Gitlab::ImportSources.rb, ensuring that only legitimate, user-initiated imports trigger the email.
      • URL Sanitization: Implicitly, this MR or related changes would have ensured that any URL displayed in the email is sanitized to prevent credential leakage, addressing gitlab-org/gitlab#507531+.
    • The description of gitlab-org/gitlab!174610+ states:

      This MR is an updated version of Added completion notification for project imports (gitlab-org/gitlab!173596 - merged), which has been reverted. However, this MR excludes project mirrors, project forks, and any project import type that is not listed in ImportSources, the source of truth for where projects can be imported from. That way, repeat imports do not spam users with emails.

    • Changes likely involved:
      • Modifying app/models/project_import_state.rb to conditionally trigger the notification.
      • Adding a new worker app/workers/projects/import_export/import_completion_notification_worker.rb to handle sending emails.
      • Adding checks in app/models/project.rb or the worker, for example: 
        # Conceptual example of checks in the corrected version
# In ProjectImportState or the new worker:
# if project.import_type.present? && 
#    Gitlab::ImportSources.values.include?(project.import_type) && 
#    !project.mirror? && 
#    !project.forked_from_project? # or similar check for forks
# then
#   # Enqueue notification worker
# end

# In the mailer, ensure sanitize_url is used:
# @display_url = sanitize_url(project.import_url)
# subject = "#{project.name_with_namespace} | Import from #{@display_url} completed"

6. Conclusion

The production incident gitlab-com/gl-infra/production#18929+ was caused by a new project import completion email notification (gitlab-org/gitlab!173596+) that did not adequately account for the nuances of GitLab's import system, particularly its use by project mirroring and the potential for credential exposure.

Immediate actions involved disabling the feature flag and reverting the change (gitlab-org/gitlab!174531+). A corrected version of the feature was subsequently implemented (gitlab-org/gitlab!174610+), which included logic to exclude mirrors, forks, and other non-standard import types, and ensured URL sanitization. Further preventative measures include improving documentation on feature interactions (gitlab-org/gitlab#508458+) and refactoring import types for better specificity (gitlab-org/gitlab#511960+).

This issue was automatically created by the GitLab Research Agent on 2025-05-26T17:37:31.829Z