Investigate 500 errors caused by GleanSearchCrawlerBot

Summary

Starting around 2025-06-05 06:00 UTC, we observed a significant increase in API error rates caused by a single user agent called GleanSearchCrawlerBot. The errors are primarily occurring in a single root namespace and are still ongoing as of 2025-06-13 (though at a slightly reduced rate).

Background

  • Impact: High error rate affecting our error budget
  • Source:
    • Single user agent GleanSearchCrawlerBot
    • Single user ID: 25982361 (bot user: group_5892345_bot_e0274bf2bdedfe8e1726720f7526c348)
  • Success rate: ~50% overall, varies by project

Error Details

Primary error type:

Errno::ENOENT
No such file or directory @ rb_sysopen - /srv/gitlab/shared/external-diffs/merge_request_diffs/mr-ABC/diff-XYZ

Pattern:

  • Almost all API errors come from one user agent / user / root namespace
  • Errors occur when requesting Merge Request diffs
  • Missing external diff files may be the root cause

Investigation Areas

  1. Root Cause Analysis

    • Determine why external diff files are missing for these projects
    • Investigate if this is related to a migration from self-managed GitLab
    • Check if migration process missed external diff files

  2. Glean Integration

  3. Customer Communication

    • Verify if customer has intentionally enabled this integration
    • Follow up on account team inquiry in Slack
    • Questions for cutomer
      • Did they recently migrate from self-managed GitLab to GitLab.com?
      • Are they intentionally using the Glean search integration?
      • Are they experiencing any issues with merge request diffs in their projects?
      • When did they first enable/configure the Glean integration?

Next Steps

  • Immediate: Continue monitoring error rates and impact
  • Short-term: Get customer input on recent changes and Glean usage
  • Medium-term: Determine if missing diffs need restoration or if bot behavior needs adjustment

Monitoring

Edited by François Rosé