Skip to content

GitLab Next

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
GitLab
GitLab
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 34,832
    • Issues 34,832
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
    • Iterations
  • Merge Requests 1,218
    • Merge Requests 1,218
  • Requirements
    • Requirements
    • List
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Security & Compliance
    • Security & Compliance
    • Dependency List
    • License Compliance
  • Operations
    • Operations
    • Metrics
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Code Review
    • Insights
    • Issue
    • Repository
    • Value Stream
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • GitLab.org
  • GitLabGitLab
  • Issues
  • #34086

Closed
Open
Opened Oct 15, 2019 by Dylan Griffith@DylanGriffith🔴Maintainer

Buffer Writes to ElasticSearch and use the Bulk API

Problem

At present we are writing to ElasticSearch for every change to every model. These very frequent writes to ElasticSearch cause performance issues at scale.

It seems very plausible that before we reach all of GitLab.com scale we're likely to need to buffer/batch these updates and use the bulk API.

It's also plausible that our current implementation is not adequate for our largest self-hosted customers.

Solution

Summarised from #34086 (comment 230326472)

  1. Updates to models go into sorted sets in Redis rather than queued as sidekiq jobs.
  2. A periodic sidekiq worker runs every 1 minute and pops the top 1k update jobs from the sorted set
  3. The 1k update jobs are transformed into a bulk update for Elasticsearch
  4. Failed updates are retried

Likely follow up issues

  1. Timeouts for updates are retried in increasingly smaller batches similar to #195774 (closed) . This may be preferable to do in a follow up issue since running this every 1 minute likely won't build up large payloads (ie. we're unlikely to hit 1k jobs in a minute until we roll this out more widely)
  2. #205148 (closed)
  3. #205178

Feature flag

This functionality is gated behind a default-off feature flag: :elastic_bulk_incremental_updates . An issue to roll out the feature flag and subsequently remove the FF is at: #208717 (closed)

Edited Mar 02, 2020 by Nick Thomas
Assignee
Assign to
12.9
Milestone
12.9 (Past due)
Assign milestone
Time tracking
None
Due date
None
Reference: gitlab-org/gitlab#34086