Queue Elasticsearch indexing after transactions commit

What does this MR do and why?

Two callbacks queue an issue or merge request for advanced search indexing before their database transaction commits. If an indexing run (Search::Elastic::IndexBulkCronWorker) reads the record in that window, it indexes the pre-commit data and removes the item from the queue. The document stays wrong until the record changes again.

  • Deleted labels: LabelLink#maintain_target_elasticsearch! runs in after_destroy. label.destroy! deletes every link to the label in one transaction, so for a widely used label the window is long enough for an indexing run to read the old labels. Advanced search filtered by that label keeps returning the affected issues and merge requests.
  • Upvotes: Issue#update_upvotes_count runs from AwardEmoji's after_save and after_destroy callbacks and queued the issue before the new upvotes_count committed.

Both now defer track! with current_transaction.after_commit, so nothing is queued until the commit, and nothing is queued on rollback. LabelLink cannot use after_destroy_commit because it is BulkInsertSafe, which forbids commit callbacks.

update_upvotes_count now passes updated_attributes: ['upvotes_count'] instead of the default previous_changes.keys: super calls lock!, which reloads the record and clears its dirty state, and the keys it held came from the issue's last save, which had already queued its own update.

How these were found

While investigating day-time growth of the incremental indexing queue, I audited every caller of Elastic::ProcessBookkeepingService.track! and every model method that queues indexing. These were the only two that queue before commit; all other callers use after_commit callbacks, current_transaction.after_commit, or run after their transaction finishes. Production logs showed no Elasticsearch indexing workers enqueued inside transactions in the last 24 hours.

Reading the batch from replicas, which is being considered to reduce primary load during indexing, would widen these windows from milliseconds to the length of a whole indexing run, so these callers need fixing first.

The stale documents cannot be observed in logs, because they look like a normal successful index. The updated specs reproduce the race: they fail on the previous code.

References

None.

Screenshots or screen recordings

Not applicable: no UI changes.

How to set up and validate locally

Requires advanced search with Elasticsearch running in GDK, and the indices created (bundle exec rake gitlab:elastic:create_empty_index). The script below deletes a label applied to an issue and upvotes a second issue, then repeats both inside a transaction that rolls back. When track! queues the issue, it starts an indexing run straight away on a separate database connection (Elastic::ProcessBookkeepingService.new.execute, as IndexBulkCronWorker does), while the transaction is still open. After the transaction, it reads the issue's document back from the work_items index and compares it with the database.

Rails console output on master:

before: Elasticsearch document label_ids=["29"]
label deleted (Issue#669):
  queued with open_transactions=1
  Elasticsearch document: label_ids=["29"]
  database:               label_ids=[]
upvoted (Issue#670):
  queued with open_transactions=1
  Elasticsearch document: upvotes=0
  database:               upvotes=1
label deleted, transaction rolled back (Issue#671):
  queued with open_transactions=1
  Elasticsearch document: label_ids=["30"]
  database:               label_ids=["30"]
upvoted, transaction rolled back (Issue#672):
  queued with open_transactions=1
  Elasticsearch document: upvotes=0
  database:               upvotes=0
  • Label deleted and upvoted: both transactions committed, but the indexing run read the uncommitted state. The stored document still has the deleted label in label_ids, and upvotes is 0 while the database has 1. It stays wrong until the issue changes again.
  • Rolled back: the issue is still queued and indexed for a change that never happened. The document happens to stay correct here, because the run read the state from before the change.

Rails console output on this branch:

before: Elasticsearch document label_ids=["31"]
label deleted (Issue#673):
  queued with open_transactions=0
  Elasticsearch document: label_ids=[]
  database:               label_ids=[]
upvoted (Issue#674):
  queued with open_transactions=0
  Elasticsearch document: upvotes=1
  database:               upvotes=1
label deleted, transaction rolled back (Issue#675):
  queued: no
  Elasticsearch document: label_ids=["32"]
  database:               label_ids=["32"]
upvoted, transaction rolled back (Issue#676):
  queued: no
  Elasticsearch document: upvotes=0
  database:               upvotes=0

track! now runs after the commit (open_transactions=0), so the indexed document matches the database, and a rolled-back transaction queues nothing.

Console script

Change project to any project indexed by advanced search. Use a fresh console for each run, because the script overrides track! for the rest of the session. It runs the indexing queue inline, so it also indexes anything else already queued.

ActiveRecord::Base.logger = nil # keep the console output readable
user = User.find_by_username!('root')
project = Project.find_by_full_path('toolbox/gitlab-smoke-tests') # any project indexed by advanced search
client = Gitlab::Search::Client.new
index = Search::Elastic::References::WorkItem.index
watched = nil
queued = []

# Index the queue now, the way IndexBulkCronWorker does.
run_indexer = -> { Elastic::ProcessBookkeepingService.new.execute }

# The issue's document as stored in Elasticsearch.
indexed_document = lambda do |issue|
  client.indices.refresh(index: index)
  hit = client.search(index: index, body: { query: { term: { id: issue.id } } })['hits']['hits'].first
  hit && hit['_source']
end

# Queue as usual, then start an indexing run on a separate connection straight away, which is
# what happens when IndexBulkCronWorker runs before the transaction commits.
Elastic::ProcessBookkeepingService.singleton_class.prepend(Module.new do
  define_method(:track!) do |*items|
    super(*items).tap do
      next unless watched && items.grep(Issue).any? { |item| item.id == watched }

      queued << ApplicationRecord.connection.open_transactions
      Thread.new { ApplicationRecord.connection_pool.with_connection { run_indexer.call } }.join
    end
  end
end)

new_issue = lambda do |label|
  Issues::CreateService.new(container: project, current_user: user,
    params: { title: 'track! race', label_ids: Array(label&.id) }).execute[:issue].tap do
    run_indexer.call # index the new issue before the test starts
  end
end

report = lambda do |name, issue, field, database_value|
  puts "#{name} (Issue##{issue.id}):"
  puts queued.empty? ? '  queued: no' : "  queued with open_transactions=#{queued.join(', ')}"
  puts "  Elasticsearch document: #{field}=#{indexed_document.call(issue)&.dig(field).inspect}"
  puts "  database:               #{field}=#{database_value.inspect}"
end

label_params = { title: "track-race-#{SecureRandom.hex(3)}", color: '#336699' }
label = Labels::CreateService.new(user, label_params).execute(project: project)
issue = new_issue.call(label)
puts "before: Elasticsearch document label_ids=#{indexed_document.call(issue)['label_ids'].inspect}"
queued.clear
watched = issue.id
label.destroy!
watched = nil
report.call('label deleted', issue, 'label_ids', issue.reload.label_ids.map(&:to_s))

issue2 = new_issue.call(nil)
queued.clear
watched = issue2.id
AwardEmojis::AddService.new(issue2, AwardEmoji::THUMBS_UP, user).execute
watched = nil
report.call('upvoted', issue2, 'upvotes', issue2.reload.upvotes_count)

label_params = { title: "track-race-#{SecureRandom.hex(3)}", color: '#336699' }
label3 = Labels::CreateService.new(user, label_params).execute(project: project)
issue3 = new_issue.call(label3)
queued.clear
watched = issue3.id
ApplicationRecord.transaction do
  label3.destroy!
  raise ActiveRecord::Rollback
end
watched = nil
report.call('label deleted, transaction rolled back', issue3, 'label_ids', issue3.reload.label_ids.map(&:to_s))

issue4 = new_issue.call(nil)
queued.clear
watched = issue4.id
ApplicationRecord.transaction do
  AwardEmojis::AddService.new(issue4, AwardEmoji::THUMBS_UP, user).execute
  raise ActiveRecord::Rollback
end
watched = nil
report.call('upvoted, transaction rolled back', issue4, 'upvotes', issue4.reload.upvotes_count)

[issue, issue2, issue3, issue4].each(&:destroy!)
label3.destroy!
run_indexer.call

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Ravi Kumar

Merge request reports

Loading
Loading