Skip to content
Snippets Groups Projects

Optimize merge request refresh by using the database to check commit SHAs

Merged Stan Hu requested to merge sh-optimize-mr-commit-sha-lookup into master
All threads resolved!

Previously for a given merge request, we would:

  1. Create the array of commit SHAs involved in the push (A)
  2. Request all merge request commits and map the SHA (B)
  3. Reload the diff if there were any common commits between A and B

We can avoid additional database querying and overhead by checking with the database whether the merge request contains any of the commit SHAs.

Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/53213

In https://gitlab.com/gitlab-org/gitlab-ce/issues/49703#note_113602024, we see that a push to the staging www-gitlab-com resulted in 204 queries for SELECT * FROM merge_request_diff_commits.

Edited by Stan Hu

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Stan Hu added 1 commit

    added 1 commit

    • 296087e3 - Optimize merge request refresh by using the database to check commit SHAs

    Compare with previous version

  • Stan Hu resolved all discussions

    resolved all discussions

  • Stan Hu added 1 commit

    added 1 commit

    • dbc03ce3 - Optimize merge request refresh by using the database to check commit SHAs

    Compare with previous version

  • Stan Hu changed the description

    changed the description

  • assigned to @DouweM

  • Stan Hu resolved all discussions

    resolved all discussions

  • Stan Hu mentioned in merge request !22725 (merged)

    mentioned in merge request !22725 (merged)

  • Douwe Maan approved this merge request

    approved this merge request

  • merged

  • Thanks @stanhu!

  • Douwe Maan mentioned in commit 4d3ff28a

    mentioned in commit 4d3ff28a

  • Author Owner

    With a simulated push on the staging www-gitlab-com repository with https://gitlab.com/gitlab-org/gitlab-ce/issues/49703#note_113329681:

    • Memory usage went down from 150 MB to 120 MB
    • Full ActiveRecord model loads dropped from 1,190 SQL queries to 969, not counting SELECT 1 and SELECT COUNT(*) etc.:
    Total allocated: 119999038 bytes (1609014 objects)
    Total retained:  3257369 bytes (14142 objects)
    
    allocated memory by gem
    -----------------------------------
      67553020  activerecord-4.2.10
      13792697  activesupport-4.2.10
       9769264  activemodel-4.2.10
       6840365  psych
       3634513  arel-6.0.4
       3615746  grpc-1.15.0-x86_64-linux
       2835456  state_machines-0.5.0
       2715759  gitlab-rails/lib
       1888623  other
       1162802  gitlab-rails/app

    SQL queries:

    $ cat profile_sql11.txt | sed -e "s/.*DEBUG -- ://g" | grep Load | sed -e "s/Load.*//g" | sort | uniq -c  
        339    MergeRequestDiff 
        235    Project 
        175    User 
        168    MergeRequest::Metrics 
         32    ForkNetwork 
          7    MergeRequest 
          7    MergeRequestDiffCommit 
          1    GitlabIssueTrackerService 
          1    License 
          1    Namespace 
          1    ProjectFeature 
          1    ProjectRepositoryState 
          1    Route 

    profile_data.zip

    Edited by Stan Hu
  • Please register or sign in to reply
    Loading