Skip to content
Snippets Groups Projects

Batch migration to set `OauthAccessToken#expires_in`

Merged Jessie Young requested to merge jy-migrate-oauth-access-tokens into master
All threads resolved!

What does this MR do and why?

  • We enforced OAuth token expiry as a breaking change for 15.0
  • We did it 2 steps:
    1. enforced expiring tokens for OAuth applications in !86362 (merged)
    2. run a background migration to set expiry for all existing tokens in !86379 (merged)
  • Both changes have been successfully deployed, and the background migration successfully set expiry for all existing tokens.
  • However, since then, the number of OAuth tokens without expiry is growing which shouldn't happen.
  • This migration is adapted from this previous MR, which was used to add an expiration to all existing OAuth access tokens: !86379 (merged)
  • We've discovered 2 areas where new tokens with expires_in: nil were created:
    • Calling OauthAccessToken.new directly does not set a default expires_in. We stopped calling OauthAccessToken.new directly via !105734 (merged)
    • Doorkeeper's refresh token flow will set the expires_in value of any refreshed tokens to match the expires_in value of the token doing the refreshing. This means that any tokens that happened to be refreshed while the previous batch migration was running could have been created with expires_in: nil. This issue was fixed via !107306 (merged)
  • Now, re-running the same migration as before should result in zero oauth_access_token records in the DB with expires_in: nil
  • Once that is confirmed, we will add a database constraint that ensures this is the case going forward.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jessie Young

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Mehmet Emin INAC
  • Mehmet Emin INAC removed review request for @minac

    removed review request for @minac

  • Jessie Young marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

    marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

  • Jessie Young added 1 commit

    added 1 commit

    Compare with previous version

  • Ghost User
  • Jessie Young added 1 commit

    added 1 commit

    Compare with previous version

  • Author Maintainer

    @minac Thank you for the review, I believe that I have addressed all of your feedback here. Back to you!

  • Jessie Young requested review from @minac

    requested review from @minac

  • Database migrations (on the main database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20221209235939 - AddTmpIndexOauthAccessTokensOnIdWhereExpiresInNull Post deploy 744.1 s :white_check_mark: +47.69 MiB
    20221209235940 - CleanupOAuthAccessTokensWithNullExpiresIn Post deploy 1.9 s :white_check_mark: +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 1

    Migration: 20221209235939 - AddTmpIndexOauthAccessTokensOnIdWhereExpiresInNull

    • Type: Post deploy
    • Duration: 744.1 s
    • Database size change: +47.69 MiB
    Calls Total Time Max Time Mean Time Rows Query
    1 742298.9 ms 742298.9 ms 742298.9 ms 0
    CREATE INDEX CONCURRENTLY "tmp_index_oauth_access_tokens_on_id_where_expires_in_null" ON "oauth_access_tokens" ("id")
    WHERE expires_in IS NULL /*application:test,db_config_name:main,line:/lib/gitlab/database/migration_helpers.rb:176:in `block in add_concurrent_index'*/
    Histogram for AddTmpIndexOauthAccessTokensOnIdWhereExpiresInNull
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 0
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 1

    Migration: 20221209235940 - CleanupOAuthAccessTokensWithNullExpiresIn

    • Type: Post deploy
    • Duration: 1.9 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 12.4 ms 12.4 ms 12.4 ms 1
    SELECT MAX("id")
    FROM "oauth_access_tokens" /*application:test,db_config_name:main,line:/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:88:in `queue_batched_background_migration'*/
    1 0.2 ms 0.2 ms 0.2 ms 1
    INSERT INTO "batched_background_migrations" ("created_at", "updated_at", "max_value", "batch_size", "sub_batch_size", "interval", "status", "job_class_name", "table_name", "column_name", "total_tuple_count", "max_batch_size", "started_at", "gitlab_schema") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) RETURNING "id" /*application:test,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    1 0.0 ms 0.0 ms 0.0 ms 0
    SELECT $1 AS one
    FROM "batched_background_migrations" WHERE "batched_background_migrations"."job_class_name" = $2 AND "batched_background_migrations"."table_name" = $3 AND "batched_background_migrations"."column_name" = $4 AND (job_arguments = $5) AND "batched_background_migrations"."gitlab_schema" = $6
    LIMIT $7 /*application:test,db_config_name:main,line:/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:79:in `queue_batched_background_migration'*/
    1 0.0 ms 0.0 ms 0.0 ms 0
    SELECT $1 AS one
    FROM "batched_background_migrations" WHERE "batched_background_migrations"."job_arguments" = $2 AND "batched_background_migrations"."job_class_name" = $3 AND "batched_background_migrations"."table_name" = $4 AND "batched_background_migrations"."column_name" = $5
    LIMIT $6 /*application:test,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    Histogram for CleanupOAuthAccessTokensWithNullExpiresIn
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Background migrations

    Background Migration: TruncateOverlongVulnerabilityHtmlTitles

    Sampled 591 batches

    Calls Total Time Max Time Mean Time Rows Query
    591 348.5 ms 38.7 ms 0.6 ms 591
    SELECT SUM("batched_background_migration_jobs"."batch_size")
    FROM "batched_background_migration_jobs" WHERE "batched_background_migration_jobs"."batched_background_migration_id" = $1 AND ("batched_background_migration_jobs"."status" IN ($2)) /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/database/background_migration/batched_migration.rb:189:in `migrated_tuple_count'*/
    1180 247.7 ms 17.2 ms 0.2 ms 1180
    INSERT INTO "batched_background_migration_job_transition_logs" ("batched_background_migration_job_id", "created_at", "updated_at", "previous_status", "next_status") VALUES ($1, $2, $3, $4, $5) RETURNING "id" /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    590 322.2 ms 13.0 ms 0.5 ms 590
    UPDATE "batched_background_migration_jobs"  SET "updated_at" = $1, "started_at" = $2, "status" = $3, "attempts" = $4
    WHERE "batched_background_migration_jobs"."id" = $5 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    591 397.1 ms 23.3 ms 0.7 ms 591
    UPDATE "batched_background_migration_jobs"  SET "updated_at" = $1, "finished_at" = $2, "status" = $3, "metrics" = $4
    WHERE "batched_background_migration_jobs"."id" = $5 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    1180 82.1 ms 16.4 ms 0.1 ms 1180
    SELECT "batched_background_migration_jobs".*
    FROM "batched_background_migration_jobs" WHERE "batched_background_migration_jobs"."id" = $1
    LIMIT $2 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    591 14.4 ms 0.4 ms 0.0 ms 2
    SELECT "vulnerabilities"."id"  FROM "vulnerabilities"  WHERE "vulnerabilities"."id" BETWEEN $1 AND $2 AND (LENGTH(title_html) > $3)
    ORDER BY "vulnerabilities"."id" ASC
    LIMIT $4 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/app/models/concerns/each_batch.rb:62:in `each_batch'*/
    2 1612.0 ms 1516.6 ms 806.0 ms 50
    UPDATE "vulnerabilities"  SET title_html = left(title_html, $1)
    WHERE "vulnerabilities"."id" BETWEEN $2 AND $3 AND (LENGTH(title_html) > $4) AND "vulnerabilities"."id" >= $5 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/background_migration/truncate_overlong_vulnerability_html_titles.rb:17:in `block in perform'*/
    2 0.1 ms 0.0 ms 0.0 ms 0
    SELECT "vulnerabilities"."id"  FROM "vulnerabilities"  WHERE "vulnerabilities"."id" BETWEEN $1 AND $2 AND (LENGTH(title_html) > $3) AND "vulnerabilities"."id" >= $4
    ORDER BY "vulnerabilities"."id" ASC
    LIMIT $5
    OFFSET $6 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/app/models/concerns/each_batch.rb:81:in `block in each_batch'*/
    2 3.4 ms 3.1 ms 1.7 ms 2
    INSERT INTO "batched_background_migration_job_transition_logs" ("batched_background_migration_job_id", "created_at", "updated_at", "previous_status", "next_status") VALUES ($1, $2, $3, $4, $5) RETURNING "id" /*application:test,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    1 3.2 ms 3.2 ms 3.2 ms 1
    UPDATE "batched_background_migration_jobs"  SET "updated_at" = $1, "started_at" = $2, "status" = $3, "attempts" = $4
    WHERE "batched_background_migration_jobs"."id" = $5 /*application:test,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    2 0.1 ms 0.0 ms 0.0 ms 2
    SELECT "batched_background_migration_jobs".*
    FROM "batched_background_migration_jobs" WHERE "batched_background_migration_jobs"."id" = $1
    LIMIT $2 /*application:test,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    Histogram of batch runtimes for TruncateOverlongVulnerabilityHtmlTitles
    Batch Runtime Count
    0 seconds - 10 seconds 591
    10 seconds - 1 minute 0
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of TruncateOverlongVulnerabilityHtmlTitles
    Query Runtime Count
    0 seconds - 0.1 seconds 4713
    0.1 seconds - 0.5 seconds 18
    0.5 seconds - 1 second 0
    1 second - 2 seconds 1
    2 seconds - 5 seconds 0
    5 seconds + 0

    Background Migration: ReExpireOAuthTokens

    Sampled 466 batches

    Calls Total Time Max Time Mean Time Rows Query
    48 31138.7 ms 1626.8 ms 648.7 ms 4800
    UPDATE "oauth_access_tokens"  SET "expires_in" = $1
    WHERE "oauth_access_tokens"."id" BETWEEN $2 AND $3 AND "oauth_access_tokens"."expires_in" IS NULL AND "oauth_access_tokens"."id" >= $4 AND "oauth_access_tokens"."id" < $5 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/background_migration/expire_o_auth_tokens.rb:13:in `block in perform'*/
    354 99885.9 ms 1427.9 ms 282.2 ms 13241
    UPDATE "oauth_access_tokens"  SET "expires_in" = $1
    WHERE "oauth_access_tokens"."id" BETWEEN $2 AND $3 AND "oauth_access_tokens"."expires_in" IS NULL AND "oauth_access_tokens"."id" >= $4 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/background_migration/expire_o_auth_tokens.rb:13:in `block in perform'*/
    932 411.1 ms 22.3 ms 0.4 ms 932
    INSERT INTO "batched_background_migration_job_transition_logs" ("batched_background_migration_job_id", "created_at", "updated_at", "previous_status", "next_status") VALUES ($1, $2, $3, $4, $5) RETURNING "id" /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    402 33.6 ms 2.9 ms 0.1 ms 48
    SELECT "oauth_access_tokens"."id"  FROM "oauth_access_tokens"  WHERE "oauth_access_tokens"."id" BETWEEN $1 AND $2 AND "oauth_access_tokens"."expires_in" IS NULL AND "oauth_access_tokens"."id" >= $3
    ORDER BY "oauth_access_tokens"."id" ASC
    LIMIT $4
    OFFSET $5 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/app/models/concerns/each_batch.rb:81:in `block in each_batch'*/
    466 191.5 ms 9.3 ms 0.4 ms 466
    SELECT SUM("batched_background_migration_jobs"."batch_size")
    FROM "batched_background_migration_jobs" WHERE "batched_background_migration_jobs"."batched_background_migration_id" = $1 AND ("batched_background_migration_jobs"."status" IN ($2)) /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/database/background_migration/batched_migration.rb:189:in `migrated_tuple_count'*/
    466 86.8 ms 9.9 ms 0.2 ms 466
    UPDATE "batched_background_migration_jobs"  SET "updated_at" = $1, "started_at" = $2, "status" = $3, "attempts" = $4
    WHERE "batched_background_migration_jobs"."id" = $5 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    932 54.3 ms 3.6 ms 0.1 ms 932
    SELECT "batched_background_migration_jobs".*
    FROM "batched_background_migration_jobs" WHERE "batched_background_migration_jobs"."id" = $1
    LIMIT $2 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    466 114.6 ms 10.9 ms 0.2 ms 466
    UPDATE "batched_background_migration_jobs"  SET "updated_at" = $1, "finished_at" = $2, "status" = $3, "metrics" = $4
    WHERE "batched_background_migration_jobs"."id" = $5 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/lib/gitlab/database.rb:333:in `block in transaction'*/
    466 228.9 ms 12.4 ms 0.5 ms 354
    SELECT "oauth_access_tokens"."id"  FROM "oauth_access_tokens"  WHERE "oauth_access_tokens"."id" BETWEEN $1 AND $2 AND "oauth_access_tokens"."expires_in" IS NULL
    ORDER BY "oauth_access_tokens"."id" ASC
    LIMIT $3 /*application:test,correlation_id:70e8a1e62cb0b059dea4368f29ffa5e5,db_config_name:main,line:/app/models/concerns/each_batch.rb:62:in `each_batch'*/
    Histogram of batch runtimes for ReExpireOAuthTokens
    Batch Runtime Count
    0 seconds - 10 seconds 466
    10 seconds - 1 minute 0
    1 minute - 2 minutes 0
    2 minutes - 3 minutes 0
    3 minutes - 5 minutes 0
    5 minutes + 0
    Histogram across all sampled batches of ReExpireOAuthTokens
    Query Runtime Count
    0 seconds - 0.1 seconds 4152
    0.1 seconds - 0.5 seconds 285
    0.5 seconds - 1 second 75
    1 second - 2 seconds 20
    2 seconds - 5 seconds 0
    5 seconds + 0

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20221110100600 - AddTempIndexOnOverlongVulnerabilityHtmlTitle Post deploy 531.2 s :white_check_mark: +168.00 KiB
    20221110100602 - QueueTruncateOverlongVulnerabilityHtmlTitles Post deploy 1.9 s :white_check_mark: +0.00 B
    20221223123019 - DeleteQueuedJobsForVulnerabilitiesFeedbackMigration Post deploy 1.9 s :warning: +0.00 B
    20230104220137 - ReindexVulnReadsOnDefaultBranchWithCommonQuery Post deploy 123.2 s :white_check_mark: +1.41 GiB
    20230104224020 - DropVulnReadsOnDefaultBranchIndex Post deploy 2.5 s :white_check_mark: -1.68 GiB

    Clone Details

    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-1657728-8949970-main 2023-01-10T13:17:44Z 2023-01-10T11:36:48Z 2023-01-11 02:15:24 +0000
    database-testing-1657728-8949970-ci 2023-01-10T13:17:44Z 2023-01-10T12:50:07Z 2023-01-11 02:15:24 +0000

    Artifacts


    Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

    Database migrations (on the ci database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20221209235939 - AddTmpIndexOauthAccessTokensOnIdWhereExpiresInNull Post deploy 2.4 s :white_check_mark: +8.00 KiB
    20221209235940 - CleanupOAuthAccessTokensWithNullExpiresIn Post deploy 1.8 s :white_check_mark: +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 1
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20221209235939 - AddTmpIndexOauthAccessTokensOnIdWhereExpiresInNull

    • Type: Post deploy
    • Duration: 2.4 s
    • Database size change: +8.00 KiB
    Calls Total Time Max Time Mean Time Rows Query
    1 38.1 ms 38.1 ms 38.1 ms 0
    CREATE INDEX CONCURRENTLY "tmp_index_oauth_access_tokens_on_id_where_expires_in_null" ON "oauth_access_tokens" ("id")
    WHERE expires_in IS NULL /*application:test,db_config_name:ci,line:/lib/gitlab/database/migration_helpers.rb:176:in `block in add_concurrent_index'*/
    Histogram for AddTmpIndexOauthAccessTokensOnIdWhereExpiresInNull
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 1
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20221209235940 - CleanupOAuthAccessTokensWithNullExpiresIn

    • Type: Post deploy
    • Duration: 1.8 s
    • Database size change: +0.00 B

    No histogram available for visualization


    Background migrations


    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20221110100600 - AddTempIndexOnOverlongVulnerabilityHtmlTitle Post deploy 3.0 s :white_check_mark: +8.00 KiB
    20221110100602 - QueueTruncateOverlongVulnerabilityHtmlTitles Post deploy 1.2 s :white_check_mark: +0.00 B
    20221223123019 - DeleteQueuedJobsForVulnerabilitiesFeedbackMigration Post deploy 1.8 s :white_check_mark: +0.00 B
    20230104220137 - ReindexVulnReadsOnDefaultBranchWithCommonQuery Post deploy 2.9 s :white_check_mark: +8.00 KiB
    20230104224020 - DropVulnReadsOnDefaultBranchIndex Post deploy 2.8 s :white_check_mark: -8.00 KiB

    Clone Details

    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-1657728-8949970-main 2023-01-10T13:17:44Z 2023-01-10T11:36:48Z 2023-01-11 02:15:24 +0000
    database-testing-1657728-8949970-ci 2023-01-10T13:17:44Z 2023-01-10T12:50:07Z 2023-01-11 02:15:24 +0000

    Artifacts


    Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

  • Jessie Young added 1314 commits

    added 1314 commits

    Compare with previous version

  • Alexandru Croitor approved this merge request

    approved this merge request

  • requested review from @DylanGriffith

  • Mehmet Emin INAC approved this merge request

    approved this merge request

  • Dylan Griffith approved this merge request

    approved this merge request

  • Dylan Griffith resolved all threads

    resolved all threads

  • Dylan Griffith enabled an automatic merge when the pipeline for 7b67c433 succeeds

    enabled an automatic merge when the pipeline for 7b67c433 succeeds

  • Dylan Griffith mentioned in commit e075b053

    mentioned in commit e075b053

  • Author Maintainer

    I just realized that this is not on prod yet. For posterity, there are currently 2,225,004 oauth access tokens on gprd with expires_in: null (source)

    Will be looking for that number to be 0 when the batched background migration finishes

  • added workflowstaging label and removed workflowcanary label

  • Author Maintainer

    Just confirmed on prod db that this ran / worked! zero records with expires_in NULL

    tsh db connect db-secondary
    psql (12.13, server 12.11 (Ubuntu 12.11-1.pgdg20.04+1))
    SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, bits: 128, compression: off)
    Type "help" for help.
    
    teleport.gprd.gitlab.net console-ro@gitlabhq_production=> select count(*) from oauth_access_tokens where expires_in is NULL;
     count
    -------
         0
    (1 row)
  • mentioned in issue #363355 (closed)

  • Jessie Young mentioned in commit 468c192b

    mentioned in commit 468c192b

  • Jessie Young mentioned in merge request !109047 (merged)

    mentioned in merge request !109047 (merged)

  • Jessie Young mentioned in commit c1ab940e

    mentioned in commit c1ab940e

  • Jessie Young mentioned in merge request !105940 (closed)

    mentioned in merge request !105940 (closed)

  • Jessie Young mentioned in commit f07e5a3c

    mentioned in commit f07e5a3c

  • Jessie Young mentioned in commit 1f952633

    mentioned in commit 1f952633

  • Jessie Young mentioned in commit b7cffddf

    mentioned in commit b7cffddf

  • Jessie Young mentioned in merge request !112765 (merged)

    mentioned in merge request !112765 (merged)

  • Jessie Young mentioned in commit 6a68043c

    mentioned in commit 6a68043c

  • mentioned in issue #369123 (closed)

  • mentioned in issue #387057 (closed)

  • mentioned in issue #414327 (closed)

  • Jon Jenkins mentioned in merge request !122682 (merged)

    mentioned in merge request !122682 (merged)

  • Jessie Young mentioned in merge request !123002 (merged)

    mentioned in merge request !123002 (merged)

  • Jessie Young mentioned in commit 14a3d3ac

    mentioned in commit 14a3d3ac

  • Jessie Young mentioned in merge request !123337 (merged)

    mentioned in merge request !123337 (merged)

  • mentioned in issue #406885 (closed)

  • Jessie Young mentioned in commit 6b27be2c

    mentioned in commit 6b27be2c

  • Jessie Young mentioned in commit 1c7284bc

    mentioned in commit 1c7284bc

  • Jessie Young mentioned in issue #415971

    mentioned in issue #415971

  • Jessie Young mentioned in merge request !126378 (merged)

    mentioned in merge request !126378 (merged)

  • Jessie Young mentioned in merge request !128842 (merged)

    mentioned in merge request !128842 (merged)

  • Please register or sign in to reply
    Loading