Skip to content
Snippets Groups Projects

New SCIM users not created with group SAML connection

    • Closed Issue created by Cynthia "Arty" Ng @cynthia

    Summary

    Users not previously sync'ed are being created in GitLab.com but without a SAML link to the group, unverified, and without terms of use accepted.

    Steps to reproduce

    In Azure AD,

    1. Turn on SCIM for the first time.

    or

    1. Add new users (no existing account) to SCIM setup.

    Example Project

    https://gitlab.com/groups/gitlab-silver/-/saml

    should have this user: https://gitlab.com/admin/users/GitLab37

    Turned on SCIM sync for the first time at approximately 10:30 UTC, which created the user, but unverified email, unconfirmed user, not a member of gitlab-silver.

    This is what happened:

    • this user was created: https://gitlab.com/admin/users/GitLab37 however user is unconfirmed, unverified, not a member of the group
    • I had to sign in via AzureAD as the user then it asked me to accept the TOS, which I did,
    • then it asked me to "Authorize" the app, which I did
    • now the user shows up as a member of the group
    • though still unconfirmed (since i don't even know how to access the inbox for that user)

    identity part of API response for this user:

    | identities       |                                        |
|------------------|----------------------------------------|
| provider         | "group_saml"                           |
| extern_uid       | "00377106-2df4-4815-bcd7-09f1cc4a1008" |
| saml_provider_id | 7                                      |

    For additional context, please see (internal) Slack thread

    Originally reported by customer via (internal) https://gitlab.zendesk.com/agent/tickets/145008

    What is the current bug behavior?

    User is created, but unverified email, unconfirmed user, not a member of gitlab-silver.

    What is the expected correct behavior?

    User should be created and added to the group.

    Output of checks

    GitLab.com, GitLab Enterprise Edition 12.7.0-pre 33d6233e

    Workaround

    1. Have users confirm their account. If they need a new confirmation email: https://gitlab.com/users/confirmation/new
    2. Reset their password.
    3. Sign into GitLab if not already. Accept the Terms of use if prompted.
    4. Have the affected user sign into Azure AD.
    5. Access GitLab through the list of available apps (or using the Azure app's User access URL, also known as the GitLab single sign on URL on the GitLab SAML settings page with the token).
    6. In GitLab, authorize the SSO app.
    Edited by Cynthia "Arty" Ng

    Designs

      Child items ...

      2. Activity

      • Cynthia "Arty" Ng added devopsmanage groupauthentication and authorization [DEPRECATED] scim severity3 typebug labels

        added devopsmanage groupauthentication and authorization [DEPRECATED] scim severity3 typebug labels

      • 🤖 GitLab Bot 🤖 added customer label

        added customer label

      • 🤖 GitLab Bot 🤖
        🤖 GitLab Bot 🤖 @gitlab-bot ·
        Maintainer

        Setting ~"Category:Authentication and Authorization" based on ~"group::access".

      • 🤖 GitLab Bot 🤖 added 1 deleted label

        added 1 deleted label

      • 🤖 GitLab Bot 🤖 mentioned in issue #201613 (closed)

        mentioned in issue #201613 (closed)

      • 🤖 GitLab Bot 🤖 mentioned in issue #204650 (closed)

        mentioned in issue #204650 (closed)

      • 🤖 GitLab Bot 🤖 mentioned in issue #206812 (closed)

        mentioned in issue #206812 (closed)

      • 🤖 GitLab Bot 🤖 mentioned in issue #207749 (closed)

        mentioned in issue #207749 (closed)

      • Cynthia "Arty" Ng changed the description

        changed the description

      • Cynthia "Arty" Ng
        Cynthia "Arty" Ng @cynthia ·
        Author Developer

        @jeremy since we're seeing an uptick in tickets on this, could we prioritize?

        I'm changing this to ~S2 as well since it's causing a lot of headache and confusion, and blocking some customers from properly moving over.

        Since this worked before, I'm also going to mark this as a regression . Please feel free to change any of the labels of course.

      • Cynthia "Arty" Ng added regression severity2 labels and removed severity3 label

        added regression severity2 labels and removed severity3 label

      • Ghost User
        Ghost User @ghost1 ·
        Contributor

        Also having this issue, do we have an ETA for when the work will be starting?

      • 🤖 GitLab Bot 🤖
        🤖 GitLab Bot 🤖 @gitlab-bot ·
        Maintainer

        This issue has regression without specifying which milestone introduced it. We're assuming that it's introduced in the current version 12.9.0-pre so we're labeling regression:12.9.

        Please correct it accordingly and keep regression so that we could easily search for all the regressions across version, too.

      • 🤖 GitLab Bot 🤖
        🤖 GitLab Bot 🤖 @gitlab-bot ·
        Maintainer

        This issue has regression without specifying which milestone introduced it. We're assuming that it's introduced in the current version 12.9.0-pre so we're labeling regression:12.9.

        Please correct it accordingly and keep regression so that we could easily search for all the regressions across version, too.

      • Jeremy Watson (ex-GitLab) changed milestone to %12.9

        changed milestone to %12.9

      • Jeremy Watson (ex-GitLab)
        Jeremy Watson (ex-GitLab) @jeremy-gl ·
        Contributor

        @lmcandrew: Do you think we might be able to take a look at this in this release? Could this be related to any other changes we've made recently to mapping SAML users to groups?

        • Drew Blessing
          Drew Blessing @dblessing ·
          Maintainer

          @cynthia I'm confused about the error message:

          User receives the error message: Email has already been taken and Identities extern uid has already been taken","status":409

          Why would this error be present if the user is new to GitLab.com - that is, SCIM has to create the entire user on GitLab.com? As far as I know, it's a known issue/shortcoming in our SCIM implementation that if a user already exists on .com, and has not yet signed in with SAML, that SCIM cannot associate them with the group. This is partly for security reasons so SCIM can't take over a random account without their authorization.

          But SCIM is supposed to be able to create totally new .com accounts and associate them. Just wanted to clarify what the issue is, and whether this error is present.

        • Cynthia "Arty" Ng
          Cynthia "Arty" Ng @cynthia ·
          Author Developer

          Sorry, I realize the way the description is written is not clear.

          It creates the user upon first sync, which is fine, but on subsequent syncs (default is every 40 minutes), then they will get that error. So, the error message is a bit of a red herring here.

        • Drew Blessing
          Drew Blessing @dblessing ·
          Maintainer

          Thanks for clarifying. That makes sense. I will dig into this next week. Seems like the issue is we're not getting the SAML identity created.

        • Cynthia "Arty" Ng
          Cynthia "Arty" Ng @cynthia ·
          Author Developer

          @dblessing I've updated the description with some clarification and information. The identity is created (included API identities response in description).

          if I remember correctly, in the past though, the user would be confirmed/verified and show up as a member with the SAML badge immediately. @jamedjo would be able to confirm whether that was the case

        • Please register or sign in to reply
      • Cynthia "Arty" Ng changed the description

        changed the description

      • 🤖 GitLab Bot 🤖
        🤖 GitLab Bot 🤖 @gitlab-bot ·
        Maintainer

        This issue has regression without specifying which milestone introduced it. We're assuming that it's introduced in the current version 12.9.0-pre so we're labeling regression:12.9.

        Please correct it accordingly and keep regression so that we could easily search for all the regressions across version, too.

      • 🤖 GitLab Bot 🤖 added [deprecated] Accepting merge requests label

        added [deprecated] Accepting merge requests label

        • Drew Blessing
          Drew Blessing @dblessing ·
          Maintainer

          @cynthia Do you happen to know if the SCIM provider was getting 401 Unauthorized as a response to the create/POST API call?

        • Cynthia "Arty" Ng
          Cynthia "Arty" Ng @cynthia ·
          Author Developer

          @dblessing can't imagine it would since the user was created, but I don't have a copy of the logs handy. If we want to test it using the sandbox instance, I'd have to delete the current example user on the GitLab.com side (since we can only have 1 user in the system with email address).

          Let me know if you want me to do that, and/or feel free to set up a zoom.

        • Drew Blessing
          Drew Blessing @dblessing ·
          Maintainer

          I agree it seems odd. But in my testing the user is created, but something happens immediately after that halts further provisioning and renders 401 unauthorized. It's not an issue with the SCIM token or any superficial authentication like that. I'm stumped so far. But I want to make sure I'm chasing the same issue.

        • Please register or sign in to reply
      • Drew Blessing assigned to @dblessing

        assigned to @dblessing

      • Drew Blessing
        Drew Blessing @dblessing ·
        Maintainer

        This is blocking my Okta work so I'm assigning and going after it.

      • 🤖 GitLab Bot 🤖
        🤖 GitLab Bot 🤖 @gitlab-bot ·
        Maintainer

        This issue has regression without specifying which milestone introduced it. We're assuming that it's introduced in the current version 12.9.0-pre so we're labeling regression:12.9.

        Please correct it accordingly and keep regression so that we could easily search for all the regressions across version, too.

      • 🤖 GitLab Bot 🤖 removed [deprecated] Accepting merge requests label

        removed [deprecated] Accepting merge requests label

      • Drew Blessing added regression:12.8 label

        added regression:12.8 label

        • Drew Blessing
          Drew Blessing @dblessing ·
          Maintainer

          I think I got to the bottom of this, with help from @mksionek. The problem seems to happen as a result of changes from !20791 (merged). Namely, the wrapper at !20791 (diffs) that calls current_user. In the context of the SCIM API we don't have a user because there is a group-specific SCIM token.

          For some reason, within this call to find the current_user, the entire operation hangs and the client gets a 401 Unauthorized. It's still strange to me that the whole user save call hangs but the client immediately gets a 401. Below is some output from pry where you can see a bit what's happening.

          From: /Users/dblessing/development/gdk/gitlab/ee/lib/ee/gitlab/scim/provisioning_service.rb @ line 26 EE::Gitlab::Scim::ProvisioningService#execute:

    19: def execute
    20:   return success_response if existing_member?
    21:
    22:   clear_memoization(:identity)
    23:
    24:   Rails.logger.info "Preparing to save user"
    25:   binding.pry
 => 26:   user.save!
    27:   Rails.logger.info "Finished saving user"
    28:
    29:   if user.save && member.errors.empty?
    30:     success_response
    31:   else
    32:     error_response
    33:   end
    34:
    35: rescue => e
    36:   logger.error(error: e.class.name, message: e.message, source: "#{__FILE__}:#{__LINE__}")
    37:
    38:   error_response(errors: [e.message])
    39: end

[1] pry(#<EE::Gitlab::Scim::ProvisioningService>)> user.save!


^CStopping ...

          It hung here so we started again. This time we dug a little deeper and put the pry binding in a different location and got the caller instead of saving.

          ✔ ~/development/gdk/gitlab [dblessing_okta_scim L|✚ 3…1⚑ 12]
08:50 $ gdk thin
(in /Users/dblessing/development/gdk)
(in /Users/dblessing/development/gdk)
ok: down: ./services/rails-web: 905s
Using rack adapter
Thin web server (v1.7.2 codename Bachmanity)
Maximum connections set to 1024
Listening on /Users/dblessing/development/gdk/gitlab.socket, CTRL+C to stop

From: /Users/dblessing/development/gdk/gitlab/app/models/concerns/token_authenticatable_strategies/digest.rb @ line 9 TokenAuthenticatableStrategies::Digest#find_token_authenticatable:

     5: def find_token_authenticatable(token, unscoped = false)
     6:   return unless token
     7:
     8:   binding.pry
 =>  9:   token_authenticatable = relation(unscoped).find_by(token_field_name => Gitlab::CryptoHelper.sha256(token))
    10:
    11:   if @options[:fallback]
    12:     token_authenticatable ||= fallback_strategy.find_token_authenticatable(token)
    13:   end
    14:
    15:   token_authenticatable
    16: end

[1] pry(#<TokenAuthenticatableStrategies::Digest>)> Gitlab::CryptoHelper.sha256(token)
=> "B6TrgTXmnVPuhgYfEzsn+Imy89vVYvbv/EOZrt5ACD4="
[2] pry(#<TokenAuthenticatableStrategies::Digest>)> token_authenticatable = relation(unscoped).find_by(token_field_name => Gitlab::CryptoHelper.sha256(token))
=> nil
[3] pry(#<TokenAuthenticatableStrategies::Digest>)> @options[:fallback]
=> nil
          Expand for caller stack 
          [4] pry(#TokenAuthenticatableStrategies::Digest)> caller
=> ["/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/pry_instance.rb:355:in eval'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/pry_instance.rb:355:in evaluate_ruby'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/pry_instance.rb:323:in handle_line'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/pry_instance.rb:243:in block (2 levels) in eval'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/pry_instance.rb:242:in catch'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/pry_instance.rb:242:in block in eval'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/pry_instance.rb:241:in catch'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/pry_instance.rb:241:in eval'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/repl.rb:77:in block in repl'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/repl.rb:67:in loop'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/repl.rb:67:in repl'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/repl.rb:38:in block in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/input_lock.rb:61:in __with_ownership'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/input_lock.rb:79:in with_ownership'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/repl.rb:38:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/repl.rb:13:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-0.11.3/lib/pry/pry_class.rb:192:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-byebug-3.5.1/lib/byebug/processors/pry_processor.rb:115:in block in resume_pry'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-byebug-3.5.1/lib/byebug/processors/pry_processor.rb:28:in block (2 levels) in run'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/byebug-9.1.0/lib/byebug/helpers/eval.rb:92:in allowing_other_threads'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-byebug-3.5.1/lib/byebug/processors/pry_processor.rb:28:in block in run'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-byebug-3.5.1/lib/byebug/processors/pry_processor.rb:27:in catch'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-byebug-3.5.1/lib/byebug/processors/pry_processor.rb:27:in run'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-byebug-3.5.1/lib/byebug/processors/pry_processor.rb:111:in resume_pry'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/pry-byebug-3.5.1/lib/byebug/processors/pry_processor.rb:63:in at_line'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/byebug-9.1.0/lib/byebug/context.rb:96:in at_line'",
"/Users/dblessing/development/gdk/gitlab/app/models/concerns/token_authenticatable_strategies/digest.rb:9:in find_token_authenticatable'", "/Users/dblessing/development/gdk/gitlab/app/models/concerns/token_authenticatable.rb:23:in block in add_authentication_token_field'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/metrics/instrumentation.rb:161:in block in find_by_token'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/metrics/method_call.rb:36:in measure'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/metrics/instrumentation.rb:161:in find_by_token'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/auth/auth_finders.rb:152:in find_personal_access_token'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/auth/auth_finders.rb:140:in block in access_token'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/utils/strong_memoize.rb:30:in strong_memoize'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/auth/auth_finders.rb:139:in access_token'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/auth/auth_finders.rb:99:in find_user_from_access_token'",
"/Users/dblessing/development/gdk/gitlab/ee/lib/ee/gitlab/auth/auth_finders.rb:12:in find_user_from_bearer_token'", "/Users/dblessing/development/gdk/gitlab/ee/lib/ee/api/api_guard.rb:11:in find_user_from_sources'",
"/Users/dblessing/development/gdk/gitlab/lib/api/api_guard.rb:50:in find_current_user!'", "/Users/dblessing/development/gdk/gitlab/lib/api/helpers.rb:516:in block in initial_current_user'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/auth/unique_ips_limiter.rb:19:in limit_user!'", "/Users/dblessing/development/gdk/gitlab/lib/api/helpers.rb:516:in initial_current_user'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/metrics/instrumentation.rb:161:in block in initial_current_user'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/metrics/method_call.rb:36:in measure'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/metrics/instrumentation.rb:161:in initial_current_user'", "/Users/dblessing/development/gdk/gitlab/lib/api/helpers.rb:51:in current_user'",
"/Users/dblessing/development/gdk/gitlab/ee/lib/ee/api/helpers.rb:30:in block in current_user'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/utils/strong_memoize.rb:30:in strong_memoize'",
"/Users/dblessing/development/gdk/gitlab/ee/lib/ee/api/helpers.rb:29:in current_user'", "/Users/dblessing/development/gdk/gitlab/lib/api/api.rb:48:in block (2 levels) in class:API'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/utils/lazy_attributes.rb:29:in block (2 levels) in define_lazy_reader'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/utils/strong_memoize.rb:30:in strong_memoize'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/utils/lazy_attributes.rb:27:in block in define_lazy_reader'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/application_context.rb:68:in username'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/application_context.rb:38:in block (2 levels) in to_lazy_hash'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.9.1/lib/labkit/context.rb:123:in block in expand_data'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.9.1/lib/labkit/context.rb:122:in transform_values'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.9.1/lib/labkit/context.rb:122:in expand_data'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.9.1/lib/labkit/context.rb:90:in to_h'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.9.1/lib/labkit/middleware/sidekiq/client.rb:12:in block in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.9.1/lib/labkit/context.rb:31:in with_context'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.9.1/lib/labkit/middleware/sidekiq/client.rb:11:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in block in invoke'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/sidekiq_middleware/worker_context/client.rb:14:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in block in invoke'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/sidekiq_middleware/client_metrics.rb:19:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in block in invoke'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/sidekiq_status/client_middleware.rb:10:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in block in invoke'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/client.rb:212:in process_single'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/client.rb:71:in push'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/client.rb:132:in push'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-6.0.2/lib/active_job/queue_adapters/sidekiq_adapter.rb:22:in enqueue'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-6.0.2/lib/active_job/enqueuing.rb:57:in block in enqueue'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:112:in block in run_callbacks'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:180:in block in instrument'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications/instrumenter.rb:24:in instrument'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:180:in instrument'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-6.0.2/lib/active_job/logging.rb:35:in block (2 levels) in module:Logging'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:121:in instance_exec'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:121:in block in run_callbacks'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-6.0.2/lib/active_job/logging.rb:16:in block (3 levels) in <module:Logging>'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-6.0.2/lib/active_job/logging.rb:45:in block in tag_logger'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:80:in block in tagged'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:28:in tagged'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:80:in tagged'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-6.0.2/lib/active_job/logging.rb:45:in tag_logger'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-6.0.2/lib/active_job/logging.rb:15:in block (2 levels) in <module:Logging>'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:121:in instance_exec'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:121:in block in run_callbacks'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:139:in run_callbacks'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-6.0.2/lib/active_job/enqueuing.rb:53:in enqueue'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-6.0.2/lib/active_job/configured_job.rb:15:in perform_later'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionmailer-6.0.2/lib/action_mailer/message_delivery.rb:140:in enqueue_delivery'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionmailer-6.0.2/lib/action_mailer/message_delivery.rb:94:in deliver_later'",
"/Users/dblessing/development/gdk/gitlab/app/models/user.rb:1700:in send_devise_notification'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/devise-4.7.1/lib/devise/models/confirmable.rb:121:in send_confirmation_instructions'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/devise-4.7.1/lib/devise/models/confirmable.rb:183:in send_on_create_confirmation_instructions'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:429:in block in make_lambda'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:264:in block in conditional'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:518:in block in invoke_after'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:518:in each'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:518:in invoke_after'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:136:in run_callbacks'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:827:in _run_commit_callbacks'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/transactions.rb:340:in committed!'", "/Users/dblessing/development/gdk/gitlab/config/initializers/forbid_sidekiq_in_transactions.rb:50:in block in committed!'",
"/Users/dblessing/development/gdk/gitlab/config/initializers/forbid_sidekiq_in_transactions.rb:8:in skipping_transaction_check'", "/Users/dblessing/development/gdk/gitlab/config/initializers/forbid_sidekiq_in_transactions.rb:50:in committed!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/transaction.rb:127:in commit_records'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/transaction.rb:265:in block in commit_transaction'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/2.6.0/monitor.rb:235:in mon_synchronize'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/transaction.rb:255:in commit_transaction'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/transaction.rb:293:in block in within_new_transaction'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/2.6.0/monitor.rb:235:in mon_synchronize'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/transaction.rb:278:in within_new_transaction'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:281:in transaction'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/transactions.rb:212:in transaction'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/transactions.rb:366:in with_transaction_returning_status'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/transactions.rb:319:in save!'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/suppressor.rb:48:in save!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.6.0/lib/state_machines/integrations/active_record.rb:515:in block in save!'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.6.0/lib/state_machines/integrations/active_record.rb:530:in block in around_save'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in block in run_actions'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in catch_exceptions'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in run_actions'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in run_callbacks'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in run_callbacks'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in block (2 levels) in perform'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in catch'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in block in perform'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:186:in within_transaction'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in perform'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.6.0/lib/state_machines/integrations/active_record.rb:530:in around_save'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/state_machines-activerecord-0.6.0/lib/state_machines/integrations/active_record.rb:515:in save!'",
"/Users/dblessing/development/gdk/gitlab/ee/lib/ee/gitlab/scim/provisioning_service.rb:25:in execute'", "/Users/dblessing/development/gdk/gitlab/ee/lib/api/scim.rb:132:in block (3 levels) in class:Scim'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in block (2 levels) in generate_api_method'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:182:in instrument'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in block in generate_api_method'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in block in run'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:182:in instrument'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in run'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in block in build_stack'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in call!'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in call!'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in call'",
"/Users/dblessing/development/gdk/gitlab/ee/lib/gitlab/middleware/ip_restrictor.rb:14:in block in call'", "/Users/dblessing/development/gdk/gitlab/ee/lib/gitlab/ip_address_state.rb:10:in with'",
"/Users/dblessing/development/gdk/gitlab/ee/lib/gitlab/middleware/ip_restrictor.rb:13:in call'", "/Users/dblessing/development/gdk/gitlab/lib/api/api_guard.rb:168:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource.rb:20:in _call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource/bearer.rb:8:in _call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/abstract/handler.rb:17:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in block in call!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in catch'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in call!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:60:in block in call!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in catch'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in call!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in call!'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in exec'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:121:in process_route'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:74:in block in identity'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:93:in transaction'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:72:in identity'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:57:in block in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:137:in with_optimization'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:56:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:119:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:45:in call!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:40:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/routing/mapper.rb:19:in block in class:Constraints'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/routing/mapper.rb:48:in serve'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/journey/router.rb:49:in block in serve'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/journey/router.rb:32:in each'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/journey/router.rb:32:in serve'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/routing/route_set.rb:837:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:420:in call_app!'",
"/Users/dblessing/development/gdk/gitlab/ee/lib/omni_auth/strategies/group_saml.rb:41:in other_phase'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:190:in call!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in call!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in call'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/middleware/rails_queue_duration.rb:27:in call'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/metrics/rack_middleware.rb:17:in block in call'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/metrics/transaction.rb:62:in run'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/metrics/rack_middleware.rb:17:in call'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/request_profiler/middleware.rb:17:in call'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/query_limiting/middleware.rb:17:in block in call'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/query_limiting/transaction.rb:39:in run'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/query_limiting/middleware.rb:16:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.9.1/lib/labkit/middleware/rack.rb:19:in block in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.9.1/lib/labkit/context.rb:31:in with_context'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.9.1/lib/labkit/middleware/rack.rb:18:in call'",
"/Users/dblessing/development/gdk/gitlab/ee/lib/gitlab/jira/middleware.rb:19:in call'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/middleware/go.rb:20:in call'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/etag_caching/middleware.rb:13:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/batch-loader-1.4.0/lib/batch_loader/middleware.rb:11:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/apollo_upload_server-2.0.0.beta.3/lib/apollo_upload_server/middleware.rb:20:in call'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/middleware/multipart.rb:117:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-attack-6.2.0/lib/rack/attack.rb:169:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:36:in block in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in catch'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:38:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/http/content_security_policy.rb:18:in call'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/middleware/read_only/controller.rb:53:in call'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/middleware/read_only.rb:18:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in context'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/cookies.rb:648:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/migration.rb:567:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/callbacks.rb:27:in block in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:101:in run_callbacks'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/callbacks.rb:26:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/executor.rb:14:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/better_errors-2.5.0/lib/better_errors/middleware.rb:84:in protected_app_call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/better_errors-2.5.0/lib/better_errors/middleware.rb:79:in better_errors_call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/better_errors-2.5.0/lib/better_errors/middleware.rb:57:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/debug_exceptions.rb:32:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/show_exceptions.rb:33:in call'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/middleware/basic_health_check.rb:25:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/rack/logger.rb:38:in call_app'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/rack/logger.rb:26:in block in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:80:in block in tagged'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:28:in tagged'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:80:in tagged'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/rack/logger.rb:26:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/remote_ip.rb:81:in call'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/middleware/request_context.rb:23:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/request_id.rb:27:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in call'",
"/Users/dblessing/development/gdk/gitlab/config/initializers/fix_local_cache_middleware.rb:9:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/executor.rb:14:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/lock.rb:17:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/static.rb:126:in call'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/middleware/static.rb:11:in call'", "/Users/dblessing/development/gdk/gitlab/lib/gitlab/webpack/dev_server_middleware.rb:27:in perform_request'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:57:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in call'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/metrics/requests_rack_middleware.rb:49:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/host_authorization.rb:83:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:526:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in public_send'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in method_missing'",
"/Users/dblessing/development/gdk/gitlab/lib/gitlab/middleware/release_env.rb:12:in call'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in block in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in each'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in call'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/connection.rb:86:in block in pre_process'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in catch'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in pre_process'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/connection.rb:53:in process'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/connection.rb:39:in receive_data'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run_machine'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/backends/base.rb:73:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/server.rb:162:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/controllers/controller.rb:87:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:203:in run_command'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:159:in run!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/bin/thin:6:in <top (required)>'", "/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in load'",
"/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in <top (required)>'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in load'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in kernel_load'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in run'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in exec'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in invoke_command'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in dispatch'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in dispatch'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in block in <top (required)>'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in with_friendly_errors'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in <top (required)>'",
"/Users/dblessing/.rbenv/versions/2.6.5/bin/bundle:23:in load'", "/Users/dblessing/.rbenv/versions/2.6.5/bin/bundle:23:in '"]
(END)
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/backends/base.rb:73:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/server.rb:162:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/controllers/controller.rb:87:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:203:in run_command'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:159:in run!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/bin/thin:6:in <top (required)>'", "/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in load'",
"/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in <top (required)>'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in load'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in kernel_load'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in run'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in exec'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:2: "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/backends/base.rb:73:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/server.rb:162:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/controllers/controller.rb:87:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:203:in run_command'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:159:in run!'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/bin/thin:6:in <top (required)>'",
"/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in load'", "/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in <top (required)>'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in load'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in kernel_load'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6:
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/backends/base.rb:73:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/server.rb:162:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/controllers/controller.rb:87:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:203:in run_command'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:159:in run!'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/bin/thin:6:in <top (required)>'", "/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in load'",
"/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in <top (required)>'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in load'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in kernel_load'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2: "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/backends/base.rb:73:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/server.rb:162:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/controllers/controller.rb:87:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:203:in run_command'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:159:in run!'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/bin/thin:6:in <top (required)>'",
"/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in load'", "/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in <top (required)>'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in load'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/: "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/backends/base.rb:73:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/server.rb:162:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/controllers/controller.rb:87:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:203:in run_command'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/lib/thin/runner.rb:159:in run!'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thin-1.7.2/bin/thin:6:in <top (required)>'",
"/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in load'", "/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in load'",
"/Users/dblessing/.rbenv/versions/2.6.5/bin/thin:23:in <top (required)>'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in load'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in kernel_load'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in run'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in exec'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in invoke_command'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in dispatch'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in dispatch'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in start'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in start'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in block in <top (required)>'",
"/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in with_friendly_errors'", "/Users/dblessing/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in <top (required)>'",
"/Users/dblessing/.rbenv/versions/2.6.5/bin/bundle:23:in `load'",
        • Drew Blessing
          Drew Blessing @dblessing ·
          Maintainer

          I suppose the solution is to add a SCIM-specific wrapper in https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/scim.rb. But I'm not sure exactly what attribute we will need, or where they come from.

          @reprazent Would you have ideas on what we need in the SCIM API in order to resolve this?

        • Bob Van Landuyt
          Bob Van Landuyt @reprazent ·
          Maintainer

          Aha, I see. I think the Scim api would indeed need a different context including only a user. I think you'll need to do 2 things:

          First, read the @current_user variable directly when building the context.

          diff --git a/lib/api/api.rb b/lib/api/api.rb
index b6c6ad10d30..aef2d06369a 100644
--- a/lib/api/api.rb
+++ b/lib/api/api.rb
@@ -45,7 +45,7 @@ module API
 
     before do
       Gitlab::ApplicationContext.push(
-        user: -> { current_user },
+        user: -> { @current_user },
         project: -> { @project },
         namespace: -> { @group },
         caller_id: route.origin

          This would avoid the side effects of calling current_user and would only include the value if it was set somewhere during the runtime of the request. When we do that, we'll need to make sure that all of our API request still include a correct meta.user keyword in the logs.

          After that, we'll need to make sure that the SCIM api you linked to here correctly populates an @group variable when it called find_and_authenticate_group!, that way we have the correct group in the context.

          Does that make sense?

          Edit: We'd also want a spec that exercises the problem you saw and I missed when adding the context in !20791 (merged) :grimacing:.

          Edited by Bob Van Landuyt
        • Drew Blessing
          Drew Blessing @dblessing ·
          Maintainer

          @reprazent Thanks for the insight. I'm not sure we will ever have a user in this API. It's authenticated via a static SCIM token that's specific to the group and not tied to any user. And the API is only intended to be called by a SCIM provider such as Okta, Azure, etc. Is it OK to not include a user at all in that case?

        • Bob Van Landuyt
          Bob Van Landuyt @reprazent ·
          Maintainer

          @dblessing Yes that's totally totally fine :smile:.

        • Drew Blessing
          Drew Blessing @dblessing ·
          Maintainer

          Adding the context in the SCIM API isn't necessary, and doesn't fix the problem. The issue seems to be when trying to enqueue the confirmation email. Somehow in that process the regular API context is used and it hangs. I'm not sure what's calling the regular API, though. https://gitlab.com/gitlab-org/gitlab/-/blob/40f5b9b34273af83ee875593a5c002f27e951add/app/models/user.rb#L1712

        • Drew Blessing
          Drew Blessing @dblessing ·
          Maintainer

          What's interesting is the issue doesn't occur when calling the SCIM provisioning service directly from the Rails console with the same parameters. It only occurs when using the SCIM API. So there's still something SCIM API-specific here. But I'm not sure what.

          By the way, changing to @current_user does resolve the problem. But I'm not yet sure if that's the right answer.

        • Please register or sign in to reply
      • Drew Blessing mentioned in merge request !26514 (merged)

        mentioned in merge request !26514 (merged)

      • Drew Blessing mentioned in commit 438acc93

        mentioned in commit 438acc93

      • Conor Brady
        Conor Brady @Cbrady ·

        This is blocking GitLab Gold opportunity - https://gitlab.my.salesforce.com/0064M00000V6HcN / https://gitlab.my.salesforce.com/0064M00000Wtawp cc @chrisanderton

      • Drew Blessing mentioned in commit fce74df9

        mentioned in commit fce74df9

      • Drew Blessing mentioned in commit 3061a06b

        mentioned in commit 3061a06b

      • Drew Blessing mentioned in commit 771805e6

        mentioned in commit 771805e6

      • Bob Van Landuyt closed via merge request !26514 (merged)

        closed via merge request !26514 (merged)

      • Bob Van Landuyt mentioned in commit 589d96a3

        mentioned in commit 589d96a3

      • Cynthia "Arty" Ng
        Cynthia "Arty" Ng @cynthia ·
        Author Developer

        just for reference, SMB Silver encountered the same issue for users created during this time period (internal): https://gitlab.zendesk.com/agent/tickets/150550

      • Cynthia "Arty" Ng changed the description

        changed the description

      • Liam McAndrew set weight to 3

        set weight to 3

      Please register or sign in to reply