CustomersDot <> GitLab.com Access Token Short Expiry

added devopsfulfillment grouputilization sectionfulfillment typebug + 1 deleted label

@sjohnson5 @kyla adding for visibility

There is a recent change on Gitlab Remove support non OAuth tokens that don't expire (gitlab#340848 - closed). I think this is the reason.

The MR Add background migration to expiry all OAuth to... (gitlab!86379 - merged) was merged 1 week ago(May 12, 2022). This means, starting from May 12 2022, all CustomersDot <> GitLab.com Access Tokens will expire in 2 hours. (Update: it seems there is a reschedule of the migration script Remove unused batch_size in ExpireOAuthTokens (gitlab!87496 - merged), so the migration should happened after May 13 2022)

Before Remove support non OAuth tokens that don't expire (gitlab#340848 - closed), CustomersDot <> GitLab.com Access Token do not expire. This is a breaking change.

Just some note for people who are interested in: per this issue comment, the migration script took a long time running and finished on 2022-05-16.

So, the impact is:

some tokens started to expire from 2002-05-13(this is the time when the migration script started)
most tokens(>99%) expired after 2002-05-16(this is the time when the migration finished)
a small number of tokens(<0.1%) do not expire due to issue. But they will expire once the issue is fixed.

With the breaking change of Remove support non OAuth tokens that don't expire (gitlab#340848 - closed), I think we should implement token refresh on CDot. This is also suggested in the documentation https://docs.gitlab.com/ee/integration/oauth_provider.html#expiring-access-tokens:

The ability to opt-out of expiring access tokens was deprecated in GitLab 14.3 and removed in 15.0. All existing integrations must be updated to support access token refresh.

The general idea

Today we already have access_token in CDot Customer table. We need to save access_token_expires_at and refresh_token in Customer table. Every time when we use customer.access_token, we check whether the token should be refreshed. So, we can define a method Customer#access_token in Customer model.

Here is some quick outline(of what I think is required) to implement the access token refresh

1. Migrate Customer model to add new columns:

refresh_token
access_token_expires_at

2. Ensure gitlab.com API returns refresh_token and access_token_expires_at

refresh_token(to check or add)
expires_at(to check or add) Or, it can be calculated from created_at and expires_in(if these are available from API response)
access_token(already)

3. update CDot to save `refresh_token` and `access_token_expires_at` in everywhere.

There are multiple places where customer.access_token is saved. We need to save refresh_token and access_token_expires_at together.
Including trial user?

4. Implement `Customer#refresh_token!`. Changes in `app/models/customer.rb` might like (the codes are not tested)

SECONDS_BEFORE_REFRESH = 180 # 3 minutes

def access_token
    refresh_token! if should_refresh_token?

    self.attributes['access_token']
end

def should_refresh_token?
    Time.current.to_i >  access_token_expires_at - SECONDS_BEFORE_REFRESH
end

def refresh_token!
    return unless refresh_token.present?

    oauth = OmniAuth::Strategies::GitLab.new(
      nil,
      Rails.application.secrets.gitlab_app_id,
      Rails.application.secrets.gitlab_app_secret,
      client_options: { 
        site: "#{Rails.application.secrets.gitlab_url}/api/v4" 
      }
    )

    client = oauth.client

    token = OAuth2::AccessToken.new(client, nil, { refresh_token: refresh_token })

    new_token = token.refresh!

    if new_token.present?
      self.update(
        access_token: new_token.token,
        access_token_expires_at: new_token.expires_at,
        refresh_token: new_token.refresh_token
      )
    end
    # Question: do we want to log error(and report to Sentry) if token.refresh! fail?
end

cc @jameslopez @vij

BTW: I think token refresh is needed for https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/1868+ as well. So token refresh support aligns with our long-term goal.

@qzhaogitlab we are currently fielding a user request reporting that some of their tokens are becoming invalid after a while.

Looking at the source code you shared, Is it possible that a specific user can only have a single active OAuth "refresh token" at a given time?

if new_token.present?
      self.update(
        access_token: new_token.token,
        access_token_expires_at: new_token.expires_at,
        refresh_token: new_token.refresh_token
      )
    end

My understanding of this logic is that adding a new token will replace the previous one and make it invalid, but please confirm if that's correct.

My understanding of this logic is that adding a new token will replace the previous one and make it invalid, but please confirm if that's correct.

@dnldnz This is also my understanding. For the same <application_id, resource_owner_id>, generating a new token(such as in the above codes we call OAuth2::AccessToken#refresh!) will invalidate the old token.

If we check the tokens from the database(table oauth_access_tokens), we could see data like the below. Look at the revoked_at and created_at, I think: when OAuth generates a new token, it always revokes the old token first.

-[ RECORD 7 ]-----+-----------------------------------------------------------------
id                | 25
resource_owner_id | 1
application_id    | 1
token             | d6b09ddc9eeedef957cf8b4b1f2af2645ef6b39d9fbe82eb7cd2ca559813ccd7
refresh_token     | 442da43debd7bb59d87654ee1dbbac81150172e294b9a44e541fa5354ecfa3c7
expires_in        | 7200
revoked_at        | 2022-05-24 12:06:54.182882
created_at        | 2022-05-24 12:05:12.285978
scopes            | api
-[ RECORD 8 ]-----+-----------------------------------------------------------------
id                | 24
resource_owner_id | 1
application_id    | 1
token             | 868b323df711bd454bc8d9705b5c39274b43af5b4852b07c15a4fa8e7703add9
refresh_token     | 9c8082459ba83bf1c7326cc98f513ba8dcea6d42caff30ac38d8c640714ea7de
expires_in        | 7200
revoked_at        | 2022-05-24 12:05:12.280959
created_at        | 2022-05-24 12:03:23.849164
scopes            | api
-[ RECORD 9 ]-----+-----------------------------------------------------------------
id                | 23
resource_owner_id | 1
application_id    | 1
token             | b0be1d0ff6de9c6e95e053e0ddb4845b119bf42dfc98876122fb3f5e3e723702
refresh_token     | 4b9bed161400d4a3dabf46782359d63efaf08c5148895b7254e66ccfab1480ee
expires_in        | 7200
revoked_at        | 2022-05-24 12:03:23.844374
created_at        | 2022-05-24 05:35:10.692492
scopes            | api

Looking at the source code you shared, Is it possible that a specific user can only have a single active OAuth "refresh token" at a given time?

So yes, for the specific <user, application> combination, it can only have a single active refresh_token(and only a single active access_token at most) at a given time.

That also matches what our users are reporting. I'll look for a good place to add this to the documentation, thanks for confirming!

@ofernandez2 @doniquesmit @csouthard wdyt which team should own this? I'm actually leaning more towards groupfulfillment platform rather than grouputilization, given the "authentication" theme and fix for this.

For the PMs, I'm also not 100% sure of the urgency (see Support and Customer impact), but this is also a pre-req for https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/1868+ (just blocking that issue now on this) so we may want to consider scheduling this relatively soon cc @ofernandez2 @tgolubeva

given the "authentication" theme and fix for this

groupfulfillment platform makes sense to me @jameslopez

Agree on groupfulfillment platform and on scheduling soon. Not sure what else is on the group's queue, will need to TAL.

I'm adding this one to the current milestone, given that this is affecting more parts than we thought - as Jerome's pointed out

If you happen to see some customer's access token is NOT expired within 2 hours, it could be due to this issue OAuth tokens without expiry in the DB (gitlab#363355 - closed).

Such some customers should be a pretty small percentage. Roughly estimation is: about 0.035%(985 / (2825278 + 985)). These customers's access tokens still have expires_in: nil.

I am sure OAuth tokens without expiry in the DB (gitlab#363355 - closed) will be fixed soon. So, after the fix(of OAuth tokens without expiry in the DB (gitlab#363355 - closed)), all customer's access token should have expires_in: 7200. --- I am not sure: is this good news or bad news?

added groupfulfillment platform label and removed grouputilization label

Setting label(s) ~"Category:Fulfillment Platform" based on groupfulfillment platform.

added 1 deleted label

mentioned in epic gitlab-com/support&197 (closed)

This looks to be related to some users experiencing issues with starting Trials if they have an expired token https://gitlab.com/gitlab-org/gitlab/-/issues/364119#note_968352321

Added severity2 label to match the severity on the related issue.

added severity2 label

changed milestone to %15.1

set weight to 3

removed 1 deleted label

mentioned in issue gitlab-com/Product#4044 (closed)

assigned to @qzhaogitlab

Copy the suggested options from https://gitlab.com/gitlab-org/gitlab/-/issues/364119#note_968352321

What do we have

Users with expired access token unable to apply for a trial or any request to the GitLab API as their token expired.

Solution

We are dealing with 2 problems here:

lack of ability to refresh the access token from the CustomersDot side

customers with expired tokens who can't access the GitLab API on behalf of CustomersDot application.

Solution for 1st problem: When the user authenticates on CustomersDot via GitLab OAuth we should store refresh_token and use it to fetch the new access token if it is expired.

Solution for 2nd problem: Because access tokens that we store on the CustomersDot are no longer valid we have to nullify them. I propose to write a simple Rake task that will go through all users on CustomersDot side, make a call to a special GitLab API in order to receive an information about access token: if token invalid the nullify it.

I would choose to take these actions:

First, implement solution for problem 2: nullify invalid access_token to quickly alleviate customer impact in short term.
And then, implement solution for problem 1: add token refresh support for the long term. The solution could follow #4345 (comment 957695713).

added priority2 label

added workflowin dev label

Solution for 2nd problem: Because access tokens that we store on the CustomersDot are no longer valid we have to nullify them. I propose to write a simple Rake task that will go through all users on CustomersDot side, make a call to a special GitLab API in order to receive an information about access token: if token invalid the nullify it.

MR https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/4881+ created nullify rake task. Run the rake task on production server:

95080 customers' access_tokens are invalid
- 87637 access_tokens nullified successfully
- 7443 access_tokens failed to nullify, due to the customers being in rejected countries.

These are the 12 rejected countries: ["Cuba", "Hong Kong", "United States Minor Outlying Islands", "Belarus", "Virgin Islands (U.S.)", "Russian Federation", "Macao", "Iran (Islamic Republic of)", "Korea, Democratic People's Republic of", "China", "Syrian Arab Republic", "Sudan"].

I have a question: for these rejected countries' customers, are they still allowed to login Gitlab.com and CustomersDot portal to view/manage their existing subscriptions? If yes, we should nullify their access_token the same way as customers from other countries? @tgolubeva maybe you know the answer?

Per this slack discussion: for these rejected countries, existing customers are also blocked by default and they are redirected to Sales for manual review..

I think it is not urgent/important to nullify access_token for the 12 rejected countries.

I will focus on enabling access_token refresh. After token refresh is supported, we do not need to worry about the nullify access_token anyway. Please correct me if you have concerns. Thanks.

Adding the Support Priority and Support Efficiency labels since this is kind of a blocker for some of our workflows. Most specifically, the team is blocked from "extending" subscriptions (which creates a trial) since the trial creation relies on the Customer's UID, which isn't working because of this bug.

note for console users needing to workaround it: pass is_admin=true to BaseTrialService

ETA:

here is an example of all the steps we have to jump thru to support customers during this

added Support Efficiency Support Priority labels

changed milestone to %15.2

added backend label

mentioned in issue gitlab-com/www-gitlab-com#13550 (closed)

changed the description

mentioned in issue #3698 (closed)

There seems to be some weird thing going on with CustomersDot. From our admin view, it seems the account isn't linked but when the customer use the Sign in with your GitLab.com account, it takes the customer to the linked account anyway.

More info in this ticket: https://gitlab.zendesk.com/agent/tickets/300112 (GitLab internal)

Please note that at the time we ask the customer for screen recording, we don't see any account link. The customer provide the screenshot and we still don't see any account linked.

Thanks @rotanak for this comment!

The facts are:

when link Gitlab.com account or Sign in with your GitLab.com account, it will set a new acccess_token. The new access_token will be valid within 2 hours.
after 2 hours the access_token expired. Now if we login CustomersDot using username + password(this could be customer login using username/password, or admin impersonates customer login), this will set access_token: nil and thus the previously linked account is unlinked again.

From our admin view, it seems the account isn't linked

As said above, it this happens after access_token expired(expire in 2 hours), it will unlink the account.

but when the customer use the Sign in with your GitLab.com account, it takes the customer to the linked account anyway.

As said above, this will set a new access_token and link the account. This access_token will be valid for 2 hours.

The customer provide the screenshot and we still don't see any account linked. Does this refer to this screen recording? https://gitlab.zendesk.com/attachments/token/w1H17McYB19r54JSrClhDGHeU/?name=GitLab+Subscription.mp4.

In this screen recording, it does not check account details. Or you might mean to check from admin view again? As said above, if login CDot using username/password(or an admin impersonates this customer login) after access_token expired, it will unlink the account.

OTOH, the ticket(https://gitlab.zendesk.com/agent/tickets/300112)'s remaining issue is: when user Sign in with GitLab.com account, it logins another previously linked account. Because there were two customer accounts had ever been linked to the same Gitlab.com account, this confused the system. This seems like a scenario CustomersDot may not have handled correctly? I would prefer to create another issue for this, as this seems not related to access_token.

OTOH, the ticket(https://gitlab.zendesk.com/agent/tickets/300112)'s remaining issue is: when user Sign in with GitLab.com account, it logins another previously linked account. Because there were two customer accounts had ever been linked to the same Gitlab.com account, this confused the system. This seems like a scenario CustomersDot may not have handled correctly? I would prefer to create another issue for this, as this seems not related to access_token.

Regarding this scenario, I think it is due to this issue https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/4280+

I see you mentioned in the other issue. You're right, it's possible that the problem exist for admin view because the account still have uuid linked. I'll go ahead and unlink it for them via console.

it's possible that the problem exist for admin view because the account still have uuid linked. I'll go ahead and unlink it for them via console.

I believe if we set the uid: nil, provider: nil, access_token: nil, it should work.

The reason why the user is directed to the unexpected Customer account is this code:

    def init_customer
      Customer.with_oauth_info(params[:uid], params[:provider]).last || Customer.new
    end

When signin with GitLab.com account, it will find the existing CustomersDot customer by uid and provider, and if there are multiple one, it returns the last(order by customer.id) one.

That make sense now. I've set the uid: nil, provider: nil, access_token: nil for the customer and update them.

Thanks a lot!

added customer label

CustomersDot <> GitLab.com Access Token Short Expiry

Summary

Support impact

Customer impact

Workarounds

Designs

Child items ...

Activity

The general idea

Here is some quick outline(of what I think is required) to implement the access token refresh

1. Migrate Customer model to add new columns:

2. Ensure gitlab.com API returns refresh_token and access_token_expires_at

3. update CDot to save `refresh_token` and `access_token_expires_at` in everywhere.

4. Implement `Customer#refresh_token!`. Changes in `app/models/customer.rb` might like (the codes are not tested)

What do we have

Solution

CustomersDot <> GitLab.com Access Token Short Expiry

Summary

Support impact

Customer impact

Workarounds

Activity

The general idea

Here is some quick outline(of what I think is required) to implement the access token refresh

1. Migrate Customer model to add new columns:

2. Ensure gitlab.com API returns refresh_token and access_token_expires_at

3. update CDot to save refresh_token and access_token_expires_at in everywhere.

4. Implement Customer#refresh_token!. Changes in app/models/customer.rb might like (the codes are not tested)

What do we have

Solution

3. update CDot to save `refresh_token` and `access_token_expires_at` in everywhere.

4. Implement `Customer#refresh_token!`. Changes in `app/models/customer.rb` might like (the codes are not tested)