Automatically delete unverified unconfirmed users after number of days

Proposal

Allow an admin to turn on automatic removal of unconfirmed/unverified users after a specified number of days.

Rationale

Especially on GitLab.com, we have a lot of accounts with non-existent emails (typos, misconfigured SCIM/SAML) which are never verified/confirmed.

As a result, we get a lot of Support requests. See #29279 (closed) for a long list.

Implementation

• Allow some way to turn it on/off (gitlab.rb, admin UI, or API). Preferably Admin/API.
• Need to choose default, suggest 7 days.
• Number of days should be configurable.
• consider throwing an error if < 3 days when soft confirmation is on.
• Add wording that it will auto delete if not confirmed within the set number of days (ideally, all of) the following:
  • post-registration "Almost there..." page
  • confirmation email
  • Documentation
• Some technical details in #352514 (comment 1054143903)

Workaround

• For Self-managed, admins can delete the accounts through any number of methods.
• For SaaS, GitLab Support will delete the account manually if it's the cause of another issue.

added Support Priority devopsmanage documentation featureaddition gitlab.com groupauthentication and authorization [DEPRECATED] sectiondev typefeature labels

@hsutor I know the group has a lot of priorities. Wondering if we could at least get this in the backlog and a weight?

Thinking if it's a lower weight, we might be able to find resources in Support

changed milestone to %Backlog

@cynthia This is a good idea! Is this worthy of being an admin level setting, or just a global feature in GitLab that everyone would get? Is there a reason an admin would want unverified, unconfirmed user requests to sit there any longer than, say, 90 days?

@hsutor In the MVC, we can set it to 90 days, where the number of days is non-configurable if that's easier.

in .com we'd want it on, but for self-managed, I'd want to give them the choice. So, we should have something in the gitlab.rb config file, or an admin UI setting to turn it on/off, or feature flag it? I'd leave that up to the engineer to decide which way is best.

Setting label(s) ~"Category:Authentication and Authorization" based on ~"group::authentication and authorization".

added 1 deleted label

added [deprecated] Accepting merge requests label

changed the description

added candidate15.0 label

mentioned in merge request gitlab-com/www-gitlab-com!102621 (merged)

marked this issue as related to #350498 (closed)

Is there any reason why we can't consider auto-deleting an unverified account in something less than 90 days for .com (say, 7 days)?

If the verification link expires in 24 hours, this will give the user a week to request a new confirmation email. Maybe we can also send a confirmation reminder with a new verification link 72 hours after the original one expires, or explicitly note in the confirmation email that their account will be deleted in 7 days if they fail to verify.

@hsutor for your consideration . Do we also want to pull in appsec for their thoughts?

@izzyfee

Is there any reason why we can't consider auto-deleting an unverified account in something less than 90 days for .com (say, 7 days)?

Is your suggestion to not make this configurable and just make it a general rule for GitLab.com?

If a user is provisioned but they never activate their account, and we delete the unconfirmed account, I assume the admin can provision a new one easily...but I'm just wondering if they'd understand what happened.

Probably easier than contacting GitLab support, though, to delete the unconfirmed user.

As long as we document this, it's likely fine.

@rshambhuni What do you think?

@hsutor We should definitely have a way for instances to turn it off and on but depending on effort, as a MVC, we considered not making the number of days configurable.

Ideally, you could turn it on/off and have the number of days configurable in the admin panel

@hsutor

If the verification link expires in 24 hours, this will give the user a week to request a new confirmation email. Maybe we can also send a confirmation reminder with a new verification link 72 hours after the original one expires, or explicitly note in the confirmation email that their account will be deleted in 7 days if they fail to verify.

This flow proposed by @izzyfee sounds good to me. 90 days seems like a very long time period to wait before deleting an unconfirmed/unverified user account. The above flow alerts the user enough about the upcoming action that is going to happen on their account after 7 days, if no action is taken from their side. 7 days also seems like a reasonable amount of time to wait before removing an unconfirmed/unverified user account.

We can make the number of days before account deletion configurable for an admin but I'm ok with defaulting to 7 days.

Whenever we work on this feature, we should also document this auto-deletion in 7 days behavior so users and admins are aware about it.

Let me know if you have any questions.

Thanks for your thoughts! I've updated the description with the proposal to make it 7 days by default

https://gitlab.zendesk.com/agent/tickets/292212

This user had SAML provision an email but they had an existing account they wanted to add the email to. Had to release it using our support workflow.

added customer label

removed candidate15.0 label

added priority3 label

added severity3 label

changed the description

Another ticket on this: https://gitlab.zendesk.com/agent/tickets/292020

User created an account with an email typo which we cannot fixed for free users, so the user created a new account but he cannot use the former username and his company is using a specific username format.

Internal ZD link

mentioned in issue #367823 (closed)

marked this issue as related to #367823 (closed)

As a robust and anti-abuse solution, I would suggest we add email verification before creating a user during the sign-up flow via the Web. That would resolve this issue and prevent malicious user behavior, where someone performs multiple sign-ups to reserve usernames/emails or something.

Similar to !86352 (merged).

Edited:

It wouldn't cover the case with "misconfigured SCIM/SAML"

@bdenkovych could you estimate the weight of your proposal?

We won't be able to reuse implementation from !86352 (merged) since it requires the user to exist in the database. In our case, we want to create the user only after we can confirm that the email belongs to a person who registers a new account for that email.

We haven't discussed it, but here is the way I see it should work:

• User opens /users/sign_up in their browser and fills the form:

• The user presses the "Register" button

  • frontend performs Ajax request with the email mentioned in the form to backend
  • New backend endpoint generates a token tied to the email and sends that token to that email
    • Token should expire in 5 minutes
    • (?) We need to figure out where to store the token
    • We might use Redis to store those tokens or use ActiveSupport::MessageVerifier
  • frontend appends new input value to the form, something like:

• The user fills the token they received in their inbox in the "Verification token" field and presses the "Register" button.

• backend (RegistrationsController#create) checks the token before creating the user in the database.

I would estimate this the following way:

• 8 for backend
• 5 for frontend

cc @ifarkas @dmoraBerlin @eduardosanz

FYI @jackib this might impact your team.

@bdenkovych I agree with your proposal here as it would improve security as well. However I think this impacts the devopsgrowth team so I tagged their design manager for feedback as well.

@bdenkovych are you suggesting that this would replace confirmation emails?

The problem I would foresee with your proposal is that not everyone creates an account using the sign up form. They can use one of the OAuth options or be created via SAML/SCIM.

That's why my original proposal was to auto-delete accounts if unverified for a set number of hours/days.

@dmoraBerlin Thanks, I shared with the Growth team.

As a side note, I think we are seeing enough of these types of verification interactions to document these components and flow in Pajamas so we can implement them consistently when different teams work on them. @emilybauman WDYT? Have you already considered this?

I'm not opposed to this but we'd have to think through where we then add the username portion to both net new free signups and trials.

I also want to add @jstava to this thread as I know he has plans to add challenges to the signup experience when we detect a high spam/abuse potential

As a side note, I think we are seeing enough of these types of verification interactions to document these components and flow in Pajamas so we can implement them consistently when different teams work on them. @emilybauman WDYT? Have you already considered this?

Thanks for the tag here @jackib - Just to confirm I'm on the same page, are we talking about documenting the anti-abuse verification work done with @jstava on the registration flow (email, phone number, credit card etc)? Or about something else. In this case, I have not thought to document anything yet, but it would probably be a good idea!

@emilybauman Yes, I was thinking about exactly that!

@bdenkovych, I think it's a nice improvement. We still need to find a way to handle unconfirmed users created via the API, right?

Also cc @rshambhuni (and @nmalcolm as Rohit is OOO)

We still need to find a way to handle unconfirmed users created via the API, right?

@ifarkas Not sure we need to handle that case since only administrators can create users via the API as I see https://docs.gitlab.com/ee/api/users.html#user-creation

are you suggesting that this would replace confirmation emails?

@cynthia No, I see it as an additional guard during submitting sign-up form via the web only.

The problem I would foresee with your proposal is that not everyone creates an account using the sign up form. > They can use one of the OAuth options or be created via SAML/SCIM.

Does this issue exist for OmniAuth Providers? I assume that users can create GitLab account via any provider when they have confirmed their account on provider's side.

That's why my original proposal was to auto-delete accounts if unverified for a set number of hours/days.

Yes, the issue would remain for SAML/SCIM. Implementing auto-deletion of unverified accounts, is probably only way to handle this

@bdenkovych if that's the case, I think auto-deleting would be a lower weight implementation to solve the issue?

If you agree, would it make the most sense to create a new issue for your proposal and we'll keep auto-deletion in this issue?

I do think your proposal is a good one, and we should implement it at some point.

if that's the case, I think auto-deleting would be a lower weight implementation to solve the issue?

@cynthia Yes, definitely: #352514 (comment 1052130786)

If you agree, would it make the most sense to create a new issue for your proposal and we'll keep auto-deletion in this issue? I do think your proposal is a good one, and we should implement it at some point.

Completely agree with that.

@jackib - I currently have a Figma File that's acting as the SSOT for these type of flows. I can work with Foundations on if we should be writing up rules or anything about them if we think that's the next step forward.

@emilybauman Maybe you can make an issue in the design project to discuss. If the Foundations team decides that it's too early to add these to Pajamas we could just close it out without actioning.

@jackib - I've opened up an issue here Document Verification Interactions/Methods (gitlab-org/gitlab-services/design.gitlab.com#1402) - we can discuss the best next steps with foundations

For self-managed, I love this idea.

For GitLab.com SaaS - this could come in handy, but there are risks involved. For example:

• what would happen if an account has an unverified email but they log in to that account via OAuth on a daily basis?
• would happen if we delete a user for having unconfirmed email, but it turns out that the confirmation email wasn't delivered due to an issue with our outbound email system?
• how do we warn users that their account will be deleted unless they confirm their email?
• who will handle inquiries/complaints about data that's been deleted deleted as part of a newly-implemented email confirmation requirement?

These are all really awesome questions and it got me thinking. Just a few comments from a support perspective:

what would happen if an account has an unverified email but they log in to that account via OAuth on a daily basis?

Is this possible? I always thought that a primary email always needed to be confirmed.

would happen if we delete a user for having unconfirmed email, but it turns out that the confirmation email wasn't delivered due to an issue with our outbound email system?

I wonder if it would make sense to exclude those users with a provisioned_by_group. For free users, we do have automation that is built in that would automatically remove suppressions. If we did set this to 7 days, I think that would be reasonable even for a free user.

how do we warn users that their account will be deleted unless they confirm their email?

Good idea, do we have reminder emails for confirmations? Maybe that is something that should be considered before implementing any actions. I'm 100% for anything that is more user friendly. It would be harmful if users just found out we deleted their accounts immediately without any obvious indication that we would do so.

who will handle inquiries/complaints about data that's been deleted deleted as part of a newly-implemented email confirmation requirement?

I think the best way to scope this would be to only delete unverified and unconfirmed users that have 1. Never logged in and 2. Do not have data (Not a part of any projects or groups). That should be easy to validate and would be the safest first iteration IMO.

@greg thanks for the questions. On SaaS, we would need to be careful if we ever decide to turn "soft" confirmation email back on, but I don't see that happening anytime soon, so until then, you cannot do anything with the account until it's confirmed/verified.

We can add a warning in the email when this feature is on, as well as on the "you'll need to confirm your email first" screen.

As to complaints about deletions: if the account is never confirmed and has no data aside from what was included in the registration form, and users are warned, I'm not sure what valid complaint there is? It's fairly standard for companies to delete unconfirmed accounts after a set number of days (I've even seen 24 hours), so that bots or malicious users cannot prevent legitimate users from signing up.

If the issue is that emails aren't getting delivered, then I don't see auto deletion after 7 days to change how we currently handle those cases.

I wonder if it would make sense to exclude those users with a provisioned_by_group

This will not be an issue with #238461 (closed)

If the account has no data, no activity, and it's associated with an unverified email, I don't see a problem with deleting their account data.

It's the edge cases I'm worried about.

Has email verification always been a requirement for GitLab.com accounts? If not, what happened to accounts that were created before email verification became a requirement once we made it a requirement?

What about accounts created via OAuth? If one uses Twitter OAuth to creates and login to an account, do we still require a verified email?

Do all users with unconfirmed emails have no activity? If not, how many active GitLab.com users have an unconfirmed email?

Yes, accounts always have to be confirmed before using it.

Specifically for .com, we should definitely pull a report before we let the system do any runs on it. There are two cases where accounts may have some data or contributions tied to it:

1. Created during the "soft confirmation" email period. Confirmation was required to continue using GitLab after 2-3 days.
2. Security issue where we had to set some (70?) user accounts to unconfirmed, roughly 3 years ago.

marked this issue as related to customers-gitlab-com#4559 (closed)

mentioned in issue gitlab-org/manage/general-discussion#17550

Premium customer had the need to delete two accounts provisioned by mistake. This could not be done by regular means as the provisioned email addresses were not valid. This functionality would have automatically cleaned up the accounts without user/support intervention. See Support Ticket (Internal)

@bdenkovych starting a new thread for weighing this issue assuming we go with auto-deletion.

My guess is:

• frontend 1: Editing wording on confirmation screen and/or confirmation email
• backend 3: Add auto-deletion behind feature flag; query .com to ensure we wouldn't be deleting any accounts with data (depending on the outcome of this, I could see the weight increasing)
  • How much more weight would it add if we want the number of hours/days configurable? preferably UI or API or both

frontend 1: Editing wording on confirmation screen and/or confirmation email

, I would suggest adding the note about auto-deletion of unverified users somewhere on https://docs.gitlab.com/ too.

backend 3: Add auto-deletion behind feature flag; query .com to ensure we wouldn't be deleting any accounts with data (depending on the outcome of this, I could see the weight increasing)

The first idea that comes to my mind is to create a new cron job for that. That job would be executed once an hour 0 * * * *. In that job, we would iterate through unconfirmed users that were created_at more than 7 days ago and perform Users::DestroyService for those.

I would agree that this work reflects the weight 3.

@dblessing @ifarkas Any other ideas or concerns about that?

How much more weight would it add if we want the number of hours/days configurable? preferably UI or API or both

@cynthia The estimate for adding a new config to Application Settgings is 2.

Any other ideas or concerns about that?

@bdenkovych, I agree with the proposal.

The estimate for adding a new config to Application Settgings is 2.

We allow the configuration of some of the cron workers via config/gitlab.yml, which then can be modified via charts or omnibus. I think we should keep this consistent.

@ifarkas I'm not sure whether we would be able to set arguments in the following way:

  args:
    allowed_period_since_user_creation: <%= 7.days =%>

We probably will only be able to set this duration as either an integer(7.days.to_i)

allowed_period_since_user_creation: 604800

or days:

allowed_days_since_user_creation: 7

Does this way of setting the configuration look like what you would expect?

@bdenkovych, if we go with ApplicationSettings, I think either one is fine.

If we do it consistently with other cron jobs, they are configured with standard cron expression. That would make something like:

  cron_jobs:
    delete_unconfirmed_users_worker:
      cron: "30 10 * * 6"

Edited: nevermind, that's for configuring the worker schedule, not about configuring the number of hours after which we want the deletion to kick in.

For simplicity I think it's fine to just have the config file cron option for now.

changed milestone to %15.4

changed the description

added Support Efficiency label

mentioned in issue gitlab-org/gitlab-services/design.gitlab.com#1402

removed severity3 label

removed priority3 label

This didn't get the Deliverable label for %15.4 so I am nominating it again for %15.5

changed milestone to %15.5

Setting weight based on previous thread.

cc @bdenkovych

added workflowready for development label

set weight to 3

mentioned in issue gitlab-org/quality/triage-reports#9122 (closed)

added quad-planningcomplete-no-action label