Skip to content

Add rake task for resetting unreadable encrypted tokens

Vladimir Shushlin requested to merge vshushlin/rake-fix-encrypted-tokens into master

What does this MR do and why?

Add rake task for resetting unreadable encrypted tokens

We have runners_tokens in DB that can not be decrypted. Automatically regenerating them is dangerous, so we provide admins with the rake task to do that.

It's an attempt to provide a solution for https://gitlab.com/gitlab-org/gitlab/-/issues/424195+ and it's influenced by already existing integrity checking task https://docs.gitlab.com/ee/administration/raketasks/check.html#verify-database-values-can-be-decrypted-using-the-current-secrets

I'll work on the documentation on the separate MR once this is close to being merged.

Screenshots or screen recordings

Local tests 
VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token ./bin/rails gitlab:doctor:reset_encrypted_tokens
I, [2023-09-26T16:20:23.230942 #88920]  INFO -- : Resetting runners_token on Project, Group if they can not be read
I, [2023-09-26T16:20:23.230975 #88920]  INFO -- : Executing in DRY RUN mode, no records will actually be updated
D, [2023-09-26T16:20:30.151585 #88920] DEBUG -- : > Fix Project[1].runners_token
I, [2023-09-26T16:20:30.151617 #88920]  INFO -- : Checked 1/29 Projects
D, [2023-09-26T16:20:30.151873 #88920] DEBUG -- : > Fix Project[3].runners_token
D, [2023-09-26T16:20:30.152086 #88920] DEBUG -- : > Fix Project[5].runners_token
D, [2023-09-26T16:20:30.152360 #88920] DEBUG -- : > Fix Project[6].runners_token
D, [2023-09-26T16:20:30.152565 #88920] DEBUG -- : > Fix Project[7].runners_token
D, [2023-09-26T16:20:30.152775 #88920] DEBUG -- : > Fix Project[9].runners_token
D, [2023-09-26T16:20:30.152975 #88920] DEBUG -- : > Fix Project[10].runners_token
I, [2023-09-26T16:20:30.152992 #88920]  INFO -- : Checked 11/29 Projects
I, [2023-09-26T16:20:30.153230 #88920]  INFO -- : Checked 21/29 Projects
D, [2023-09-26T16:20:30.153437 #88920] DEBUG -- : > Fix Project[23].runners_token
D, [2023-09-26T16:20:30.153501 #88920] DEBUG -- : > Fix Project[24].runners_token
D, [2023-09-26T16:20:30.153874 #88920] DEBUG -- : > Fix Project[29].runners_token
I, [2023-09-26T16:20:30.153882 #88920]  INFO -- : Checked 29 Projects
D, [2023-09-26T16:20:30.195929 #88920] DEBUG -- : > Fix Group[22].runners_token
I, [2023-09-26T16:20:30.196125 #88920]  INFO -- : Checked 1/19 Groups
D, [2023-09-26T16:20:30.196192 #88920] DEBUG -- : > Fix Group[25].runners_token
D, [2023-09-26T16:20:30.196456 #88920] DEBUG -- : > Fix Group[27].runners_token
D, [2023-09-26T16:20:30.196669 #88920] DEBUG -- : > Fix Group[29].runners_token
D, [2023-09-26T16:20:30.196933 #88920] DEBUG -- : > Fix Group[70].runners_token
D, [2023-09-26T16:20:30.197336 #88920] DEBUG -- : > Fix Group[78].runners_token
D, [2023-09-26T16:20:30.197557 #88920] DEBUG -- : > Fix Group[82].runners_token
I, [2023-09-26T16:20:30.197581 #88920]  INFO -- : Checked 11/19 Groups
D, [2023-09-26T16:20:30.197778 #88920] DEBUG -- : > Fix Group[86].runners_token
D, [2023-09-26T16:20:30.197990 #88920] DEBUG -- : > Fix Group[88].runners_token
D, [2023-09-26T16:20:30.198431 #88920] DEBUG -- : > Fix Group[95].runners_token
I, [2023-09-26T16:20:30.198455 #88920]  INFO -- : Checked 19 Groups
I, [2023-09-26T16:20:30.198462 #88920]  INFO -- : Done!
DRY_RUN=false VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token ./bin/rails gitlab:doctor:reset_encrypted_tokens
I, [2023-09-26T16:22:10.725916 #89509]  INFO -- : Resetting runners_token on Project, Group if they can not be read
D, [2023-09-26T16:22:17.492325 #89509] DEBUG -- : > Fix Project[1].runners_token
I, [2023-09-26T16:22:17.827000 #89509]  INFO -- : Checked 1/29 Projects
D, [2023-09-26T16:22:17.827356 #89509] DEBUG -- : > Fix Project[3].runners_token
D, [2023-09-26T16:22:17.870473 #89509] DEBUG -- : > Fix Project[5].runners_token
D, [2023-09-26T16:22:17.936484 #89509] DEBUG -- : > Fix Project[6].runners_token
D, [2023-09-26T16:22:17.965389 #89509] DEBUG -- : > Fix Project[7].runners_token
D, [2023-09-26T16:22:17.994648 #89509] DEBUG -- : > Fix Project[9].runners_token
D, [2023-09-26T16:22:18.102966 #89509] DEBUG -- : > Fix Project[10].runners_token
I, [2023-09-26T16:22:18.130017 #89509]  INFO -- : Checked 11/29 Projects
I, [2023-09-26T16:22:18.130146 #89509]  INFO -- : Checked 21/29 Projects
D, [2023-09-26T16:22:18.130258 #89509] DEBUG -- : > Fix Project[23].runners_token
D, [2023-09-26T16:22:18.167098 #89509] DEBUG -- : > Fix Project[24].runners_token
D, [2023-09-26T16:22:18.192214 #89509] DEBUG -- : > Fix Project[29].runners_token
I, [2023-09-26T16:22:18.219828 #89509]  INFO -- : Checked 29 Projects
D, [2023-09-26T16:22:18.221949 #89509] DEBUG -- : > Fix Group[22].runners_token
I, [2023-09-26T16:22:18.246488 #89509]  INFO -- : Checked 1/19 Groups
D, [2023-09-26T16:22:18.246681 #89509] DEBUG -- : > Fix Group[25].runners_token
D, [2023-09-26T16:22:18.261127 #89509] DEBUG -- : > Fix Group[27].runners_token
D, [2023-09-26T16:22:18.275577 #89509] DEBUG -- : > Fix Group[29].runners_token
D, [2023-09-26T16:22:18.289741 #89509] DEBUG -- : > Fix Group[70].runners_token
D, [2023-09-26T16:22:18.304026 #89509] DEBUG -- : > Fix Group[78].runners_token
D, [2023-09-26T16:22:18.319274 #89509] DEBUG -- : > Fix Group[82].runners_token
I, [2023-09-26T16:22:18.339047 #89509]  INFO -- : Checked 11/19 Groups
D, [2023-09-26T16:22:18.339310 #89509] DEBUG -- : > Fix Group[86].runners_token
D, [2023-09-26T16:22:18.357352 #89509] DEBUG -- : > Fix Group[88].runners_token
D, [2023-09-26T16:22:18.376493 #89509] DEBUG -- : > Fix Group[95].runners_token
I, [2023-09-26T16:22:18.396568 #89509]  INFO -- : Checked 19 Groups
I, [2023-09-26T16:22:18.396595 #89509]  INFO -- : Done!
DRY_RUN=false VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token ./bin/rails gitlab:doctor:reset_encrypted_tokens
I, [2023-09-26T16:22:46.875931 #89758]  INFO -- : Resetting runners_token on Project, Group if they can not be read
I, [2023-09-26T16:22:53.515036 #89758]  INFO -- : Checked 1/29 Projects
I, [2023-09-26T16:22:53.515318 #89758]  INFO -- : Checked 11/29 Projects
I, [2023-09-26T16:22:53.515564 #89758]  INFO -- : Checked 21/29 Projects
I, [2023-09-26T16:22:53.515645 #89758]  INFO -- : Checked 29 Projects
I, [2023-09-26T16:22:53.562587 #89758]  INFO -- : Checked 1/19 Groups
I, [2023-09-26T16:22:53.562893 #89758]  INFO -- : Checked 11/19 Groups
I, [2023-09-26T16:22:53.563137 #89758]  INFO -- : Checked 19 Groups
I, [2023-09-26T16:22:53.563148 #89758]  INFO -- : Done!

How to set up and validate locally

  1. break runner tokens for a few projects in the rails console: Project.limit(10).each{ |p| p.update_columns(runners_token_encrypted: "aaa" + rand.to_s)}
  2. execute the rake task: DRY_RUN=false VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token ./bin/rails gitlab:doctor:reset_encrypted_tokens

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vladimir Shushlin

Merge request reports