`ci_runner` plaintext token `nil` after upgrade to 11.6
Zendesk: https://gitlab.zendesk.com/agent/tickets/113430
A customer upgraded from 11.5.5 to 11.6.x. After the upgrade their runners could no longer authenticate and they got Forbidden
errors in runner logs. On investigation we found that the ci_runner
entries in the database had their token
values cleared out (nil
).
If they re-register runners they begin working again. Similarly, we note that the new entries for ci_runner
have the plaintext token
value.
If I understand the TokenAuthenticatable
classes correctly, the specification in Ci::Runner
for add_authentication_token_field :token, encrypted: true, migrating: true
should mean that we keep both plaintext and encrypted values for now - that's what migrating: true
means. And when migrating: true
is set, we override the accessor methods to prefer the plain token
attribute. When migrating
is not true we override the methods to use encrypted_token
instead. The runners can't authenticate because Ci::Runner.find_by_token('<token>')
doesn't return a match. And because migrating: true
is set, we don't try Ci::Runner.find_by_encrypted_token('<token>')
which, if I understand correctly, would actually work.
Something must have gone wrong in the runner migration that removed these plaintext values. The various migrations were part of https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23412. Can anyone spot an edge case here that would have caused the token
to become nil
?
@grzesiek @nick.thomas Any ideas on this? You both worked on the MR mentioned above.
This isn't a big concern for the customer mentioned as they already went and re-registered all runners. But we want to be sure we don't have a widespread problem as more customers upgrade.
cc/ @dstanley