GitLab.org / GitLab Community Edition

Changed title: 500 error: OpenSSL::Cipher::CipherError (bad decrypt) despite /var/opt/gitlab/gitlab-rails/etc/secrets.yml and /etc/gitlab/gitlab-secrets.json present and unchanged → 500 error: OpenSSL::Cipher::CipherError (bad decrypt) despite /var/opt/gitlab/gitlab-rails/etc/secrets.yml and /etc/gitlab/gitlab-secrets.json present and not manually changed

So what do you guys usually do when this happens? Wipe everything, and manually re-create each single one of your projects? 😕 I'm really not sure what to do about this.. is this an error that is recoverable in any way? Do I just need to delete some of the data, or is it best to throw away the entire instance?

Ok, this seems to be an exact duplicate of #16999 because simply running this command in the docker container fixed it (suggested in that ticket as a solution):

gitlab-rails runner  "Project.where.not(import_url: nil).each { |p| p.import_data.destroy if p.import_data }"

Please note to reproduce this I simply upgraded the omnibus container, so no backup/restore or anything involved... I hope you can eventually find a solution for this weird problem.

@JonasT1 please see this documentation for Omnibus: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-11-stable/doc/update/README.md#migrating-legacy-secrets

Basically this error means your db_key_base secret has changed. In addition to the error you've seen, this will also mean any CI secret variables you've set will be unavailable.

You mentioned running this in a docker container, so how are you installing this? I tried to clarify at https://github.com/sameersbn/docker-gitlab/pull/860 what these secrets are and where they go.

Added awaiting feedback label

@smcgivern I simply shut down the 8.10 container, and recreated the 8.11 container on the unchanged volume data. I did nothing else.

@JonasT1 in that case I think you will need to move the secrets as described in the first link I gave.

Isn't the omnibus container supposed to update itself? "For most installations, this process should be transparent as the 8.11 and higher packages will try to migrate the existing secrets to the new key names." seems to suggest that.

@JonasT1 that's true for the packages (which will run gitlab-ctl reconfigure).

However, my understanding is that the docker container has static inputs, and so the config files will need to be updated manually.

The configs are usually in read-write volumes, so there is no technical reason why they couldn't be updated I think.

Wouldn't it be better if the config had a schema version, and there was a proper verbose error when the instance launches? Just throwing a 500 with a random backtrace is a bit convoluted for such an obvious mistake..

The Rails app will try to update itself, and the Omnibus package will try to update itself. And both will warn if they find an invalid config. I'm not sure what the docker container should do. /cc @marin

Added Build and removed awaiting feedback labels

Well the expected behavior would have been that the container doesn't start, and instead launches some sort of basic error site that tells me the config needs updating. I think that is how Plone and other CMS usually do it.

Starting and then having some weird internal error on only some pages sounds super suboptimal, and just erroring out on the command line would be a bit ugly as well (but still better than the current behavior).

Changed title: 500 error: OpenSSL::Cipher::CipherError (bad decrypt) despite /var/opt/gitlab/gitlab-rails/etc/secrets.yml and /etc/gitlab/gitlab-secrets.json present and not manually changed → Updating the omnibus container to 8.11 launches without error, but then runs into OpenSSL::Cipher::CipherError instead of warning about an outdated config right at launch

Changed title: Updating the omnibus container to 8.11 launches without error, but then runs into OpenSSL::Cipher::CipherError instead of warning about an outdated config right at launch → Updating the omnibus container 8.x-> 8.11 launches without error, but then runs into OpenSSL::Cipher::CipherError instead of warning about an outdated config right at launch

Changed title: Updating the omnibus container 8.x-> 8.11 launches without error, but then runs into OpenSSL::Cipher::CipherError instead of warning about an outdated config right at launch → Updating the omnibus container 8.x-> 8.11 launches without error, but then runs into OpenSSL::Cipher::CipherError instead of informing about an outdated config right at launch

#16999

Our Architect was able to resolve with :

gitlab-rails runner "Project.where.not(import_url: nil).each { |p| p.import_data.destroy if p.import_data }"

The restore had already the db_key_base on the new instance but we needed to run the above command.

We can go to all the repos now.

Well that sounds like a weird thing to be required to be done to fix it, so I guess there is some sort of migration script that should have handled it but it has a bug?

http://docs.gitlab.com/omnibus/docker/ is pretty clear about the upgrade just being removing the container and instantiating the new one on the existing data.

So this is definitely a documentation or implementation bug.

I had same problem. After upgrade gitlab today. I see 500 on some project, other not. When I run gitlab-ctl tail, I saw same your error on broken project main page. run gitlab-rails runner "Project.where.not(import_url: nil).each { |p| p.import_data.destroy if p.import_data }" resolved

@smcgivern I would suggest this is marked as a bug in the omnibus container update process, since the docs clearly state this is supposed to work without any manual fixes or other sorts of interventions. Also it seems to partially work since the config is rewritten to the new format, but then something in the project data appears to be not updated correctly resulting in this error.

@marin what is the update process for docker installs supposed to be for this?

And just to reiterate, @notteen et al: you are deleting the encrypted data from the DB, and that only works if you don't care about that data. If you have CI secret variables encrypted using the same key, they will no longer be available.

I'm sorry that this isn't clear, as I'm not really aware of what we normally do for docker installations in cases like this. https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-11-stable/doc/update/README.md#migrating-legacy-secrets describes which secret key names changed - there is no need to guess or delete data.

@smcgivern as for the docs, check out the section "Upgrade GitLab to newer version" in http://docs.gitlab.com/omnibus/docker/ which would very badly need an update if this was supposed to be upgraded/migrated manually.

@JonasT1 I understand that, which is why I'm asking @marin, who will know what the process should be 🙂

Hi! I'm using gitlab from source code~ I'm also able to reproduce the error

Started GET "/kei/thrallMod" for 119.95.50.118 at 2016-10-01 08:44:20 +0800
Processing by ProjectsController#show as HTML
Parameters: {"namespace_id"=>"kei", "id"=>"thrallMod"}
Completed 500 Internal Server Error in 253ms (ActiveRecord: 12.9ms)
OpenSSL::Cipher::CipherError (bad decrypt):
app/models/project.rb:497:in `import_url'
app/models/project.rb:533:in `external_import?'
app/models/project.rb:525:in `import?'
app/models/project.rb:541:in `import_in_progress?'
app/controllers/projects_controller.rb:94:in `show'
lib/gitlab/request_profiler/middleware.rb:15:in `call'
lib/gitlab/middleware/go.rb:16:in `call'

@MrKeiKun if you're not using a Docker install, then this isn't necessarily the same issue. The value of db_key_base has probably changed, see https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-11-stable/doc/update/README.md#migrating-legacy-secrets for more details.

@smcgivern I moved my gitlab server to new machine .Diffurent accounts accessed in different gitlab-secrets.json file, I mean they met 500 error!So,how to combine these two gitlab-secrets.json file ?

@kono123 if you only have data from your old server, then your gitlab-secrets.json file should be identical on the new server. If it isn't, you should take a backup of the version on the new server, and replace it with the old one. However, this is really nothing to do with the topic of this issue 🙂 I would recommend following the steps appropriate for your circumstances at https://about.gitlab.com/support/

@smcgivern Thanks for your reply. I shuold see that when I migrated to new gitlab server , we enabled two-factor athentication.But some users can success enable two-factor athentication, some not. So I copy old gitlab-secrets.json to overwrite the same name file on new server. So, the other users can enable two-factor athentication, some users can not now. I think these two gitlab-secrets.json files conflict.How to solve that? Appreciate to your reply.

@kono123 as I said before, the other participants of this thread probably aren't interested in this discussion, so please use the channels listed for support 🙂