Dev instance contains improper storage configuration

added groupProduction Engineering unblocks others workflow-infraTriage labels

mentioned in incident production#18314 (closed)

The label severity4 has been applied due to missing severity:: label.

To ensure the necessary information is captured in this issue for a ::"corrective action", please apply the corrective_action issue template, which includes a quick action to apply the default severity label.

You are welcome to help improve this comment.

added severity4 label

Hi @skarbek,

The label ServiceUnknown has been applied due to missing Service:: label.

Please update accordingly.

You are welcome to help improve this comment.

added ServiceUnknown label

Setting a due date of 2024-10-20 based on corrective action SLOs

You are welcome to help improve this comment.

changed due date to October 20, 2024

I'm not actually sure who should own this... It's clearly an infrastructure configuration w/i the application...

added priority1 severity1 labels and removed severity4 label

In https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/4742, we had to drop the alternate storage from the gitlab.rb due to gitlab-org/omnibus-gitlab!7564 (merged). This Omnibus change led to a production incident in production#17988 (closed).

It seems that we should probably fix ApplicationSetting to ignore/drop the storages with weight of 0.

Actually, all it took was to click Save changes in the UI via https://dev.gitlab.org/admin/application_settings/repository#js-repository-storage-settings:

gitlabhq_production=# select repository_storages_weighted from application_settings;
-[ RECORD 1 ]----------------+-----------------
repository_storages_weighted | {"default": 100}

irb(main):002:0> ApplicationSetting.current.valid?
=> true
irb(main):003:0> ApplicationSetting.current.max_yaml_size_bytes
=> 2097152
irb(main):004:0> ApplicationSetting.current.repository_storages_weighted
=> {"default"=>100}

@stanhu does this mean we can close this issue? Or is there still a lingering problem with our chef runs that are causing a bad config on this instance?

Yeah, I think we can close this issue. I didn't close it immediately because I was concerned that this might be a problem for other users who dropped alternate storages. But I think the problem happened here because we updated the settings through the console, though the same issue might happen if someone uses the API to update the settings. Admins just have to save the application settings once through the UI for this problem to go away.

Setting a due date of 2024-07-29 based on corrective action SLOs

You are welcome to help improve this comment.

changed due date to July 29, 2024

Hi @skarbek,

Please add a team label to this issue to help get it triaged.

If this relates to a specific service, find the appropriate team in the service catalog. Use teamOps for reliability_general services and teamFoundations for reliability_foundations services.

If you are still unsure where the issue should go, please ask in the #infrastructure-lounge Slack channel or ping @gitlab-org/production-engineering/managers in the issue.

Thanks for your help!

added auto updated label

closed

added workflow-infraDone label and removed workflow-infraTriage label

Dev instance contains improper storage configuration

Designs

Child items ...

Activity

Setting a due date of 2024-10-20 based on corrective action SLOs

Setting a due date of 2024-07-29 based on corrective action SLOs