Rescue FS errors in knowledge graph ensure_secret!
What does this MR do and why?
When the knowledge graph feature is enabled but the Rails process doesn't have write permission to the secret file location (common in Omnibus where Rails runs as git user but the directory is owned by root), the application crashes on boot. This happened across all Omnibus QA jobs (#591610 (closed)).
In case where CNG is used the secret is shared across the environments between gitlab and knowledge graph, which makes a higher chance of infrastructure team or self-managed users to enable KG while failing to mount the secret at the same time causing downtime for them.
!225107 (merged) fixed the most common case by skipping secret generation when the feature is disabled. However, the boot crash still happens when an operator enables knowledge graph before the secret file has been provisioned.
This MR makes secret generation fail gracefully — a warning is logged and the application continues to boot. JWT auth will not work until the secret is properly created, but the instance stays up.
Behavior matrix
| Knowledge graph | Write access | Before (master) | After (this MR) |
|---|---|---|---|
| Disabled | Yes | No-op (gate from !225107 (merged)) | No-op |
| Disabled | No | No-op (gate from !225107 (merged)) | No-op |
| Enabled | Yes | Secret created | Secret created |
| Enabled | No | Boot crash (EACCES) | Warning logged, boot continues |
When the last case happens, downstream token generation returns nil and gRPC requests fail with a clear auth error instead of taking the whole instance down.
References
- Original MR: !224386 (merged)
- Incident: #591610 (closed)
- First fix: !225107 (merged)
How to set up and validate locally
- Make the secret path read-only:
touch .gitlab_knowledge_graph_secret && chmod 000 .gitlab_knowledge_graph_secret - Enable knowledge graph in config and boot Rails — should see a warning log instead of a crash
- Clean up:
rm .gitlab_knowledge_graph_secret
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.