Database Group Triage for week ending 2025-07-25
About
This issue is used by groupdatabase frameworks to triage issues and make sure they get properly assigned and prioritized. Each week, a bot will look up the old issue, pick the next assignee in the list, and submit a new issue with a list of any issues that may need attention from the team.
Process
-
Review any issues with undefined types -
Post any questions or pressing issues to the database group meeting doc
Bugs needing Severity
For each issue below:
- For each typebug, spend up to 1 hour investigating or fixing the issue.
- Assign it one of severity1, severity2, severity3, or severity4
- Document any findings you make in a comment on the issue, and if the issue still needs additional work or refinement, consider looping in
@alexivesand@to help with scheduling and priority.
-
Stale FK after CI decomposition causing '"ci_triggers" is write protected within this Gitlab database' errors -
Verify and fix inconsistencies when updating routes
Issues with Undefined Type
For each issue below:
- Assess if the issue is appropriately assigned to groupdatabase frameworks, if not add the correct group label.
- Add the proper work type label, or if the issue is a request for support, redirect the user to our support resources with the following message:
Hey @author. Based on the information given, this request for support is out of the scope of the issue tracker (which is for new bug reports and feature proposals). Unfortunately, I won't be able to help get it resolved. However, for support requests we have several resources that you can use to find help and support from the Community, including: * [Technical Support for Paid Tiers](https://about.gitlab.com/support/) * [Community Forum](https://forum.gitlab.com/) * [Reference Documents and Videos](https://about.gitlab.com/get-help/#references) Please refer to our [Support page](https://about.gitlab.com/support/) for more information. If you believe this was closed in error, please feel free to re-open the issue. /label ~"support request" /close - If the issue needs further investigation, add databasetriage and spend up to 1 hour of investigating the issue.
- Document any findings you make in a comment on the issue, and if the issue still needs additional work or refinement, consider looping in
@alexivesand@to help with scheduling and priority.
-
Split out ActiveRecord monkey patches in lib/gitlab/database.rb -
Keep non-concurrent vs concurrent steps as two separate migrations -
Document: Mixing concurrent and non-concurrent database operations -
Database: Consider removing the surrounding SELECT when using FromUnion.from_union -
Consider removing index_mr_metrics_on_target_project_id_merged_at_time_to_merge index -
Migrate deprecated bulk insert calls to use BulkInsertSafe -
Modify add_concurrent_foreign_keyto usevalidate_foreign_keymethod -
Modify UpdateColumnInBatches bot to also consider ee/spec/migrations -
Usage of Arel (internal Rails API)
Recent issues labeled database
For each issue below:
- If the issue has no
grouplabel, consider if it should be addressed by groupdatabase frameworks and if so label it. - If the issue has a group, and you think they may need assistance from us:
- If the issue needs further investigation, add databasetriage and spend up to 1 hour of investigating the issue.
- Document any findings you make in a comment on the issue, and if the issue still needs additional work or refinement, consider looping in
@alexivesand/or@
-
groupgeo Geo: Transient Geo verification failures should not be treated as failures -
groupsecurity insights Update relevant services to retrieve vulnerabilities by their occurrence_id -
~"group::not_owned" Application Limit for Graph Size -
groupapi Clean-up expired dynamic OAuth applications -
groupsecurity infrastructure Add ref contextual columns to the vulnerability_occurrencestable -
groupsecurity infrastructure Remove no longer needed columns from the vulnerability_occurrencestable -
groupsecurity insights Update the vulnerability ingestion process to fill in data moved from the occurrences table -
groupsecurity infrastructure Backfill non-ref contextual columns added to the vulnerabilitiestable -
groupsecurity infrastructure Add not null contraints to new vulnerability_occurence_idcolumns -
groupsecurity infrastructure Backfill vulnerability_occurrence_id to tables that now relate to occurrences -
grouprunner Fix naming inconsistency of runner tables -
grouporganizations Refactor Schema for Deprecated gitlab_mainTables -
groupsecurity infrastructure When a project is created, add it's default branch to the vulnerabilitity tracked refs list by default -
~"group::not_owned" Backfill the organization tracked ref quota with a value that ensures existing users will not exceed their quota immediately -
groupsecurity infrastructure Create table/setting that stores the organization quota for tracked vulnerability refs -
groupsecurity infrastructure Remove vulnerability_id from tables that now relate to a vulnerability_occurrence record -
groupsecurity infrastructure Remove no longer needed columns from the vulnerabilities table -
groupsecurity infrastructure Remove state transitioned at/by attributes from the vulnerabilities table -
groupsecurity infrastructure Add vulnerability_occurrence_id to to tables that should relate to a vulnerability in a ref context -
groupsecurity infrastructure Update the vulnerability ElasticSearch reference to reflect the ref relationship and details
Recent mentions of @gitlab-org/database-team
For each mention below:
- If the item already has an adequate response, move on to the next
- If the mention is from someone looking to provide feedback on database review, and nobody has responded yet, set up a coffee chat with them and make a note in the thread that you did. Record feedback in the feedback issue.
- If you know how to respond to the comment, post a response
- If you don't know how, redirect with a specific mention to someone who may be able to respond
-
@alexivesmentioned the team on Plan for Database Sharding -
@alexivesmentioned the team on Implement ancestor archived status inheritance for namespaces -
@krasiomentioned the team on Database Capacity Report for week ending 2025-07-13 -
@project_71442692_bot_b452c386215b5679e51a3c0c5b75eb5ementioned triage on Database Capacity Report for week ending 2025-07-20
Customer Issue Hand-offs
For each issue below:
- If the item already has a back and forth, check in with the @stomlinson to see if what needs to be handed off
- Consider if it's right for our team and if not, ask the support rep to follow up with the correct team.
- If the item is right for our team, spend some time accessing it and trying to assist.
- If you need help, and the request seems pressing, ask in the team channel if there's someone who can help dig into it.
-
Background migration job "BackfillPartitionedWebHookLogsDaily: web_hook_logs" stuck at finializing~"Help group::Database Frameworks", ~"RFH-Lifecycle::Last comment from Support", severity3
Review Top Queries for Changes
There were some new anomalous queries to review on the main database, broken down by metric type:
| Metric | # of queries |
|---|---|
| by_total_time | 0 |
| by_time_avg | 1 |
Please find the detailed query report here (Ops access required)
For each query listed:
- Spend up to 30 minutes trying to understand its source.
- If needed, determine the team that owns the query and file an issue with them. A good place to start would be to check the relevant table's owner under
db/docs.
For additional context around how these queries were determined to be anomalous as well as historical rankings for known queries, all of the data we have collected lives in a database dump stored in artifacts in the query-stats-reporting repo on ops. Specifically check query-stats.yml for an example of how you can use this dump to locally rehydrate this database on your own Postgres instance and analyze the collected statistics. For this iteration of this report, queries are grouped by fingerprint.
Legacy query analysis
It is not necessary to look at this unless we need to check the new anomalous query report against the top query report on the main database.
Click to expand
For each database:
- Are there new queries in the top queries (See: K003 Top-15 Queries by total_time) compared to a previous report?
- If there are new queries, Spend up to 30 minutes trying to understand their sources
- If there are new queries, file an issue and assign to the team that owns the query, or if unable to source, then the team that owns the table
- If there are no new queries, review the top 5 queries for each to see if there are already investigations, or file issues to investigate them
-
Review recent Primary Checkup for new Top Queries (K003) -
Review recent CI Checkup for new Top Queries (K003)
Review Dashboards on queries with large In-Lists
For both Sidekiq and Rails:
- If the report is empty, no action is needed
- For each item on the report:
- Determine what feature the query belongs to
- Create an issue for the team owning the feature category asking them to limit the maximum number of items in the in query
Int4 Saturation Review
For issue(s) linked below:
-
Ensure each table referenced in the report has an associated issue with ~"Gitlab.com Resource Saturation" and infradev attached
-
Make sure the issue is assigned to a team.
-
All tables approaching saturation in Capacity warning for patroni: pg_int4_id have an issue assigned to a team.
-
@alexives if @irina.bronipolsky isn't available this week, please reassign to @jon_jenkins
cc @
This is generated by this project.