Skip to content

Database Group Triage for week ending 2025-10-24

About

This issue is used by groupdatabase frameworks to triage issues and make sure they get properly assigned and prioritized. Each week, a bot will look up the old issue, pick the next assignee in the list, and submit a new issue with a list of any issues that may need attention from the team.

Last week's issue

Process

Issues needing triage (Labeled databasetriage)

For each issue below:

  1. If the issue needs further investigation, spend up to 1 hour of investigating or fixing the issue.
  2. If the issue is a typebug, assign it one of severity1, severity2, severity3, or severity4
  3. Document any findings you make in a comment on the issue, and if the issue still needs additional work or refinement, consider looping in @alexives and @ to help with scheduling and priority.
  4. If the issue is incomplete, labeled ~"workflow::scheduled", and will take more than an hour to fix, remove databasetriage

Bugs needing Severity

For each issue below:

  1. For each typebug, spend up to 1 hour investigating or fixing the issue.
  2. Assign it one of severity1, severity2, severity3, or severity4
  3. Document any findings you make in a comment on the issue, and if the issue still needs additional work or refinement, consider looping in @alexives and @ to help with scheduling and priority.

Recent issues labeled database

For each issue below:

  1. If the issue has no group label, consider if it should be addressed by groupdatabase frameworks and if so label it.
  2. If the issue has a group, and you think they may need assistance from us:
    • If the issue needs further investigation, add databasetriage and spend up to 1 hour of investigating the issue.
    • Document any findings you make in a comment on the issue, and if the issue still needs additional work or refinement, consider looping in @alexives and/or @

Recent mentions of @gitlab-org/database-team

For each mention below:

  1. If the item already has an adequate response, move on to the next
  2. If the mention is from someone looking to provide feedback on database review, and nobody has responded yet, set up a coffee chat with them and make a note in the thread that you did. Record feedback in the feedback issue.
  3. If you know how to respond to the comment, post a response
  4. If you don't know how, redirect with a specific mention to someone who may be able to respond

Customer Issue Hand-offs

For each issue below:

  1. If the item already has a back and forth, check in with the @morefice to see if what needs to be handed off
  2. Consider if it's right for our team and if not, ask the support rep to follow up with the correct team.
  3. If the item is right for our team, spend some time accessing it and trying to assist.
  4. If you need help, and the request seems pressing, ask in the team channel if there's someone who can help dig into it.

Review Current Saturation Report

DRAFT: Database Capacity Report for week ending 2025-10-26

Please update this week's database capacity report! Remember:

  • You can always reach out to the team for support
  • Consider reaching out in #g_database_operations for feedback too

Process:

Review Top Queries for Changes

There were some new anomalous queries to review on the main database, broken down by metric type:

Metric # of queries
by_total_time 0
by_time_avg 0

Please find the detailed query report here (Ops access required.)

For each query listed:

  1. Spend up to 30 minutes trying to understand its source.
  2. If needed, determine the team that owns the query and file an issue with them. A good place to start would be to check the relevant table's owner under db/docs.

For additional context around how these queries were determined to be anomalous as well as historical rankings for known queries, all of the data we have collected lives in a database dump stored in artifacts in the query-stats-reporting repo on ops. Specifically check query-stats.yml for an example of how you can use this dump to locally rehydrate this database on your own Postgres instance and analyze the collected statistics. For this iteration of this report, queries are grouped by fingerprint.

Legacy query analysis

It is not necessary to look at this unless we need to check the new anomalous query report against the top query report on the main database.

Click to expand

For each database:

  1. Are there new queries in the top queries (See: K003 Top-15 Queries by total_time) compared to a previous report?
  2. If there are new queries, Spend up to 30 minutes trying to understand their sources
  3. If there are new queries, file an issue and assign to the team that owns the query, or if unable to source, then the team that owns the table
  4. If there are no new queries, review the top 5 queries for each to see if there are already investigations, or file issues to investigate them

Review Dashboards on queries with large In-Lists

For both Sidekiq and Rails:

  1. If the report is empty, no action is needed
  2. For each item on the report:
    1. Determine what feature the query belongs to
    2. Create an issue for the team owning the feature category asking them to limit the maximum number of items in the in query

Int4 Saturation Review

For issue(s) linked below:

  1. Ensure each table referenced in the report has an associated issue with ~"Gitlab.com Resource Saturation" and infradev attached

  2. Make sure the issue is assigned to a team.

@alexives if @praba.m7n isn't available this week, please reassign to @stomlinson

This is generated by this project.

Edited by Prabakaran Murugesan