In the light of recent database performance issues we have created an epic to address the queries that are either top 10 in time taken or top 10 in the frequency of calls during the monitoring period - [Volunteer Research Database Performance Peak queries and create issues (&5652)](https://gitlab.com/groups/gitlab-org/-/epics/5652) Now we are asking teams to review their backlogs (and possibly tribal knowledge) to proactively prioritize and address known query performance issues that are not captured within this epic. This is an urgent request to help reduce the load on our database. If there are no queries that immediately come to mind for your teams, please take some time to brainstorm some areas of improved efficiencies and reducing database pressure on our primary. What to look for? - Known n+1 issues - Read-only queries that can be redirected to read-only replicas - Opportunities for caching data on frequent reads - Lazy loading data (example [Lazily load pipeline artifact in project pipelines index page (gitlab#324003)] (https://gitlab.com/gitlab-org/gitlab/-/issues/324003#note_526293202)) - Removing or not exact caching costly but not super needed information totally or through a feature flag (so we only turn it off for .com for the time being). See [comment](https://gitlab.com/gitlab-org/gitlab/-/issues/325440#note_534199203) below. - **Unknown unknowns:** please think creatively about ways to find and optimize queries, or otherwise relieve DB load! - [**Use Elasticsearch with PostgreSQL slow logs**](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/3361) to collect more information when you have suspicious queries. - Look at this [dashboard](https://dashboards.gitlab.net/d/api-rails-controller/api-rails-controller?viewPanel=2155933118&orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-controller=Grape&var-action=All) for endpoints that have a high number of SQL calls ## To Do - Create issues for each query optimization and add to the [Volunteer Research Database Performance Peak queries and create issues](https://gitlab.com/groups/gitlab-org/-/epics/5652) epic. - Prioritize these efforts as gitlab~3713902 and gitlab~4107614 using the [Availability Severity SLO](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#availability) timeline of resolution within 7 days ## Instructions 1. Please follow the guidance described above. 2. Create an issue for your team, here are examples - https://gitlab.com/gitlab-com/create-stage/create-engineering-managers/-/issues/131, https://gitlab.com/gitlab-com/create-stage/create-engineering-managers/-/issues/130 3. Add your team's issue to the table below under `Team Audit Issue Link` | Team | Eng Manager | Team Audit Issue Link | | ------------------- | --------------- | :---------: | | Configure | @nicholasklick | | | Create:Editor | @rkuba | https://gitlab.com/gitlab-com/create-stage/create-engineering-managers/-/issues/128 | | Create:Source Code BE | @sean_carroll -> @nick.thomas | https://gitlab.com/gitlab-com/create-stage/create-engineering-managers/-/issues/129 | | Create:Code Review BE | @m_gill | https://gitlab.com/gitlab-com/create-stage/create-engineering-managers/-/issues/130| | Create:Code Review, Create:Source Code FE | @andr3 | https://gitlab.com/gitlab-com/create-stage/create-engineering-managers/-/issues/126| | Ecosystem BE | @mnohr | https://gitlab.com/gitlab-org/ecosystem-team/team-tasks/-/issues/67 | | Global Search | @changzhengliu | https://gitlab.com/gitlab-org/search-team/team-tasks/-/issues/36 | | Growth:Activation, Adoption, Conversion, Expansion | @pcalder | https://gitlab.com/gitlab-org/gitlab/-/issues/325714 | | Fulfillment:License | @jameslopez | | | Fulfillment:Purchase | @chris_baus | | | Fulfillment:Utilization | @csouthard | https://gitlab.com/gitlab-org/fulfillment-meta/-/issues/192 | | Geo | @nhxnguyen | https://gitlab.com/gitlab-com/create-stage/create-engineering-managers/-/issues/131 | | Gitaly | @zj-gitlab | | | Manage:Access, Import BE | @lmcandrew | https://gitlab.com/gitlab-org/manage/access/discussion/-/issues/10, https://gitlab.com/gitlab-org/manage/import/backend/discussion/-/issues/6 | | Manage:Optimize, Compliance BE | @djensen | | | Monitor | @crystalpoole | https://gitlab.com/gitlab-org/monitor/monitor/-/issues/67 | | Package | @dcroft | | | Plan:Product Planning & Certify | @johnhope | | | Plan:Project Management | @jlear | | | Product Intelligence | @jeromezng | https://gitlab.com/gitlab-org/gitlab/-/issues/326978 | | Protect:Container Security | @thiagocsf | https://gitlab.com/gitlab-org/gitlab/-/issues/326559 | | Release | @nicolewilliams | https://gitlab.com/gitlab-org/ci-cd/release-group/release/-/issues/60 | | Secure:Composition Analysis BE | @gonzoyumo| https://gitlab.com/gitlab-org/gitlab/-/issues/327019 | | Secure:Dynamic Analysis, Fuzz Testing BE | @sethgitlab | https://gitlab.com/gitlab-org/gitlab/-/issues/327159 | | Secure:Static Analysis BE | @twoodham | https://gitlab.com/gitlab-org/gitlab/-/issues/327996 | | Secure:Threat Insights | @thiagocsf | https://gitlab.com/gitlab-org/gitlab/-/issues/326557 | | Verify:CI, Pipeline Authoring BE | @cheryl.li | | | Verify:Runner | @erushton | https://gitlab.com/gitlab-com/runner-group/team-tasks/-/issues/65 | | Verify:Testing | @rickywiens | |