- [Handbook page](https://handbook.gitlab.com/handbook/engineering/development/dev/plan/working-groups/work-items-api-performance/) <details> <summary> Related issues can belong to different groups and epic, so we use ~"WG::WorkItemsPerformance" to track all the work in one place: </summary> ```glql display: table fields: state, epic, title, labels("group::*"), milestone, labels("workflow::*"), assignees query: group = "gitlab-org" and label = "WG::WorkItemsPerformance" ``` </details> ## Updates <details> <summary>2025-03-04</summary> [Agenda](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.cp5awd4mr4sc) - [Recording](https://youtu.be/2ckIJEl4T1Q) For now we're focusing on establishing performance baseline for: * Works Items list GraphQL API on the group level. * Children list widget API for a single work item. We are focusing only on the new Work Items API and not the legacy APIs because: * Underlying finders are the same, so speeding up the new API should also improve the old one * Work Items API blocks multiple high-priority roadmap items Action items: * `@nicolasdular` to select a few Work Items queries ([example](https://gitlab.com/-/snippets/4817251)) and measure baseline performance. * `@nicolasdular` to find most frequently used filters on the group level and coordinate with Product on which are the most important, which we should optimize first. * `@afrnz` to help if needed. * `@egrieff` to find a way measure performance of the children widget and establish a baseline for it * `@acroitor` research how much time can be saved by moving permissions check from graphql layer to finder * `@kassio` research difference in performance of queries for signed-in/anonymous/membership users * `@engwan` map out how current issues finders work Other updates: * Fetching first 100 work items for `gitlab-org` takes 6s without any filters applied. That could cause a significant performance degradation when moving Epics list to Work Items API. cc `@jachapman` `@kushalpandya` </details> <details> <summary>2025-03-11</summary> [Agenda](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.cp5awd4mr4sc) - [Recording](https://youtu.be/b7neTWHB9Yo) * The biggest performance problem right now are a few N+1 queries, `@egrieff` will be working on it * We discovered that GraphQL has a huge overhead compared to REST, especially when N+1 are disabled and on the project levle: * In local tests for `@acroitor` GraphQL took over 85% of the execution time * From the [dashboard](https://log.gprd.gitlab.net/app/dashboards#/view/f0352650-7348-47b4-80da-1604e3ae3140?_g=h@5852f73) `@nicolasdular` created, you can see that for projects we spend twice as much time in cpu as in database(but it's reversed for group queries) * `@engwan` discovered that basic queries right now select all open issues, filter and sort them in memory. Simplifying some parts of the query makes it use index which gives a good performance boost. * We had a long discussion about using Elastic Search in the future and `@afrnz` will work on the PoC for that with some help from `@dgruzd` and `@terrichu` * `@nicolasdular` and `@egrieff` did [a lot of benchmarking of the current API](https://gitlab.com/groups/gitlab-org/-/epics/16919#note_2389958249) **Actions** * `@engwan` will continue looking into optimizing SQL queries * `@acroitor` and `@engwan` will pair to inspect graphql overhead closer * `@egrieff` continues to address the most pressing problem - children widget that creates N+1 * `@afrnz` will work on the PoC for using Elastic Search * `@kassio` will continue investigating requests when user has different level of access to the issue list </details> <details> <summary>2025-03-18</summary> [Agenda](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.cp5awd4mr4sc) - [Recording](https://www.youtube.com/watch?v=VT-U8wGOanI) We spend most of the time discussing how to optimize blocked users query. But we struggle to find capacity for working group among other deliverables in Plan, so we decided to delay some of action items till next week to see if we'll have capacity for them. **Updates** * `@egrieff` is still working on N+1 in the linked items widget, but needed to change the approach * `@acroitor and @engwan discovered that with some disabled debugging CPU vs db time difference is less significant, but still very high (`1.5s total time vs 50ms` db in development environment) * `@kassio` and `@mdangelo6` [continue to investigate slow down for unauthorized users](https://gitlab.com/gitlab-org/gitlab/-/issues/513012#note_2401943362). * `@engwan` discovered that removing fitlering for either blocked users or work item type yields dramatic speed up of the SQL query execution time. Type filtering can be speed by introducing better indexes, and we opened https://gitlab.com/gitlab-org/gitlab/-/issues/525549+ to discuss what to do about blocked users * `@dgruzd` suggested looking into [recent optimisation introduced in search](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/183748) that can speed up permissions search * `@afrnz` is working on https://gitlab.com/gitlab-org/gitlab/-/issues/524326+s **Actions** * `@acroitor` to repeat flamegraph experiments on staging trying to reproduce the difference seen in development environment * DELAYED: Look into https://gitlab.com/gitlab-org/gitlab/-/issues/525507+s * `@vshushlin` to drive discussion with product and anti-abuse team about https://gitlab.com/gitlab-org/gitlab/-/issues/525549+s </details> <details> <summary>2025-03-25</summary> [Agenda](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.cp5awd4mr4sc) - [Recording](https://youtu.be/Ymq7KkpgrNg) We're narrowing down on 4 main objectives: | Problem | DRI's | Status | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DB queries for large groups (e.g. gitlab-org) | `@engwan` `@nicolasdular` `@kassio` | Have [promising ideas](https://gitlab.com/gitlab-org/gitlab/-/issues/525507#note_2403928089 "Research if Plan can adopt work items/epic optimizations from Advanced Search"), trying out PoC's. | | Blocked users slow DB queries a lot | `@vshushlin` until we make the decision) | Opened [Remove issues created by blocked users to speed... (gitlab#525549)](https://gitlab.com/gitlab-org/gitlab/-/issues/525549 "Remove issues created by blocked users to speed up search queries"), no response, escalating. | | GraphQL overhead | `@acroitor` | Still investigating, it's tricky | | N+1's | `@egrieff` | Should merge the solution for hierarchy widget this week, will continue working on other N+1's | | Investigating Elastic Search as potential solution | `@afrnz` `@dgruzd` | [Have a promising PoC for GLQL](https://gitlab.com/gitlab-org/gitlab/-/issues/524326) | Updates/actions items - `@dgruzd` is working on [the design document for finders that can use both Postgress and Elastic Search as backend](https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/12300) - `@terrichu` has shared the [optimization the Global Search team is using to speedup permissions check](https://gitlab.com/gitlab-org/gitlab/-/issues/525507#note_2403928089), `@nicolasdular` will sync with @engwan and @mdangelo6 on how we can use it. - `@egrieff` found a good solution for N+1 in hierarchy widget and expect to get it merged this week, she'll continue working on other N+1's after that. - `@engwan` is looking at group level queries and dashboard level queries together with @mdangelo6 - We opened https://gitlab.com/gitlab-org/gitlab/-/issues/525549+, but didn't get any response from anti-abuse team yet - `@acroitor` continues to look into graphql overhead, but it's still hard to point to the problem, he'll continue to investigate in review app or production - `@mdangelo6` has contributed [a great N+1 improvement](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/185034#note_2412684523) - `@afrnz` has opened [a PoC for integrating Elastic Search into GLQL](https://gitlab.com/gitlab-org/gitlab/-/issues/524326) </details> <details> <summary>2025-04-15</summary> [Agenda](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.24xzjw3rmoh2) - [Recording](https://www.youtube.com/watch?v=9H-b_HLJAlI) - Dmitry has finished the ltree spike and concluded that https://gitlab.com/gitlab-org/gitlab/-/issues/533333+ is a better approach. To move forward, we need to present this proposal to the DB team so `@dgruzd` and `@nicolasdular` will collaborate on creating the plan, with `@acroitor` helping review. - In the meantime, we’ll work on smaller improvements that could benefit the non-combined WIs queries: - `@engwan` will focus on converting the project-level queries to namespace-based ones and creating the equivalent indexes. `@mdangelo6` will collaborate with Heinrich if any delegation is needed. - `@jachapman` and `@egrieff` will continue investigating [Epic lists performance parity](https://gitlab.com/groups/gitlab-org/-/epics/17511) to identify non-DB-related optimizations. - `@afrnz` has completed a POC for GLQL ES and shared some fantastic results (findings are in this [Slack thread](https://gitlab.slack.com/archives/C08G0G394CD/p1744181067093849)). Next steps will be defined soon, potentially including integration of all existing GLQL fields and filters (limited to work items). Pagination and aggregations may also be considered. - Soon, we’ll need to update the way project issues are authorized to support using project namespaces instead of projects. I’ll look into the available options and investigate how the tree structure used by Advanced Search works and if it can be reused for our case ([related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/525507)). </details> <details> <summary>2025-04-22</summary> [Agenda](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.trol2h1tut9e) - [Recording](https://www.youtube.com/watch?v=VnC9Zos_e44) - `@engwan` Discovered that existing indexes for `project_ids` are not being used for the work items query, so the new indexes impact could be greater. He created https://gitlab.com/groups/gitlab-org/-/epics/17531+ and will focus on https://gitlab.com/gitlab-org/gitlab/-/work_items/536353+. Once we have more info on its impact, we can decide if we move forward with other DB indexes. This epic also requires updating the finders, so `@mdangelo6` will start looking into https://gitlab.com/gitlab-org/gitlab/-/issues/536351+ next. - `@nicolasdular` is back from PTO, so he will start collaborating with `@dgruzd` on the plan for `traversal_ids` denormalisation - `@jachapman` and `@egrieff` will continue working on non-DB-related optimisations </details> <details> <summary>2025-04-29</summary> [Agenda](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.xs2iwt87pagb) - [Recording](https://youtu.be/k5NZ8uzpxow) #### Traversal_ids denormalisation - @nicolasdular will continue working on the traversal_ids proposal with Dimitry's assistance. The plan will have to include the index proposed on https://gitlab.com/gitlab-org/gitlab/-/work_items/536353+. - @vshushlin will help find a DB counterpart to review the proposal once ready. #### Switch to use namespace-based queries - @engwan will keep working on dropping the unused indexes and adding the new ones for namespace_id. He will aim to get as much done this week so the migrations can run during the weekend. - There is a chance we don't have to add all the sorting indexes once we have the `traversal_ids` denormalisation. - @mdangelo6 is working on [updating](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/188961) finders to query by `namespace_id`. This will be merged soon, but will be behind a feature flag so we can wait for the indexes to test it. We also need to check if we have any other places where we still query the `project_id` due to not using the finder. #### Advanced Search - Advanced Search integration now has Keyset pagination support, which means we could switch the work items API to using Advanced Search in the future. This requires having all filters and finders ready, so @egrieff will revise the epic https://gitlab.com/groups/gitlab-org/-/epics/17460 to find a way to organise this work so it can be shared between ~"group::knowledge" and ~"group::product planning". - This is low priority for the performance WG, but will be good to have as an option if we have the capacity. #### Other optimisations - @engwan merged an [optimization for the banned users filter](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/189141) which improved the project list by ~300ms but does not affect group level query. - Most other smaller optimisations (N+1s and ~frontend) may be completed by next week, so we can re-test the epics list performance and have a better estimation for release. </details> <details> <summary>2025-05-06</summary> - [Agenda](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.7yq4rx5c1gfm) - [Recording](https://youtu.be/Uxumm5CfDhA) #### Traversal_ids denormalisation - The plan for normalising travelsal_ids has been [reviewed](https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/13283#note_2481254584) by the DB team, and the only concern is the number of indexes needed for sorting. We'll discuss the option of not covering all sorting indexes and recommend the use of ES when these present performance problems. - Given that larger instances might still encounter problems that are not solvable by PG, we agreed that ES support is needed, at least in the long term. We will have to decide if the `traversal_ids` optimisation is still worth pursuing, or we can prioritise ES support. The help make this decision, @nicolasdular will look for more info on ES adoption in Plan customers and what size of data makes queries timeout. #### Switch to use namespace-based queries - There were two indexes dropped last weekend, and there will be four more to be dropped this weekend. We may be able to add the new indexes this weekend too, but this depends on DB approval for the [schema change exception request](https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/510). Otherwise, we'll need to wait until the following weekend. The [MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/188961) that updates the finders to use `namespace_id` is in review. We don't see any blockers for it getting merged, so it's probably going to be available for testing the new indexes. #### Advanced Search We now have a https://gitlab.com/gitlab-org/gitlab/-/issues/538806+ to track what filters are missing for using ES for work item lists. @afrnz and @egrieff will sync up to consolidate the planning with the [implementation](https://gitlab.com/groups/gitlab-org/-/epics/17460) for GLQL #### Other optimisations https://gitlab.com/gitlab-org/gitlab/-/merge_requests/189332+ has been merged but not yet deployed. We'll re-meassure epic's list performance once the optimization is available. </details> <details> <summary>2025-05-13</summary> [Agenda](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.rk0jfiyq3rtv) - [Recording](https://youtu.be/U9XCoXYHS-o) - **Traversal_ids denormalization** - This presents a significant challenge due to the number of indexes that would be required to support all the sorting options. There are other possible approaches that we could consider (explained in https://gitlab.com/gitlab-org/gitlab/-/issues/533333#note_2497848869), but we decided to try the optimisation that uses # [Efficient `IN` operator queries](https://docs.gitlab.com/development/database/efficient_in_operator_queries/). This could present a significant improvement and could be implemented relatively quickly (1-2 weeks). @nicolasdular will spend time on it this week. - Given that the sorting options are a challenge for all optimisations, @nicolasdular will try to find data on their usage so we can discuss with Product any alternative solutions. - **Switch to use namespace-based queries** - @engwan's indexing plan was approved by the DB team. A total of 6 indexes have already been dropped on `GitLab.com`. The MR to add the new indexes is close to being merged: [https://gitlab.com/gitlab-org/gitlab/-/merge_requests/188963](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/188963) and might be available next week. - @mdangelo6's [MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/188961) that updates finders to use `namespace_id` has been merged. This is behind the FF `use_namespace_id_for_issue_and_work_item_finders` and can be tested once we have the new indexes that support this column. - **Advanced Search** - @egrieff continued the investigation for https://gitlab.com/gitlab-org/gitlab/-/issues/538806+, but given the limited access to ES, we are not prioritising this work for now. We can consider using Advanced Search for lists which have the default sorting and no filters, as the effort would be minimal, but the only benefit we see for now is improving error budgets. - @alisa will keep working on https://gitlab.com/groups/gitlab-org/-/epics/17460+ and every progress there ultimately contributes to https://gitlab.com/groups/gitlab-org/-/epics/17705+ - **Other optimisations** - @dgruzd raised concerns about the impact of filtering by custom fields (See [Slack thread (internal](https://gitlab.slack.com/archives/C08G0G394CD/p1747036988108269)). We don't have any metrics on this yet, but we will try to find more information from the ~"group::project management" team - @engwan is currently looking at optimisations for the notes endpoint. - The N+1s [fix](https://gitlab.com/gitlab-org/gitlab/-/issues/512056) and the frontend optimisation (https://gitlab.com/gitlab-org/gitlab/-/issues/535657) have reduced the time spent on the main Graphql query for the [epics list](https://gitlab.com/groups/gitlab-org/-/epics/17511#updated-metrics-comparison---2025-05-09), but we hope to see an improvement when using the new indexes too. - We are concerned about this query's high CPU time, and @egrieff is looking into this. As a first measurement, we will try removing the `workItems` field authorisation ([MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/190860) in progress). </details> <details> <summary>2025-05-20</summary> - [**Agenda**](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.7gp3x6cmzo2d) - [**Recording**](https://www.youtube.com/watch?v=tWyk2GS6a7M) **Traversal_ids denormalization** @nicolasdular found a way around the limitations of this proposal: we can consider removing large fields (e.g., description, title) from the issues table into a new table to reduce table size. This approach would avoid the need for duplicating sorting indexes and offers better performance for traversal ID usage due to having a smaller table (we could potentially reduce the size by nearly 200GB). We will not start working on this yet, as we focus on unblocking consolidated lists, but it will be in the long-term plan to improve performance (this would require more progress on Workstream 5 anyway). **Efficient IN Operator** This proposal is in progress, @nicolasdular has opened: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/191663, which is missing a fix for keyset pagination and investigating why the query is slower when using the new `work_item_type` index. We don't expect any complications, so the estimate is to the results (behind a FF) within 1-2 weeks. **Using namespace-based queries** - @engwan merged https://gitlab.com/gitlab-org/gitlab/-/merge_requests/188963 that added the new indexes, but the migration has not been executed yet; we need to wait until next weekend. - @mdangelo6 will take https://gitlab.com/gitlab-org/gitlab/-/issues/543739 as lower priority because it can wait while project indexes are still present. **Advanced Search** @dgruzd Shared the news that the security infrastructure team is going to launch Advanced Search integration for vulnerabilities on .com this week, which includes their GraphQL API with keyset pagination. This will be helpful when working on the integration for the work items API. However, we would not be focusing on this implementation yet, but it will be added to the list of long-term optimisations that ~"group::product planning" wants to work on. **Other optimisations** - @egrieff and @jachapman worked on https://gitlab.com/gitlab-org/gitlab/-/merge_requests/191317 that removes the linked items from the main query, and could result in a ~50% reduction of CPU time for the main work items query. - @egrieff Is still working on https://gitlab.com/gitlab-org/gitlab/-/merge_requests/190860 to remove the field authorisation. This could go into review this week, and local testing showed a ~10% decrease in CPU time. - We discussed other potential optimisations for CPU time, but we have not found anything with a significant impact yet. These include: - Using raw output from GraphQL to bypass some processing. - Exploring ways to avoid redundant object creation. - Reducing field authorization where possible. At the end of the week, we'll reassess the performance parity for epic list. We expect to see good results after https://gitlab.com/gitlab-org/gitlab/-/merge_requests/191317, but potentially need to wait for new indexes to reach parity. </details> <details> <summary>2025-06-03</summary> - [**Agenda**](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.jqxq9v2p7yqw) - [**Recording**](https://youtu.be/PuTjQARxCxg) - **Efficient IN Operator** - Still work in progress, it was deprioritised for a few days, but it will be resumed this week - **Namespace indexes / Description Table** - We'll move forward with moving description and other non-filtering columns to a separate table, @nicolasdular has a plan for it in progress that might be ready within 1-2 weeks. He'll be on PTO after that, but will pick up the task when he's back. - The `namespace_id` based indexes have been added, but we missed one that covers the `updated_at` sorting. @mdangelo6 opened a [DB exception request](https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/516) and will work on adding the index next - **Other optimisations** - @egrieff will continue to focus on improving CPU time for the main GraphQL query. We have a few fixes in flight that could be available within 1-2 weeks (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/190860, https://gitlab.com/gitlab-org/gitlab/-/merge_requests/193259, https://gitlab.com/gitlab-org/gitlab/-/merge_requests/193262) - As the epic list is being rolled out, we'll next move our focus to the issues list </details> <details> <summary>2025-06-10</summary> - [**Agenda**](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.8hngs7oiachv) - [**Recording**](https://www.youtube.com/watch?v=WciN89d40LU) - **Updates** </details> <details> <summary>2025-06-17</summary> - [**Agenda**](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.8auzgt1eeg2j) - [**Recording**](https://www.youtube.com/watch?v=R2LF2vPtB4Y) - **Updates** </details> <details> <summary>2025-06-24</summary> - [**Agenda**](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.4wapl1w3e121) - **Recording** - meeting cancelled - **Updates** - **Efficient IN Operator** There has been no progress on https://gitlab.com/gitlab-org/gitlab/-/merge_requests/191663+ since @nicolasdular went on PTO - **Namespace indexes / Description Table** All indexes for https://gitlab.com/groups/gitlab-org/-/epics/17531 have now been created, but namespace-based queries are still behind the disabled FF `use_namespace_id_for_issue_and_work_item_finders` because https://gitlab.com/gitlab-org/gitlab/-/issues/543739 is a blocker to enable this. We are prioritising the feature parity work for work item lists, so this work will be put on hold for now. - **Other optimisations** The optimisation https://gitlab.com/gitlab-org/gitlab/-/merge_requests/190860+ was merged but needs a follow-up because DB queries are timing out when using production data. </details> <details> <summary>2025-07-01</summary> - [**Agenda**](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.hz76dgqpab95) - **Recording** - meeting cancelled - **Updates** - **Efficient IN Operator** No changes from last week - **Namespace indexes / Description Table** No changes from last week - **Other optimisations** - https://gitlab.com/groups/gitlab-org/-/epics/17511+ was closed because we reached parity, but this list will keep improving with the optimisation in progress. - Now that we have the new work item queries that will be used for project and group level issues list, we were able to compare the timing with the legacy lists (captured in https://gitlab.com/groups/gitlab-org/-/epics/17873#bar_chart-metrics-comparison and https://gitlab.com/groups/gitlab-org/-/epics/18270#bar_chart-metrics-comparison, respectively). We have performance parity for the project-level list, but the group list takes double the time, so it will be the next focus. - The follow-up for https://gitlab.com/gitlab-org/gitlab/-/merge_requests/190860+ is in progress. - We identified another optimisation https://gitlab.com/gitlab-org/gitlab/-/issues/552353+ and a [fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/196140) is in progress too. This and the authorisation change should contribute towards reducing the duration for the group-level issue GQL query, but the finder DB query is still the main problem. </details> ### 2025-07-15 - [**Agenda**](https://docs.google.com/document/d/1S5ZSbEOSCAUWe0U3gZPGXSUKIaFjjn_HD1UGkz2hhXY/edit?tab=t.0#heading=h.s7a1vl8iocd6) - [**Recording**](https://youtu.be/qz6lfxvf8ss) #### Efficient In Operator - Deprioritised The solution only works for limited scenarios and breaks with some filters. We'll close this work for now and focus on other solutions. #### Issues Table Decomposition - Primary Focus Given that the issues table is 300GB and thus too large to add extra indexes for traversal ids, we'll split it into two smaller tables (issues table \~100GB, descriptions table \~200GB). Although the resulting tables are still over the recommended size, we expect Database team approval for adding the additional indexes. - **Proposed plan**: - Create work items descriptions table (3-month effort, including the required migration stop) - Include description, description_html and full-text search index - Partition by namespace ID from the start - Keep title in main issues table (it's only 3% of total size) #### Issues Table Partitioning We also discussed the option of partitioning this table, but although this is a long-term objective, we identified some challenges which might not be solved in time for the release of the consolidated list. - **Challenges identified**: - Cross-namespace queries (user-assigned issues, linked items, parent-child relationships) - Organizations/cells architecture is not yet mature enough to rely on #### Next Steps **Nicolas:** - Test traversal IDs with various filters to confirm performance benefits - Begin work on table decomposition **Eugenia:** - Continue focus on error budget work during this milestone - Continue monitoring group issues list performance metrics