### Overview Searches that are across entire groups can be slow if the Group has a large number of projects. This is because the query will contain all the project IDs that are part of the selected group. Adding the Group Hierarchy in the indexed document will allow for the query to specify the group level as a filter and keep eliminate the need to have a large list of project IDs included in the query. Additionally, There are mapping needs for Wikis that may be different than Repo Blobs and may need additional rules to be included in the indexer. Problems that have been reported on indexing wikis include. - Titles don't get indexed when a project is moved. - File contents are being indexed even though these files type are supposed to be omitted from the index. - We should confirm file size limits are applied correctly ### Steps - [x] Add namespace_ancestry_ids to Wiki blobs index mapping ~~[MR Changes](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104956)~~ [2nd Iteration](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106798) - [x] Populate namespace_ancestry_ids for new/updated documents for Wiki blobs, and add a flag for namespace_ancestry_ids field to be sent or not (make changes in the indexer) [MR Changes](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/merge_requests/171) - [x] Release the new indexer version 3.1.1 [MR Link](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/merge_requests/172) - [x] Send the flag namespace_ancestry_ids to the indexer if migration is applied. [MR Link](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107352) - [x] Backfill namespace_ancestry for blobs _[MR example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68946)_ (Note: we want to avoid hitting Gitaly. We'll probably need to use painless scripts) [MR Link](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107730) - [x] Backfill namespace_ancestry for wiki_blobs _[MR example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68946)_ [MR Link](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107730) - [x] Use namespace_ancestry_ids for blobs [MR example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69741) [MR Link](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108135) - [x] Use namespace_ancestry_ids for wiki_blobs [MR example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69741) [MR Link](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/108135) <details><summary>Click to expand process progress </summary> ### Next Steps for this issue #### Validation track - `checkboxes are required steps` - `Bugs can skip Design and Solution Validation` - [x] ~"workflow::validation backlog" [Problem Validation Backlog](https://about.gitlab.com/handbook/product-development-flow/#validation-phase-1-validation-backlog) -@JohnMcGuire - [x] Issue is Up-to-date - [ ] Prioritized in the backlog - [x] ~"workflow::problem validation" [Problem validation](https://about.gitlab.com/handbook/product-development-flow/#validation-phase-2-problem-validation) - @JohnMcGuire - [x] Thorough understanding of the problem - [Problem Validation Template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Problem%20Validation.md) - [Opportunity Canvas](https://about.gitlab.com/handbook/product/product-processes/#opportunity-canvas) - Review of the opportunity canvas - [UX Research](https://gitlab.com/gitlab-org/ux-research/-/blob/master/.gitlab/issue_templates/Problem%20validation.md) - Validate your problem with users - Update issue/epic description - Ensure your issue is up-to-date with the latest understanding of the problem. - Understand and document (in the issue) the goals that people want to accomplish using the [Jobs to be Done (JTBD) framework](https://about.gitlab.com/handbook/engineering/ux/jobs-to-be-done/). - communicate the findings to Product and UX leadership. - PMs are strongly encouraged to open [Dogfooding issues](https://about.gitlab.com/handbook/product/product-processes/#dogfooding-process) - [ ] ~"workflow::ready for design" - @nickbrandt - [ ] ~"workflow::design" [Validation phase 3: Design](https://about.gitlab.com/handbook/product-development-flow/#validation-phase-3-design) - @nickbrandt - [ ] Proposed solution(s) identified and documented - Diverge: explore multiple different approaches as a team - [Think Big session](https://about.gitlab.com/handbook/engineering/ux/thinkbig/). - [document findings in Dovetail](https://about.gitlab.com/handbook/engineering/ux/dovetail/) - Converge: identify a small set of options to validate - [Think Small](https://about.gitlab.com/handbook/engineering/ux/thinkbig/#think-small) session with the team. - Design reviews with team - Low fidelity design ideas - Update issue/epic description with proposed solution - Add Figma design file link or attach design to [GitLab's Design Management](https://docs.gitlab.com/ee/user/project/issues/design_management.html) - Validate approach with help from stakeholders. - Run user validation using any of the [proposed methods](https://about.gitlab.com/handbook/engineering/ux/ux-research/#solution-validation) - [Document your findings in Dovetail](https://about.gitlab.com/handbook/engineering/ux/dovetail/) and appropriate GitLab issue. - Draw inspiration from competitive and adjacent offerings. - [ ] Shared understanding in the team of the proposed solution - Review the proposed solution as a team so that everyone has a chance to contribute - ask questions - raise concerns - suggest alternatives - Review the proposed solution with leadership. - [ ] Confidence in the technical feasibility - Discuss the technical implications with Engineering to ensure that what is being proposed is possible within the desired timeframe. - Engage engineering peers early and often - If the solution is large and complex, consider scheduling a [spike](https://about.gitlab.com/handbook/product/product-processes/#spikes) to mitigate risks and uncover the optimal iteration path. - [ ] Updated issues/epic descriptions - @JohnMcGuire - Ensure issues and epics are up-to-date, so we can continue our work efficiently and asynchronously. - [Experiment definition](https://about.gitlab.com/handbook/engineering/development/growth/#experiment-definition-standards) - Continue Dogfooding process - [ ] ~"workflow::solution validation" - [Validation phase 4: Solution Validation](https://about.gitlab.com/handbook/product-development-flow/#validation-phase-4-solution-validation) - @nickbrandt - [ ] High confidence in the proposed solution - Gather feedback from relevant stakeholders. - Follow [solution validation guidance](https://about.gitlab.com/handbook/engineering/ux/ux-research-training/solution-validation-and-methods/) to gather feedback. - [ ] Documented Solution validation Learnings - Document solution validation findings as [insights in Dovetail](https://about.gitlab.com/handbook/engineering/ux/dovetail/). - Update the [opportunity canvas](https://about.gitlab.com/handbook/product/product-processes/#opportunity-canvas) (if used) with relevant insights. - Update the issue or epic description to contain or link to the findings. #### Build track - [x] ~"workflow::planning breakdown" - @JohnMcGuire - [x] Well-scoped MVC issues - Issues are the [SSOT](https://about.gitlab.com/handbook/values/#single-source-of-truth) for all feature development. - Refine issues into something that can be delivered within a single milestone - Open follow on issues to track work that is de-prioritized - Promote existing issues to Epics and open implementation issues for the upcoming milestone - Review feature issues with contributors - Consider scheduling a POC or engineering investigation issue - Make scope tradeoffs to reach for a [right-sized MVC](https://about.gitlab.com/handbook/product/product-principles/#the-minimal-viable-change-mvc) - Request an [issue review](https://about.gitlab.com/handbook/product/product-processes/#issue-reviews) to ensure communication is clear and have proposed the [right iteration plan](https://about.gitlab.com/handbook/product/product-processes/#iteration-strategies) to execute on the solution. - [ ] Prioritized in Milestone - The team should understand what issues should be delivered during the next milestone - [ ] ~"workflow::ready for development" - @JohnMcGuire - [x] ~"type::bug" ~"type::feature" ~"type::maintenance" - @JohnMcGuire - [ ] ~"Deliverable" - @changzhengliu and @nickbrandt - [ ] Add to Planning Issue - @JohnMcGuire - [ ] Defined Quality Plan -@ebanks - [Quad Planning](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/quad-planning/) - [Test planning](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/test-engineering/#test-planning) - ~"workflow::refinement" - @changzhengliu - as needed, refine the aspects of the original feature - [ ] ~"workflow::in dev" - @changzhengliu - Applied by the engineer after work (including documentation) has begun on the issue. An MR is typically linked to the issue at this point. - [ ] ~"workflow::in review" - Engineering - Applied by an engineer indicating that all MRs required to close an issue are in review. - [ ] ~"workflow::blocked" - Engineering - Applied if at any time during development the issue is blocked. For example: technical issue, open question to PM or PD, cross-group dependency. - [ ] ~"workflow::verification" - Engineering - After the MRs in the issue have been merged, this label is applied signaling the issue needs to be verified in staging or production. - [ ] ~"workflow::awaiting security release" -Engineering - Applied by an engineer after the security issue has passed verification, this label signals that it is ready but awaiting the next monthly security release. - [ ] Close the Issue - Once available in production - [ ] Feature is available to GitLab.com hosted customers - Developer - Code is deployed to production. - [Feature flag(s)](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) enabled. - [ ] Feature is available to self-managed customers - Developer - Code is included in the self-managed release (depending upon the cut-off). - [ ] Stakeholders of a feature will know it's available in production - Developer - After the feature is deployed to production and any needed verification in production is completed, the development team will close the issue. - Prior to the issue being closed, the development team may set the workflow label to workflow::verification or workflow::production for tracking purposes. - Product Manager may follow up with individual [stakeholders](https://about.gitlab.com/handbook/product/product-processes/#what-is-a-stakeholder) to let them know the feature is available. - [ ] Customers will be informed about major changes - @JohnMcGuire - Product Manager follows the instructions in the [template](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/.gitlab/merge_request_templates/Release-Post.md), which will then cause it to appear on the [GitLab.com releases page](https://about.gitlab.com/releases/gitlab-com/) and be part of the release post. </details>