Advanced search - query work item milestones by special filters

Alisa Frunzarequested to merge
547410-advanced-search-query-work-item-milestones-by-special-filters into master

What does this MR do and why?

This MR adds additional support for GLQL milestone wildcard filters (upcoming, not_upcoming, started, not_started) in Elasticsearch by following these ActiveRecord scopes implementation:

In Elasticsearch, these filters are going to be implemented with the use of milestone_due_date, milestone_start_date and milestone_state fields.

Below is the example mapping (GLQL field => Glql::WorkItemsFinder parameters => Search::Elastic::Filter parameter) of all possible variations for these milestone wildcard fields:

Wildcard Milestone Queries

GLQL Query WorkItems Finder ES Filter Name
milestone = upcoming milestone_wildcard_id: "UPCOMING" milestone_state_filters: [:upcoming]
milestone != upcoming not: {milestone_wildcard_id: "UPCOMING"} milestone_state_filters: [:not_upcoming]
milestone = started milestone_wildcard_id: "STARTED" milestone_state_filters: [:started]
milestone != started not: {milestone_wildcard_id: "STARTED"} milestone_state_filters: [:not_started]

Combined Milestone Queries

GLQL Query WorkItems Finder ES Filter Name
milestone != upcoming and milestone = started milestone_wildcard_id: "STARTED", not: {milestone_wildcard_id: "UPCOMING"} milestone_state_filters: [:started, :not_upcoming]
milestone = upcoming and milestone != started milestone_wildcard_id: "UPCOMING", not: {milestone_wildcard_id: "STARTED"} milestone_state_filters: [:upcoming, :not_started]

References

Screenshots or screen recordings

Before After

How to set up and validate locally

  1. Follow this guide to have ES enabled locally for GDK
  2. GLQL relies on crypto.subtle, which is unavailable in insecure contexts (except for localhost), it won’t work if you're using a custom URL like http://gdk.test:3000 locally. To resolve this, one of the options is to enable HTTPS in GDK by following this link.
  3. Enable the following feature flags in rails c:
=> Feature.enable(:glql_integration)
=> Feature.enable(:glql_work_items)
=> Feature.enable(:glql_es_integration)
  1. Navigate to any issue comment or description field where Markdown is supported
  2. Add the following GLQL query into the field:
```glql
display: table
fields: title, labels, milestone
query: group = "gitlab-org" and milestone != upcoming
```
  1. In order to test different milestone wildcard variations, you will have to create a couple of milestones and assign them to various issues (make sure that all data is properly indexed in ES before starting with tests (see point 7)). Here are some of the scenarios I followed:
  • Active milestone with start_date and end_dates in future (upcoming, not_started)
  • Active milestone with no start_date and end_date in the future (started, not_upcoming)
  • Active milestone with start_date < now and no end_date (started, not_upcoming)
  • Active milestone with start_date < now and end_date in the future (started, not_upcoming)
  • Closed milestone with start_date < now and end_date in the future (should not be returned in results since it's closed)
  1. I noticed that I was initially struggling with testing things locally, the results seemed inconsistent. After some investigation, I realized that the changes I was doing to milestones were not indexed properly locally. Restarting sidekiq seemed to help gdk restart rails-background-jobs. Also, I have Kibana running locally with docker, it helps with verifying the data a lot via dev console http://localhost:5601/app/dev_tools#/console 
$ docker run -d --name kibana -p 5601:5601 \
  -e ELASTICSEARCH_HOSTS=http://host.docker.internal:9200 \
  -v kibana_data:/usr/share/kibana/data \
  docker.elastic.co/kibana/kibana:8.7.0

$ docker start kibana

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #547410 (closed)

Edited by Alisa Frunza

Merge request reports