Unable to move issue to another project - target project not in the list

Problem to solve

Sometimes it is not possible to move an issue from one project to another, because the target project cannot be found in the list.

Further details

The ability to move an issue from one project to another was introduced in 8.6 (see the implementation issue and the blog post). Now, on the Issue page, there is a button at the bottom of the right-side controls to "Move issue". If you click that button, a dropdown appears for you to select the target project. If you don't see what you are looking for in the list of 20 projects, you can enter text in the "Search project" input. Search results are also limited to 20 projects. So if you aren't able to find what you are looking for in the default list, or with search, then you cannot move the issue.

Example

1. Go to https://gitlab.com/gitlab-org/manage/-/issues/16361 (internal link)
2. Click the button to "Move issue"
3. Attempt to find the gitlab project - it apparently cannot be found

Proposal

Backend should sort the "Search project" response by best matches, so the top 20 best matches are displayed (instead of an arbitrary 20 results from all the matches).

Performance

Backend should continue to limit the number of projects returned, for performance reasons.

Permissions and Security

No permissions or security implications.

What does success look like, and how can we measure that?

The example issue above can be successfully moved to the gitlab project.

Links / references

• Problem encountered in the wild: https://gitlab.com/gitlab-org/manage/-/issues/16361#note_312033246

added typefeature workflowproblem validation labels

changed the description

@grzesiek you built this awesome feature, so I was hoping you could help me figure out which group it belongs to?

To be honest I'm not sure how the dropdown we have right now is implemented. When I built this feature we were rendering this in HAML, and today I suppose this is being rendered differently, presumably with Vue. I expect some backend endpoint for getting projects that a user has access to, to be implemented somewhere, and there is could be some LIMIT modifier added to the SQL query. Perhaps we limit the amount of projects to some number, and because of ordering not being defined explicitly we do not show a few projects. This is only an assumption, I might be completely wrong here - it simply needs to be checked in code @djensen.

I think that it belongs to Plan::Project management.

added groupproject management label

mentioned in issue #212794 (closed)

mentioned in issue #212807 (closed)

changed the description

@donaldcook @gweaver hopefully you agree this is groupproject management.

FYI, this is affecting us at GitLab. Searching for "gitlab" doesn't return the gitlab project as an option, so we are unable to move issues to there. I don't think it's technically a ~bug, because the behavior is deliberate. This can probably be resolved by returning the best matches.

Thanks for the issue @djensen. groupproject management is correct.

We're working on a refactor of the sidebar and will make sure we account for this when updating move issue. #212893 (closed)

Side note, workaround for this is using the /move quick action.

@donaldcook thanks, that worked!

Offshoot from that: I can't find where the /move command (or others, like /cc) are described in the docs. Do you happen to know where that documentation is?

@djensen https://docs.gitlab.com/ee/user/project/quick_actions.html

This is a duplicate / related to #20444 (closed)

The problem is that searching for "gitlab" in our group returns too many results. I fixed the other issue by allowing searching by namespace so you should be able to search "gitlab-org/gitlab". Though it still shows a few other projects because it still matches projects like "gitlab-org/gitlab-puma", etc..

Another suggestion in that other issue was better ordering of results. Maybe by number of stars or last activity.

I'm also not sure why Winnie's fork is showing up here:

This is probably a project shared to a group? I'm not sure we even allow moving to shared projects.

It would also be good if our dropdown component supports pagination / infinite scroll where we see the succeeding results as we scroll down. Maybe a quick fix is to just increase the limit? It's currently at 20 which I think is kind of low.

Also, we now have a Vue version of this component and maybe that one supports pagination already.

This is a duplicate / related to #20444 (closed)

@engwan ah, you're right. I didn't find that before I created this issue.

I fixed the other issue by allowing searching by namespace so you should be able to search "gitlab-org/gitlab".

When I search for "gitlab-org/gitlab", the gitlab project doesn't appear in the 20 results

Another suggestion in that other issue was better ordering of results. Maybe by number of stars or last activity.

Or by % match? (I've used the similar_text-ruby gem for something like this before.) I think this would be a better user experience than pagination or infinite scroll, because it would shift the burden from the user to the backend. Not sure if that would be feasible on the backend though.

When I search for "gitlab-org/gitlab", the gitlab project doesn't appear in the 20 results

I see, it looks like we now have more projects matching the gitlab-org/gitlab path

Or by % match? (I've used the similar_text-ruby gem for something like this before.) I think this would be a better user experience than pagination or infinite scroll, because it would shift the burden from the user to the backend. Not sure if that would be feasible on the backend though.

I don't think this would be feasible because we won't be able to use DB LIMIT in this case. We'd have to retrieve all matches from the DB, then compute the % match, then return the first N results.

We'd have to retrieve all matches from the DB

@engwan not necessarily - what if the query included a LIKE clause to match for the input?

Yeah, that's what I meant by all matches. The number of matches could still be a large number.

But now that I think about it, maybe it won't be a very large number

@engwan you're right, we should protect performance with a LIMIT even if it won't usually apply. And where it does apply, there is that risk of excluding the user's intended record. Could we solve that with two queries?

1. exact match, which would be displayed first in the results if it exists
2. LIKE match with a LIMIT

It is possible. Though it means a bit more code changes.

#search is currently implemented as an ActiveRecord scope so it can be chained, etc..

To do that we'd have to make changes in the controller that serves the data just for the dropdown. So that #search can still be a scope.

@engwan maybe that could be a separate/future enhancement then. In the meanwhile I'd support a quick fix like you talked about below. Thanks for talking this through with me

Setting devopsplan based on groupproject management.

added devopsplan label

As a quick fix, I'd like to improve the ordering of results.

@hollyreynolds what do you think is the better sort order for this? number of stars? last activity? something else?

@engwan Are you asking about the default order of results, the order for when a character has been entered, or both? Also, I noticed that the space around the / in the namespace seems to matter. For example, when I tried searching for GitLab.org/gitlab-workhorse it didn't find the project, but GitLab.org / gitlab-workhorse did. I'd love it if both could provide the same results?

As far as the result order, @djensen when you search for a project to move to, can you share what that process looks like? For example, do you already know the full namespace of the project you're looking for, or just a portion? If just a portion, what portion do you feel is most memorable? If you know the full name, how would you expect that list to filter based on what you enter?

Are you asking about the default order of results, the order for when a character has been entered, or both?

Both. It's easier to make them the same for now.

I know we can be a bit smarter when there are characters entered like "best match", but we can't do that right now due to how our search works. We'd probably need postgres full text search or something like that to implement that.

For example, when I tried searching for GitLab.org/gitlab-workhorse it didn't find the project, but GitLab.org / gitlab-workhorse did. I'd love it if both could provide the same results?

Yeah, these are problems with our search which would also be solved by using full text search. Right now we do a simple ILIKE query per word in the search term.

So as a quick iteration, I would like to just change the sort order to something better than the current one. Currently, it's ordered by project name.

@engwan I see, thanks for clarifying! Let's go with popularity order for now then as it sounds like we are limited on what we can offer with this component. Do we have the ability to define levels in the sort? For example, first sort by keyword, then popularity, then date changed?

Do we have the ability to define levels in the sort? For example, first sort by keyword, then popularity, then date changed?

We can define multiple sorts but the next one would only be used if there's a tie. So if we do stars_count DESC, last_activity_at DESC, last activity will only be considered if projects have the same number of stars.

@engwan Thanks again! Stars/popularity for the default sort order is good for now then. Please let me know if I can help with anything further!

The projects are now sorted by popularity:

I'll keep this issue open for further improving the sorting of results.

@engwan that sounds like a great approach to the iteration. Thanks for executing the first step already!

mentioned in issue #213483 (closed)

@djensen when you search for a project to move to, can you share what that process looks like?

@hollyreynolds my process, as far as I'm consciously aware of it:

1. Visually scan for the project in the given list (which is only 3-4 projects). Don't scroll the list - if a high-activity project isn't in the top 3-4 then it feels like there's no point, and even less point for a low-activity project.
2. Search by project name ("gitlab"). This feels easy because the project name is memorable and top-of-mind. This is the step where I would usually expect to be able to find my target.
3. Search with namespace ("gitlab-org/gitlab") if I need to. This feels harder, cognitively, because I have to think about the namespace (which is less memorable) and do more typing.

how would you expect that list to filter

In a perfect world, I would expect the list to be filtered by highest activity by default, then by percentage match for search. But both of those might have some technical complexity, so to simplify maybe latest activity by default, and exact match for search?

Hopefully that helps

@djensen That's very helpful, thanks!!

mentioned in merge request !29043 (merged)

mentioned in issue #214175 (closed)

mentioned in merge request !29766 (merged)

mentioned in issue #214945 (closed)

mentioned in issue #215811 (closed)

mentioned in issue #217041 (closed)

mentioned in issue #221043 (closed)

mentioned in issue gitlab-org/manage/general-discussion#17237

mentioned in issue #225546 (closed)

marked this issue as related to #225546 (closed)

Also running into this when attempting to create an issue from the issue board's Open list.

The only query that would work is a workaround - searching for end-to-end. See Slack thread. https://gitlab.slack.com/archives/CETG54GQ0/p1596048327350300

@ahegyi you recently introduced similarity sorting in the group projects API to solve this project-searching problem in some other contexts. It would be great to implement that for the board's "New issue" feature like @mikelewis describes above, and for the issue's "Move issue" feature like this issue describes. I'm hoping you can help with 2 questions:

1. Should the board's "New issue" feature be handled under a separate issue? Or will it be handled by the same change made for this issue?
2. Could you estimate this issue?

@djensen,

Should the board's "New issue" feature be handled under a separate issue? Or will it be handled by the same change made for this issue?

Yes, it'd make sense to create a separate issue. As far as I can see it's using the group projects API endpoint which already has an option for similarity search.

Note: The endpoint for moving the issue is different (see screenshot in the description): https://gitlab.com/-/autocomplete/projects?project_id=278964&search=git

Could you estimate this issue?

Fixing the "New issue" is a 1. FE only change, we just need to add order_by=similarity URL parameter.

There is a small documentation about the parameter under the table: https://docs.gitlab.com/ee/api/groups.html#list-a-groups-projects

---

For this issue I'll set 3 as the weight since we might need more work to have similarity search working.

Yes, it'd make sense to create a separate issue.

@ahegyi done, and thanks for the great writeup

set weight to 3

For the "Move Issue" dropdown, it is slightly different since we're actually listing all projects where the user can create issues. So these aren't constrained under a single group and can have a much larger result set. https://gitlab.com/gitlab-org/gitlab/blob/6d59a7a3b8233d7e30af2f9a160a16b683ce2d42/app/finders/autocomplete/move_to_project_finder.rb#L26-32

So I'm not sure if we can use the similarity order here because that can be slow for large data.

Right, we might need to make some tweaking with the scoring.

Original query:

SELECT "projects".* FROM "projects" INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id INNER JOIN "routes" ON "routes"."source_type" = 'Project' AND "routes"."source_id" = "projects"."id" WHERE "project_authorizations"."user_id" = 4156052 AND (project_authorizations.access_level >= 20) AND "projects"."archived" = FALSE AND ("project_features"."issues_access_level" > 0 OR "project_features"."issues_access_level" IS NULL) AND (("routes"."path" ILIKE '%gitl%' OR "routes"."name" ILIKE '%gitl%') OR "projects"."description" ILIKE '%gitl%') AND "projects"."id" != 278964 ORDER BY "projects"."star_count" DESC LIMIT 20 

82ms

https://explain.depesz.com/s/y8dX

New query:

SELECT "projects".* FROM "projects" INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id" LEFT JOIN project_features ON projects.id = project_features.project_id INNER JOIN "routes" ON "routes"."source_type" = 'Project' AND "routes"."source_id" = "projects"."id" WHERE "project_authorizations"."user_id" = 4156052 AND (project_authorizations.access_level >= 20) AND "projects"."archived" = FALSE AND ("project_features"."issues_access_level" > 0 OR "project_features"."issues_access_level" IS NULL) AND (("routes"."path" ILIKE '%gitl%' OR "routes"."name" ILIKE '%gitl%') OR "projects"."description" ILIKE '%gitl%') AND "projects"."id" != 278964 ORDER BY (similarity(COALESCE("projects"."path", ''), 'git') * cast('1' AS numeric)) + (similarity(COALESCE("projects"."name", ''), 'git') * cast('0.7' AS numeric)) + (similarity(COALESCE("projects"."description", ''), 'git') * cast('0.2' AS numeric)) DESC,
         "projects"."id"
LIMIT 20 

130ms

https://explain.depesz.com/s/CdlB

We could probably remove the similarity scoring on the description column, that takes away ~30ms.

@donaldcook are we addressing this as part of &3227?

@hollyreynolds Yup this is one of the problem areas that we need to account for.

mentioned in issue #233394 (closed)

marked this issue as related to #233394 (closed)

added sectiondev label

changed milestone to %Backlog

added [deprecated] Accepting merge requests label

added workflowstart label and removed workflowproblem validation label