API to get all project/group members returns duplicates

Summary

When utilizing the /members/all API endpoint, duplicate users will be returned if they are members of both the project/group and the ancestral group. This was mentioned in the MR for this feature. Customer is requesting this behavior be changed to return unique users with no duplicates. Unsure if this is a bug or not, so submitting this issue to start the discussion.

Steps to reproduce

1. Create a project that has members that are also members of an ancestral group
2. Query the project via the /members/all API endpoint, i.e. curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/:id/members/all
3. The users returned will have duplicates

What is the current bug behavior?

Duplicate users are being returned with the /members/all API endpoint for projects and groups. These duplicate users can potentially have different values for access_level and expires_at.

What is the expected correct behavior?

Return unique users for the project/group with the access level and expires_at data that applies for that project/group

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		Ubuntu 16.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.4.4p296
Gem Version:	2.7.6
Bundler Version:1.16.2
Rake Version:	12.3.1
Redis Version:	3.2.11
Git Version:	2.18.0
Sidekiq Version:5.1.3
Go Version:	unknown

GitLab information
Version:	11.2.3-ee
Revision:	aadca99
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
DB Version:	9.6.8
URL:		http://198.199.92.126
HTTP Clone URL:	http://198.199.92.126/some-group/some-project.git
SSH Clone URL:	git@198.199.92.126:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	8.1.1
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab Shell ...

GitLab Shell version >= 8.1.1 ? ... OK (8.1.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
46/1 ... repository is empty
46/2 ... repository is empty
46/3 ... repository is empty
46/4 ... repository is empty
47/5 ... repository is empty
47/6 ... repository is empty
47/7 ... repository is empty
47/8 ... repository is empty
47/9 ... repository is empty
47/10 ... repository is empty
47/11 ... repository is empty
48/12 ... repository is empty
48/13 ... repository is empty
48/14 ... repository is empty
48/15 ... repository is empty
48/16 ... repository is empty
48/17 ... repository is empty
49/18 ... repository is empty
49/19 ... repository is empty
49/20 ... repository is empty
49/21 ... repository is empty
49/22 ... ok
49/23 ... repository is empty
49/24 ... repository is empty
49/25 ... repository is empty
49/26 ... repository is empty
49/27 ... repository is empty
50/28 ... repository is empty
50/29 ... repository is empty
50/30 ... repository is empty
50/31 ... repository is empty
50/32 ... repository is empty
50/33 ... repository is empty
50/34 ... repository is empty
51/35 ... repository is empty
51/36 ... repository is empty
51/37 ... repository is empty
51/38 ... repository is empty
51/39 ... repository is empty
51/40 ... repository is empty
51/41 ... repository is empty
51/42 ... repository is empty
51/43 ... repository is empty
52/44 ... repository is empty
52/45 ... repository is empty
52/46 ... repository is empty
52/47 ... repository is empty
52/48 ... repository is empty
52/49 ... repository is empty
52/50 ... repository is empty
52/51 ... repository is empty
53/52 ... repository is empty
53/53 ... repository is empty
53/54 ... repository is empty
53/55 ... repository is empty
53/56 ... repository is empty
53/57 ... repository is empty
53/58 ... repository is empty
53/59 ... repository is empty
54/60 ... repository is empty
54/61 ... repository is empty
54/62 ... repository is empty
55/63 ... repository is empty
55/64 ... repository is empty
56/65 ... repository is empty
56/66 ... repository is empty
56/67 ... repository is empty
57/68 ... repository is empty
57/69 ... repository is empty
57/70 ... repository is empty
57/71 ... repository is empty
57/72 ... repository is empty
57/73 ... repository is empty
58/74 ... repository is empty
58/75 ... repository is empty
58/76 ... repository is empty
58/77 ... repository is empty
58/78 ... repository is empty
59/79 ... repository is empty
59/80 ... repository is empty
59/81 ... repository is empty
59/82 ... repository is empty
59/83 ... repository is empty
59/84 ... repository is empty
59/85 ... repository is empty
60/86 ... repository is empty
60/87 ... repository is empty
60/88 ... repository is empty
60/89 ... repository is empty
60/90 ... repository is empty
1/91 ... repository is empty
60/92 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Reply by email is disabled in config/gitlab.yml
Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
46/1 ... yes
46/2 ... yes
46/3 ... yes
46/4 ... yes
47/5 ... yes
47/6 ... yes
47/7 ... yes
47/8 ... yes
47/9 ... yes
47/10 ... yes
47/11 ... yes
48/12 ... yes
48/13 ... yes
48/14 ... yes
48/15 ... yes
48/16 ... yes
48/17 ... yes
49/18 ... yes
49/19 ... yes
49/20 ... yes
49/21 ... yes
49/22 ... yes
49/23 ... yes
49/24 ... yes
49/25 ... yes
49/26 ... yes
49/27 ... yes
50/28 ... yes
50/29 ... yes
50/30 ... yes
50/31 ... yes
50/32 ... yes
50/33 ... yes
50/34 ... yes
51/35 ... yes
51/36 ... yes
51/37 ... yes
51/38 ... yes
51/39 ... yes
51/40 ... yes
51/41 ... yes
51/42 ... yes
51/43 ... yes
52/44 ... yes
52/45 ... yes
52/46 ... yes
52/47 ... yes
52/48 ... yes
52/49 ... yes
52/50 ... yes
52/51 ... yes
53/52 ... yes
53/53 ... yes
53/54 ... yes
53/55 ... yes
53/56 ... yes
53/57 ... yes
53/58 ... yes
53/59 ... yes
54/60 ... yes
54/61 ... yes
54/62 ... yes
55/63 ... yes
55/64 ... yes
56/65 ... yes
56/66 ... yes
56/67 ... yes
57/68 ... yes
57/69 ... yes
57/70 ... yes
57/71 ... yes
57/72 ... yes
57/73 ... yes
58/74 ... yes
58/75 ... yes
58/76 ... yes
58/77 ... yes
58/78 ... yes
59/79 ... yes
59/80 ... yes
59/81 ... yes
59/82 ... yes
59/83 ... yes
59/84 ... yes
59/85 ... yes
60/86 ... yes
60/87 ... yes
60/88 ... yes
60/89 ... yes
60/90 ... yes
1/91 ... yes
60/92 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.4.4)
Git version >= 2.9.5 ? ... yes (2.18.0)
Git user has default SSH configuration? ... yes
Active users: ... 45
Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)

Checking GitLab ... Finished