Skip to content

WIP: Change namespace billable_members_count to include all descendants

What does this MR do?

cc/ @amandarueda Relevant for you and our discussion

Currently billable_members_count only includes members in the current namespace plus descendant namespaces. It does not include members of projects within the group hierarchy. This change includes direct and indirect users of any and all groups and projects within the hierarchy, which is more indicative of total billable users.

This change does not include counting members of groups that a given project in the hierarchy may be shared with. We currently don't have an easy mechanism for pulling this information. We'll have to do a little more digging to figure out a performant way to do that.

Concerns

I'm slightly concerned about the performance on this change. As you can see below it's already more than 3.5x slower with just 4 users in the system. What happens on GitLab.com?

Before we were doing a fairly simple query:

   (1.5ms)  SELECT COUNT(*) FROM "users" WHERE "users"."id" IN (SELECT "members"."user_id" FROM "members" LEFT OUTER JOIN "users" ON "members"."user_id" = "users"."id" WHERE "members"."type" IN ('GroupMember') AND "members"."source_type" = $1 AND "users"."state" = $2 AND "members"."requested_at" IS NULL AND "members"."source_id" IN (WITH RECURSIVE "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."id" = 8)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "id" FROM "base_and_descendants" AS "namespaces"))  [["source_type", "Namespace"], ["state", "active"]]

Now we're doing a bit more complex query:

   (5.8ms)  SELECT COUNT(*) FROM ((SELECT "users".* FROM "users" WHERE "users"."id" IN (SELECT "members"."user_id" FROM "members" LEFT OUTER JOIN "users" ON "members"."user_id" = "users"."id" WHERE "members"."type" IN ('GroupMember') AND "members"."source_type" = 'Namespace' AND "users"."state" = 'active' AND "members"."requested_at" IS NULL AND "members"."source_id" IN (WITH RECURSIVE "base_and_ancestors" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."id" = 8)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_ancestors" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."id" = "base_and_ancestors"."parent_id")), "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."id" = 8)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "namespaces"."id" FROM ((SELECT "namespaces".* FROM "base_and_ancestors" AS "namespaces" WHERE "namespaces"."type" IN ('Group'))
UNION
(SELECT "namespaces".* FROM "base_and_descendants" AS "namespaces" WHERE "namespaces"."type" IN ('Group'))) namespaces WHERE "namespaces"."type" IN ('Group'))))
UNION
(SELECT "users".* FROM "users" INNER JOIN "members" ON "members"."user_id" = "users"."id" AND "members"."type" IN ('ProjectMember') AND "members"."source_type" = 'Project' AND "members"."requested_at" IS NULL INNER JOIN "projects" ON "projects"."id" = "members"."source_id" INNER JOIN "namespaces" ON "namespaces"."id" = "projects"."namespace_id" AND "namespaces"."type" IN ('Group') AND "namespaces"."type" = 'Group' WHERE "namespaces"."id" IN (WITH RECURSIVE "base_and_descendants" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."id" = 8)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_descendants" WHERE "namespaces"."type" IN ('Group') AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "id" FROM "base_and_descendants" AS "namespaces"))) users

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Drew Blessing

Merge request reports