Add author_human filter to merge request analytics
What does this MR do and why?
The DAP impact dashboard compares MR cycle time for Duo-created merge requests with everything else. The work item asks for that baseline to hold only human-authored merge requests. Bots, service accounts and other non-human users, such as dependency-update bots, sit in the non-Duo bucket and pull its baseline down. ClickHouse merge_requests only stores author_id, so there was no way to filter them out.
This MR adds:
non_human_users_dict: a ClickHouse dictionary of user IDs whose latestsiphon_usersrow has auser_typeother thanhuman(0). It followsbanned_users_dictand refreshes every 5 to 60 minutes. Because it usesuser_type != 0, new user types are covered without a migration.author_humanfilter onAnalytics::AggregationEngines::MergeRequests, exposed in GraphQL asauthorHuman. It is true when the merge request has an author who is not in the dictionary. Merge requests with no author count as not human.- Docs: the
authorHumanGLQL filter, and how a merge request is classified as created by Duo. That is the case when an Agent Platform session records that it opened the merge request (link_type = created), whoever the author is.
The filter is meant only for the non-Duo side (createdByDuo = false). Flows that run remotely open merge requests as a service account, which is not human, so filtering the Duo side would drop real Duo merge requests.
Nothing uses the filter yet. Follow-ups:
- glql!569 (merged) adds
authorHumanto the GLQL compiler. - After a GLQL release, bump
@gitlab/query-language-rustandgitlab_query_language, and addauthorHuman = trueto the "Cycle time: all other" and "Merged: all other" panels indap_impact.yaml.
Why a dictionary of non-human users
A siphon_users subquery in the filter expression, measured on gitlab.com for gitlab-org:
| Query | Duration | Data read |
|---|---|---|
| Current baseline | ~200 ms | 17 MB |
Baseline + siphon_users subquery |
~2 s warm, >10 s cold | 1.4 GB |
The dictionary runs that scan once per refresh instead of on every dashboard load. A stored column on merge_requests would need a backfill and a new join in the enrichment view, and it would go stale when a user's type changes.
gitlab.com has about 3 million non-human users and 23 million humans. A dictionary of non-human users is therefore about an eighth of the size a dictionary of humans would be: an estimated 60–120 MB per replica, against 0.5–1 GB.
For database review:
- The dictionary source query is the same
siphon_usersscan, run on each refresh and on each replica. After deploy, checkelement_count,bytes_allocatedandloading_durationinsystem.dictionaries. If the refresh is too heavy, raiseLIFETIME; it doesn't change the result. - Placeholder users from imports are non-human, so imported merge requests not yet reassigned to a real user count as not human. So do merge requests by deleted users, because GitLab moves them to the ghost user.
- On an instance that replicates
merge_requestsbut notusers, the dictionary is empty andauthorHuman = truekeeps every merge request that has an author.
References
Screenshots or screen recordings
Not applicable: backend only, no UI change.
How to set up and validate locally
- Run
bundle exec rake gitlab:clickhouse:migrate. - In ClickHouse, insert a project bot user and reload the dictionary:
INSERT INTO siphon_users (id, user_type) VALUES (424242, 6); SYSTEM RELOAD DICTIONARY non_human_users_dict; SELECT dictHas('non_human_users_dict', toUInt64(424242)); -- 1 - In GraphiQL, query
analytics { mergeRequests(authorHuman: [true]) { ... } }for a group with merge requests synced to ClickHouse. Merge requests by user424242are excluded.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.