## Summary To improve GitLab by analysing usage, we want to collect unique IDs. In order to respect the privacy of our users, we need to pseudonymize IDs and PII data. We want to do this by hashing this data with a salt on the collector layer. ## Where can I post my question or give feedback? We welcome any question or feedback to improve the proposal. For simplicity and not to pollute this epic with comments, we kindly ask you to use the Question & Feedback in our forum: https://forum.gitlab.com/t/updates-to-de-identifying-service-usage-data/59456 after reading the proposal. ## Current State For our Snowplow Events, we do not have an association with either a `User ID`, `Namespace ID`, or `Project ID`. It's therefore not possible to group these events by ID and analyse the usage. E.g. an event that we receive X times, could be emitted by only 1 user or Y different users. We therefore can't make a good judgement about our product usage and where to invest our resources when building and optimising GitLab. While we already have qualitative measurements in place, we also want to work on quantitative measurements. However, we do not intent to reveal a user's identity, that's why we want to pseudonymize any related user data. ## Requirements 1. Pseudonymize PII data before it is stored and made available for broader access 1. Pseudonymization has to be deterministic in order to allow for historical analysis 1. The de-identification process will include the following data: https://gitlab.com/gitlab-org/gitlab/-/issues/336779#considered-data-for-pseudonymization ## Proposed Solution ### One-way Hashing We investigated different solutions as part of a technical proposal and want to design the system the following way: 1. remove PII from the incoming data stream replace it with an 'AnonPII' (which is required for counting unique events) 1. 'AnonPII' is created by sending the PII and a secret SALT through a hashing function (proposed [SHA-2/256](https://en.wikipedia.org/wiki/SHA-2)) ##### Pros - Hashing requires minimal computational power to pseudonymize data - Does not have strict size limitations regarding processed data ##### Cons - In case of the salt leakage in order to still protect already collected data, new secret would have to be issued, all existing data would need to be hashed once more with it, while all new incoming data would need to be hashed twice first with the old salt, and then with the new one in order to provide compatibility with already collected data. - The hashing is not visible to our users when sending tracking data on the client. However, we mitigate this as our infra setup of the collector is [publicly available](https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/) ### Hashing on the collector layer Instead of performing hashing on the sender side (GitLab Instance) we do it on the recipient, the collector. This has the following reasons: 1. We don't need to distribute the secret salt to a GitLab instance 1. In case of a leakage of the secret, we can react quickly as it's not constrained by our release process 1. It is easier for us technically to deal with both Frontend & Backend events in one place ```mermaid graph LR subgraph data-warehouse[Data Warehouse] subgraph private-access[Private Access Layer] DWR[RAW Database] end subgraph general-access[General Access Layer] DWR-->PREP[Prep Database] PREP-->PROD[Production Database] end end SAAS-->|Snowplow|SPCollector subgraph Snowplow[Snowplow AWS] SPCollector[Collectors]-->SPEnrich[Enrichers]-->SPPseudo[Pseudonimizer]-->DWR end ``` ## Pseudonymization of URLs As part of this initiative, we also want to remove any PII data from Page URLs that get tracked. More info in the sub-epic https://gitlab.com/groups/gitlab-org/-/epics/6551 ## Reference - Infra work: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14021 - Handbook page: https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/88412