Incubation:APM Project/environment integration
Correlating APM data with projects/environments for auth purposes and to allow deep linking from projects/environments in GitLab to the relevant dashboard.
Summary
Per agent running on a host, we need to correlate an api key, GitLab project and (optionally) environment. Once validated, subsequent APM requests with an API key and hostname will have the associated project/env ID stored in the database.
Host tags can be configured in DataDog to expose the project/env IDs.
For DataDog initial /validate
requests, the API key can be validated. This request doesn't have the other host tags.
DataDog then sends an /intake
request with the host tags, we can store a session against the host and api key to lookup the project and environment associated, that we collect from the /intake
URL.
Agent
The agent is configured with an API key, which will be an existing key in the users GitLab (access token, project token etc.)
The agent can also be configured with "global" tags that are always sent. We could have something like this:
GITLAB_PROJECT_ID: 27
GITLAB_ENVIRONMENT_ID: 2
In the DataDog agent global tags can be specified via env vars DD_TAGS
, DD_EXTRA_TAGS
or via the tags
attribute of the DataDog Helm chart.
GITLAB_PROJECT_ID
would be required, and would be cross checked against the API, verifying that the user token has certain permissions against the project (e.g. Developer
access). This will be needed in the case of public projects, we don't want users to be able to submit data for any public project.
GITLAB_ENVIRONMENT_ID
will refer to an environment in the project. This is optional and used to scope the data to a particular environment.
Both IDs can either be the integer ID or the "slug" of the ID from the URL.
DataDog API
The DataDog agent sends global tags to an undocumented /intake
url, as well as a large amount of other metadata during the lifecycle of the agent.
Example `/intake` JSON
{
"method": "POST",
"path": "/intake/",
"queryStringParameters": {
"api_key": [
"123456"
]
},
"headers": {
"Host": [
"mock-server-app:1080"
],
"User-Agent": [
"datadog-agent/7.30.1"
],
"Content-Type": [
"application/json"
],
"Dd-Agent-Version": [
"7.30.1"
],
"Dd-Api-Key": [
"123456"
],
"Accept-Encoding": [
"gzip"
],
"content-length": [
"3905"
]
},
"keepAlive": true,
"secure": true,
"body": {
"apiKey": "123456",
"agentVersion": "7.30.1",
"uuid": "4c4c4544-0047-3110-8053-b8c04f334633",
"internalHostname": "joe-xps159500",
"os": "linux",
"agent-flavor": "agent",
"python": "3.8.10 (default, Aug 19 2021, 16:19:59) [GCC 4.7.2]",
"systemStats": {
"cpuCores": 1,
"machine": "amd64",
"platform": "linux",
"pythonV": "3.8.10",
"processor": "Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz",
"macV": [
"",
"",
""
],
"nixV": [
"ubuntu",
"21.04",
""
],
"fbsdV": [
"",
"",
""
],
"winV": [
"",
"",
""
]
},
"meta": {
"socket-hostname": "c2c45179bd82",
"timezones": [
"UTC"
],
"socket-fqdn": "c2c45179bd82",
"ec2-hostname": "",
"hostname": "joe-xps159500",
"host_aliases": [],
"instance-id": ""
},
"host-tags": {
"system": [
"GITLAB_ENVIRONMENT_ID:foo",
"GITLAB_PROJECT_ID:23"
]
},
"container-meta": {
"docker_swarm": "inactive",
"docker_version": "20.10.9"
},
"network": null,
"logs": {
"transport": "HTTP"
},
"install-method": {
"tool": "docker",
"tool_version": "docker",
"installer_version": "docker"
},
"proxy-info": {
"no-proxy-nonexact-match": false,
"proxy-behavior-changed": false
},
"resources": {
"processes": {
"snaps": [
[
1635591342,
[
[
"dd-agent",
0,
0.8717030752125511,
2226495488,
291139584,
"clickhouse-server",
1
],
[
"root",
0,
0.4094167822126902,
2488012800,
136740864,
"agent",
1
],
[
"root",
0,
0.30215704170130187,
3461349376,
100917248,
"dockerd",
1
],
[
"root",
0,
0.25860806633474115,
1462378496,
86372352,
"security-agent",
1
],
[
"root",
0,
0.2457923301314024,
8000536576,
82092032,
"containerd-shim-runc-v2",
11
],
[
"root",
0,
0.22370511395703577,
454492160,
74715136,
"pamac-daemon",
1
],
[
"root",
0,
0.21680055942834703,
129081344,
72409088,
"systemd-journald",
1
],
[
"root",
0,
0.20967525537653858,
2313486336,
70029312,
"process-agent",
1
],
[
"dd-agent",
0,
0.1828787160422848,
397987840,
61079552,
"clickhouse-watchdog",
1
],
[
"root",
0,
0.14589090705733773,
16769531904,
48726016,
"docker-proxy",
14
],
[
"root",
0,
0.11491239064620498,
1902592000,
38379520,
"trace-agent",
1
],
[
"root",
0,
0.11193227208408889,
2353053696,
37384192,
"containerd",
1
],
[
"root",
0,
0.10463527395874289,
770326528,
34947072,
"coredns",
1
],
[
"root",
0,
0.09569491827239461,
2413899776,
31961088,
"snapd",
1
],
[
"root",
0,
0.06599184355039797,
417206272,
22040576,
"NetworkManager",
1
],
[
"root",
0,
0.044260279385502005,
403755008,
14782464,
"udisksd",
1
],
[
"root",
0,
0.041255633098594834,
328220672,
13778944,
"gdm-session-worker [pam/gdm-password]",
1
],
[
"root",
0,
0.036619893113080906,
170344448,
12230656,
"systemd",
1
],
[
"root",
0,
0.03577368660778868,
323530752,
11948032,
"ModemManager",
1
],
[
"root",
0,
0.0338850517988756,
34758656,
11317248,
"systemd-udevd",
1
]
]
]
]
},
"meta": {
"host": "joe-xps159500"
}
},
"gohai": "{\"cpu\":{\"cache_size\":\"16384 KB\",\"cpu_cores\":\"8\",\"cpu_logical_processors\":\"16\",\"family\":\"6\",\"mhz\":\"3500.113\",\"model\":\"165\",\"model_name\":\"Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz\",\"stepping\":\"2\",\"vendor_id\":\"GenuineIntel\"},\"filesystem\":[{\"kb_size\":\"947746588\",\"mounted_on\":\"/\",\"name\":\"overlay\"},{\"kb_size\":\"65536\",\"mounted_on\":\"/dev\",\"name\":\"tmpfs\"},{\"kb_size\":\"65536\",\"mounted_on\":\"/dev/shm\",\"name\":\"shm\"},{\"kb_size\":\"947746588\",\"mounted_on\":\"/etc/hosts\",\"name\":\"/dev/mapper/luks-b0c127c2-6224-4046-957c-a3efc822fe21\"},{\"kb_size\":\"16308076\",\"mounted_on\":\"/var/run/docker.sock\",\"name\":\"run\"},{\"kb_size\":\"16308076\",\"mounted_on\":\"/proc/asound\",\"name\":\"tmpfs\"},{\"kb_size\":\"16308076\",\"mounted_on\":\"/proc/acpi\",\"name\":\"tmpfs\"},{\"kb_size\":\"16308076\",\"mounted_on\":\"/proc/scsi\",\"name\":\"tmpfs\"},{\"kb_size\":\"16308076\",\"mounted_on\":\"/sys/firmware\",\"name\":\"tmpfs\"}],\"memory\":{\"swap_total\":\"35881832kB\",\"total\":\"32616152kB\"},\"network\":null,\"platform\":{\"GOOARCH\":\"amd64\",\"GOOS\":\"linux\",\"goV\":\"1.15.13\",\"hardware_platform\":\"x86_64\",\"hostname\":\"c2c45179bd82\",\"kernel_name\":\"Linux\",\"kernel_release\":\"5.13.19-2-MANJARO\",\"kernel_version\":\"#1 SMP PREEMPT Sun Sep 19 21:31:53 UTC 2021\",\"machine\":\"x86_64\",\"os\":\"GNU/Linux\",\"processor\":\"x86_64\",\"pythonV\":\"3.8.10\"}}"
}
}
In the above example we get the host specific tags:
"host-tags": {
"system": [
"GITLAB_ENVIRONMENT_ID:foo",
"GITLAB_PROJECT_ID:23"
]
},
/intake
with host-tags
specifically is only sent once when the agent starts, subsequent /intake
requests contain various other bits of metadata.
GitLab API
- Projects can be retrieved like so - https://docs.gitlab.com/ee/api/projects.html#get-single-project
- Environments can be accessed like so - https://docs.gitlab.com/ee/api/environments.html#get-a-specific-environment
Sequence diagram for /intake
:
sequenceDiagram
participant Agent
participant Gateway
participant GitLab
Agent->>Gateway: Post /intake invalid api key or invalid project ID
Gateway->>GitLab: Get project with api key
GitLab-->>Gateway: 401 Unauthorized
Gateway-->>Agent: 401 Unauthorized
Agent->>Gateway: Post /intake with valid key for project (environment optional)
Gateway->>GitLab: Get project with api key
GitLab-->>Gateway: 200 project JSON
alt token is not dev level
Gateway-->>Agent: 401 Unauthorized
else token is dev level
opt Validate environment ID
Gateway->>GitLab: get environment in project
GitLab-->>Gateway: 200 content or 404 if not exists
end
Gateway-->>Agent: 200
end
In the project response, we can view permissions in this attribute:
"permissions": {
"project_access": {
"access_level": 30,
"notification_level": 3
},
"group_access": null
}
Checking that permissions.project_access.access_level >= 30
should be sufficient to validate that the user or project bot account has a minimum of the Developer level permission.
ClickHouse
ClickHouse schemas will need project_id int
and environment_id int (nullable)
columns adding and the sorting key should probably include these fields.