Incubation:APM August 20, 2021 Demo
Recording
https://www.youtube.com/watch?v=JIPWNIIXWr0
Current Update
#3 (closed)
DataDog Agent SandboxProject - https://gitlab.com/gitlab-org/incubation-engineering/apm/datadog-sandbox
This allows us to test the agent in isolation:
- Docker compose isolated network & CoreDNS to capture external network traffic.
- Basic (read naive) ClickHouse series storage.
- Grafana configured with ClickHouse data source.
Some issues:
- Docker stats not being collected: suspect this is cgroupsv2 issue and won't be a problem for k8s environments (need to test this)
- Configuration for all o11y telemetry is a little awkward due to inconsistencies, requiring overriding of various config elements.
#4 (closed)
ClickHouse evaluationStarted evaluating ClickHouse. Lots of documentation to be read.
Using clickhouse-go to do some data ingestion via sqlx.
Items of Interest
- Combining o11y telemetry into context aware braids - https://thenewstack.io/modern-observability-is-a-single-braid-of-data/
- OpenTelemetry has built-in datadog exporter - https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/datadogexporter
Up Next
- Further evaluation of ClickHouse - #4 (closed)
- Design a better metrics storage schema
- Investigate correct use of table engines https://clickhouse.tech/docs/en/engines/table-engines/
- Evaluate with time series database benchmarker - https://github.com/timescale/tsbs
- APM Architecture design
- Provide an initial design with assumptions to share.
Edited by Joe Shaw