Skip to content

Auto-generate observability traces from deployed applications (eBPF, OpenTelemetry)

Release notes

Problem to solve

Generating application traces requires code changes to add frameworks, such as OpenTelemetry, supporting different programming languages. The process requires learning the instrumentation libraries and interfaces, their impact on performance, potential bugs, etc. A lot of feedback in these discussions in the past years led to the fact that developers want to focus on actual development, and not add observability instrumentation to their agenda.

Automated instrumentation SDK code change creation could help, but developers would still need to maintain the OpenTelemetry integration, for example. This discussion required new approaches to automate trace generation and software code analysis at runtime.

Continuous profiling brought an interesting technology mindset change. Leveraging low-level Kernel technology with eBPF, it can make function calls to exported function names and symbols, and help with generating specific application traces from that. Polar Signals was one of the first Open Source vendors, highlighting the power of eBPF with Parca.

Intended users

User experience goal

Proposal

Auto-generate application traces from CI/CD deployed applications. The initial scope should be Linux VMs and Kubernetes. eBPF for Windows is in early stages.

Since this is a complex topic, my suggestion is to iterate on integrations and later built-in support.

  1. Evaluate market integrations for auto-instrumentation and trace generation.
    • Documentation and tutorials.
    • Integration into CI/CD templates, Helm charts, etc.
    • Examples: Odigos from keyval
  2. Implement helpers to auto-generate traces from deployed applications.
    • Idea: Auto DevOps build, that injects trace generation helpers into container images.
    • Idea: Standalone CLI that uses eBPF to generate application traces. (OSS)

Additionally, consider more Observability scope with:

  1. Evaluate which observability data can be generated (traces, metrics, logs, profiles, etc.) using eBPF.
  2. Evaluate continuous profiling as an additional feature request.
  3. Think about AIOps and efficiency #416956 (closed)

Should happen in separate issues, not required for issue completion.

Further details

Collaboration with SRE/infrastructure teams should happen to see how they can benefit from the proposed implementation, and help verify the needs.

Permissions and Security

Documentation

Availability & Testing

Available Tier

GitLab Ultimate because it makes development teams more efficient.

Depending on the implementation, a standalone tool could be in GitLab Free and only the integration into MR widgets and dashboards remains in GitLab Ultimate in the first iterations.

Feature Usage Metrics

What does success look like, and how can we measure that?

What is the type of buyer?

Is this a cross-stage feature?

Collaboration with sectioncd and infradev

What is the competitive advantage or differentiation for this feature?

Auto-instrumentation makes developers more efficient, and provides DevSecOps engineers deep insights into potential performance problems, and bugs in distributed environments (Kubernetes, etc.).

Links / references

FYI @sguyon @stkerr @nicholasklick

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

Edited by 🤖 GitLab Bot 🤖