Add Key Abstractions handbook page

^[Eng-Architecture]

/content/handbook/engineering/architecture/ @ayufan @grzesiek @glopezfernandez @andrewn @nicholasklick @DylanGriffith @theoretick @shekharpatnaik @stanhu @fabiopitino @tkuah @ntepluhina @jessieay @vitallium

[Eng-Architecture-Abstractions]

/content/handbook/engineering/architecture/abstractions/ @stanhu @andrewn @ayufan @shekharpatnaik @grzesiek @gitlab-com/content-sites @gitlab-com/egroup

^[Eng-Development-department]

/content/handbook/engineering/devops/ @m_gill

/content/handbook/engineering/ai/ @timzallmann

At GitLab, **Architecture** is a collaborative process. It is also:

- A [catalog of **key abstractions**](abstractions/) that defines the approved

  foundational technologies and patterns engineers must use, ensuring consistency

  and reducing complexity across the organization.

- A [collection of practices](practice/) that provide technical frameworks to

  **guide** (rather than dictate) our thinking, design, and discussions so we

  can *iterate* quickly and deliver *results*. These include the

---

title: "Key Abstractions"

---

## Overview

Key Abstractions (also known as Key Primitives) are foundational technologies,

patterns, and components that have been approved for use across GitLab's

engineering organization. Once a key abstraction is approved, engineers must use

it when building functionality that requires that capability.

## Governance

- **Approval Process**: Adding a new key abstraction requires VP approval

- **Ownership**: Each approved key abstraction must have a designated owner

  within the company

- **Compliance**: If a technology or abstraction is in the approved catalog,

  teams must use it rather than introducing alternatives

- **Architecture Board**: The Architecture Board reviews and approves items for

  the catalog of key abstractions

If a needed capability is not in the catalog, teams should discuss it with the

Architecture Board.

## Architecture Board

The Architecture Board is a lightweight, informally structured group that

reviews and stewards the catalog of key abstractions. This group ensures that

architectural decisions are made collaboratively while maintaining technical

consistency across GitLab's engineering organization.

The board operates using a lightweight, informal approach that prioritizes

collaboration and rapid iteration while ensuring proper oversight of

foundational technology decisions. This represents the starting point for the

Architecture Board. As the organization's needs evolve, the board's structure,

scope, and processes may be expanded and formalized.

### Composition

- VPs appoint Architecture Board members

- We initially started with appointing all Distinguished Engineer+ as members.

- Membership is managed through CODEOWNERS for the `architecture/abstractions`

  directory

- Only Architecture Board members can approve and merge changes to the key

  abstractions catalog

### Responsibilities

The Architecture Board reviews and approves:

- Proposals for new key abstractions

- Promotion of candidate abstractions to graduated (approved) status

- Updates to existing key abstraction documentation

- Deprecation or removal of abstractions from the catalog

## Maturity States

Key abstractions can be in one of the following states:

- **Candidate**: Under consideration, requires Architecture Board review

- **Graduated**: Approved for use, requires an assigned owner

All graduated key abstractions must have a designated owner.

## Approved Key Abstractions

| Abstraction | Category | Description | Owner |

|-------------|----------|-------------|-------|

| [**PostgreSQL**](postgresql/) | Data & Storage | OLTP Database | TBD |

## Candidate Key Abstractions

| Abstraction | Category | Description | Notes |

|-------------|----------|-------------|-------|

| **Siphon** | Data & Storage | Data ingestion and transformation | |

| **ClickHouse** | Data & Storage | Analytics / OLAP queries | |

| **Redis** | Data & Storage | Cache and global locking | |

| **Object Storage** | Data & Storage | Binary and large file storage | |

| **Elasticsearch/OpenSearch** | Data & Storage | Natural language search | |

| **GitLab Zoekt** | Data & Storage | Code search | |

| [**NATS**](candidate/nats/) | Messaging & Processing | Message bus for high-throughput event streaming | Use for very high throughput and durable messages. |

| [**Sidekiq**](candidate/sidekiq/) | Messaging & Processing | Background processing | Default choice for background jobs. |

| **Active Context** | Search & AI | Semantic search (embeddings) | |

| **Ruby, Go, Python, Rust** | Languages & Frameworks | Backend languages with LabKit support | |

| **VueJS, JavaScript, TypeScript** | Languages & Frameworks | Frontend technologies | |

| **gRPC, REST** | Languages & Frameworks | Communication protocols | |

| **RAFT** | Distributed Systems | Consensus algorithm | |

| **GitLab Zoekt** | Distributed Systems | Pattern for stateful services | |

| **Secrets Storage** | Security | Secure secrets management | Includes proper database encryption (avoid initialization vector reuse) |

| **JSON Schema** | Security | Data validation | |

| **OpenMetrics** | Observability | Metrics collection | Prometheus compatible |

| **OpenTelemetry** | Observability | Distributed tracing | Configured via LabKit |

| **Rate Limiter** | Architecture Patterns | Request rate limiting | Implemented through LabKit where possible |

| **Circuit Breaker** | Architecture Patterns | Fault tolerance pattern | Implemented through LabKit where possible |

| **UBI-9** | Infrastructure | Supported Cloud Native GitLab images | |

## Services we don't want to use

The following technologies should **not** be used:

- **Consul**

- **Kafka** - use [NATS](candidate/nats/) or [Sidekiq](candidate/sidekiq/) instead

## Contributing

To propose a new key abstraction or promote a candidate to graduated status:

1. Prepare documentation including:

   - Clear description and use cases

   - Why it should be a key abstraction

   - Proposed owner

   - Integration considerations

2. Present to the Architecture Board for review

3. Obtain approval from VP of Data Engineering and VP of Infrastructure

## Related Resources

- [Architecture Workflow](/handbook/engineering/architecture/workflow/)

- [Scalability Practice](/handbook/engineering/architecture/practice/scalability/)

---

title: "NATS - Message Bus"

---

## Status

**Candidate** - Under consideration for approval

## Owner

[To be assigned]

## Description

NATS is a high-performance message bus for high-throughput event streaming and real-time communication between distributed systems.

## Why This Should Be a Key Abstraction

NATS is being considered as the standard message bus for GitLab because it provides:

1. **High Throughput**: Handles millions of messages per second

2. **Low Latency**: Optimized for real-time event streaming

3. **Cloud Native**: Well-suited for distributed architectures

## Use Cases

Use NATS for:

- Durable message bus

- Very high throughput event streaming

- Real-time data processing pipelines

- Event-driven architectures requiring low latency

## Do Not Use For

**Do not use NATS for:**

- Core GitLab functionality that must work in all deployment environments

- Standard background processing that Sidekiq can handle

## Integration Considerations

### Self-Managed Limitations

NATS is not suitable for self-managed foundational (core) features because:

- Not all self-managed instances may have NATS deployed (yet)

- Adds operational complexity for customers

## Related Key Abstractions

- [**Sidekiq**](sidekiq/) - Alternative for standard background processing

- **PostgreSQL** - May be used alongside NATS for persistent data storage

- **Redis** - Used by Sidekiq; may complement NATS for caching

---

title: "Sidekiq - Background Processing"

---

## Status

**Candidate** - Under consideration for approval

## Owner

[To be assigned]

## Description

Sidekiq is a background job processing framework for Ruby applications that uses Redis for job storage and queuing.

## Why This Should Be a Key Abstraction

Sidekiq is being considered as the standard background processing system for GitLab because:

1. **Proven at Scale**: Successfully handles millions of jobs across GitLab.com

2. **Ruby Native**: Seamlessly integrates with GitLab's Rails application

3. **Self-Managed Compatible**: Works reliably in all GitLab deployment environments

## Use Cases

Use Sidekiq for:

- Standard background job processing

- Asynchronous task execution

- Scheduled jobs and cron-like tasks

- Any feature that needs to work in self-managed instances (SMF/SMA)

- Core GitLab functionality requiring background processing

## Do Not Use For

Consider alternatives for:

- **Very high throughput scenarios**: Use NATS (candidate key abstraction) for extreme throughput requirements

- **Real-time event streaming**: NATS may be more appropriate for under 1ms latency requirements

## Integration Guidance

### For New Features

When implementing background processing for new features:

1. **Default to Sidekiq**: Unless you have specific performance requirements, use Sidekiq

2. **Measure First**: If you think you need NATS, gather performance data first

3. **Consider Self-Managed**: If the feature must work for self-managed customers, use Sidekiq

4. **Follow Patterns**: Use established Sidekiq patterns and worker conventions in the GitLab codebase

## Related Key Abstractions

- [**NATS**](nats/) - Alternative for very high throughput scenarios

- **Redis** - Required dependency for Sidekiq

- **PostgreSQL** - Often used alongside Sidekiq for persistent data storage

## Support and Resources

- GitLab [Sidekiq development documentation](https://docs.gitlab.com/development/sidekiq/)

- Sidekiq [development configuration](https://docs.gitlab.com/development/sidekiq/worker_attributes/) and [operator configuration](https://docs.gitlab.com/administration/sidekiq/)

- Internal [monitoring dashboards](https://dashboards.gitlab.net/dashboards/f/sidekiq/sidekiq-service) and [runbooks](https://gitlab.com/gitlab-com/runbooks/-/tree/master/docs/sidekiq)

- Infrastructure team office hours and Slack channels