feat: add organization_id to token metadata and request logs (#2185)

What does this MR do?

When GitLab Rails issues a registry token, it adds meta claims like project_id and root_namespace_id for each repository. The registry now also reads organization_id from those claims. It shows up in the request logs as meta_organization_id. The storage drivers get it too, the same way as the project and namespace IDs.

Rails doesn't send the claim yet. That's gitlab#598222, which I'm also working on. The two sides can be deployed in any order.

Decisions I made:

  1. The value comes only from the pull access in the token, same as the project and namespace IDs.
  2. With token auth, the log lines show meta_organization_id=0 until Rails sends the claim. Requests without token auth don't get the field at all.
  3. The storage drivers only get the ID when it's above 0. None of them use it yet, so GCS and CDN signed URLs don't change. Adding it to those URLs can be a follow-up.
  4. The field follows the meta.* naming of the other token IDs, so it logs as meta_organization_id. I didn't use labkit's gl_organization_id, because this repository is still on labkit v1.

This is not a breaking change, and it's safe to roll back. A token without the claim reads as 0, and nothing is stored.

I tested it by hand with a registry binary built from master and one built from this branch. Both ran with filesystem storage, no database, and token auth trusting a self-signed certificate. I signed a token whose pull access has "organization_id": 42 in meta, then sent one request:

$ curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:5077/v2/group/project/tags/list
{"errors":[{"code":"NAME_UNKNOWN","message":"repository name not known to registry","detail":{"name":"group/project"}}]}

The authorized request log line from master (trimmed):

level=info msg="authorized request" auth_project_paths="[]" auth_user_name=root auth_user_type=gitlab_or_ldap ... root_repo=group vars_name=group/project

The same line from this branch:

level=info msg="authorized request" auth_project_paths="[]" auth_user_name=root auth_user_type=gitlab_or_ldap ... meta_organization_id=42 root_repo=group vars_name=group/project

With a token that has no organization_id, the branch build logs meta_organization_id=0.

I added unit tests for reading the claim from a signed token and for a pull access without it. Another one checks that a zero ID stays out of the storage options. I ran the two touched packages that have tests:

$ go test -count=1 ./registry/auth/token/
ok  	github.com/docker/distribution/registry/auth/token	0.015s
$ go test -count=1 -short ./registry/storage/
ok  	github.com/docker/distribution/registry/storage	11.964s

gofumpt -l -extra on the changed files and go vet on both packages printed nothing. I didn't run golangci-lint, make test, the integration tests or make validate-log-fields locally. Some of them need Docker, Postgres or Redis, so I'm leaving them to the pipeline.

Implemented with AI assistance; I reviewed and tested the changes.

Closes #2185 (closed)

Rails side: gitlab#598222

Author checklist

  • CODEOWNERS Review: This MR requires approval from at least one CODEOWNER per category/file.
    • If codeowners are absent or the change is urgent, any registry maintainer can temporarily disable CODEOWNERS reviews in the project settings. When doing so:
      • Add a comment on the MR justifying the bypass
      • Mention/CC the designated codeowners in the MR for async review
      • Re-enable CODEOWNERS reviews in project settings once the MR is merged
    • If you lack permissions to disable CODEOWNERS reviews, reach out to the Engineering Manager (@jaime) or Senior Engineering Manager (@crystalpoole) for assistance.
  • Assign one of conventional-commit prefixes to the MR.
    • fix: Indicates a bug fix, triggers a patch release.
    • feat: Signals the introduction of a new feature, triggers a minor release.
    • perf: Focuses on performance improvements that don't introduce new features or fix bugs, triggers a patch release.
    • docs: Updates or changes to documentation. Does not trigger a release.
    • style: Changes that do not affect the code's functionality. Does not trigger a release.
    • refactor: Modifications to the code that do not fix bugs or add features but improve code structure or readability. Does not trigger a release.
    • test: Changes related to adding or modifying tests. Does not trigger a release.
    • chore: Routine tasks that don't affect the application, such as updating build processes, package manager configs, etc. Does not trigger a release.
    • build: Changes that affect the build system or external dependencies. May trigger a release.
    • ci: Modifications to continuous integration configuration files and scripts. Does not trigger a release.
    • revert: Reverts a previous commit. It could result in a patch, minor, or major release.
  • MR contains database changes including schema/background migrations:
    • Do not include code that depends on the schema migrations in the same commit. Split the MR into two or more.
    • Do not include code that depends on background migrations in the same release.
    • Manually run up and down migrations in a postgres.ai production database clone and add a link for the query plan(s) to the MR.
    • If adding new schema migrations make sure the REGISTRY_SELF_MANAGED_RELEASE_VERSION CI variable in migrate.yml is pointing to the latest GitLab self-managed released registry version. Find the correct registry version here. Make sure to select the branch of the latest GitLab release.
    • If adding new queries, extract a query plan from postgres.ai and post the link here. If changing existing queries, also extract a query plan for the current version for comparison.
      • I do not have access to postgres.ai and have made a comment on this MR asking for these to be run on my behalf.
    • If adding new background migration, follow the guide for performance testing new background migrations and add a report/summary to the MR with your analysis.
  • Change contains a breaking change - apply the breaking change label.
  • Change is considered high risk - apply the label high-risk-change
  • I created or linked to an existing issue for every added or updated TODO, BUG, FIXME or OPTIMIZE prefixed comment
  • Changes cannot be rolled back
    • Apply the label cannot-rollback.
    • Add a section to the MR description that includes the following details:
      • The reasoning behind why a release containing the presented MR can not be rolled back (e.g. schema migrations or changes to the FS structure)
      • Detailed steps to revert/disable a feature introduced by the same change where a migration cannot be rolled back. (note: ideally MRs containing schema migrations should not contain feature changes.)
      • Ensure this MR does not add code that depends on these changes that cannot be rolled back.
Documentation/resources

Code review guidelines

Go Style guidelines

Feature flags

When documentation is required

Documentation workflow

Reviewer checklist

  • Ensure the commit and MR tittle are still accurate.
  • If the change contains a breaking change, verify the breaking change label.
  • If the change is considered high risk, verify the label high-risk-change
  • Identify if the change can be rolled back safely. (note: all other reasons for not being able to rollback will be sufficiently captured by major version changes).
Edited by Hamza Hassanain

Merge request reports

Loading
Loading