### Problem to solve The GKG proto file (`gkg.proto`) lives in the knowledge-graph repo at `crates/gkg-server/proto/gkg.proto`. The generated Ruby proto files (`gkg_pb.rb`, `gkg_services_pb.rb`) are currently hand-generated and committed directly into the Rails repo at `ee/lib/ai/knowledge_graph/proto/`. This breaks in a few ways: someone has to manually regenerate and copy the files whenever the proto changes, the two repos can silently drift apart, and the proto files define their own module structure (`Gkg::V1`) that doesn't match Rails autoloading conventions, so we need a Zeitwerk exclusion hack. Gitaly already solved this. It publishes proto definitions as the [`gitaly` RubyGem](https://rubygems.org/gems/gitaly), and Rails pulls them in through a normal Gemfile dependency. We should do the same thing for GKG. ### Current state **GKG repo (source of truth for proto)**: - Proto definition: [`crates/gkg-server/proto/gkg.proto`](https://gitlab.com/gitlab-org/orbit/knowledge-graph/-/blob/main/crates/gkg-server/proto/gkg.proto) (package `gkg.v1`) - Generated Rust protos: `crates/gkg-server/src/proto/gkg.v1.rs` (generated and committed) - gRPC server implementation: `crates/gkg-server/src/grpc/` **Rails repo (consumer, on feature branch only)**: - Ruby proto files committed at `ee/lib/ai/knowledge_graph/proto/gkg_pb.rb` and `gkg_services_pb.rb` - Zeitwerk exclusion added to `config/initializers_before_autoloader/004_zeitwerk.rb` - Proto files loaded via `require_relative` from `grpc_client.rb` **Reference MR**: [gitlab-org/gitlab!221417](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/221417) ### How Gitaly does it The Gitaly proto gem pipeline works like this: 1. [`tools/protogem/build-proto-gem`](https://gitlab.com/gitlab-org/gitaly/-/blob/master/tools/protogem/build-proto-gem) is a Ruby script that runs `grpc_tools_ruby_protoc` against all `.proto` files and produces `*_pb.rb` (messages) and `*_services_pb.rb` (stubs). 2. The [`tools/protogem/Gemfile`](https://gitlab.com/gitlab-org/gitaly/-/blob/master/tools/protogem/Gemfile) declares `grpc-tools ~> 1.55.0` as the build dependency. 3. The [Gitaly Makefile](https://gitlab.com/gitlab-org/gitaly/-/blob/master/Makefile) has `build-proto-gem` (builds to `_build/gitaly.gem`) and `publish-proto-gem` (pushes to rubygems.org). 4. [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab-ci.yml) has two jobs: - `build-proto-gem` -- manual, produces `_build/gitaly.gem` as CI artifact - `publish-proto-gem` -- runs automatically on tagged releases, calls `gem push` 5. The gem entry point (`gitaly.rb`) auto-requires all `*_services_pb.rb` files. The gemspec depends only on `grpc ~> 1.0`. Version comes from the repo's `VERSION` file. 6. Rails declares `gem 'gitaly', '~> 18.8.0'` in [`Gemfile:637`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/Gemfile) and loads everything with `require 'gitaly'` in [`lib/gitlab/gitaly_client.rb:5`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/gitaly_client.rb). No Zeitwerk exclusion needed because gem paths are outside the autoloader. 7. The [`stub_class` method](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/gitaly_client.rb) uses `Gitaly.const_get(name.to_s.camelcase.to_sym, false).const_get(:Stub, false)` to resolve service stubs at runtime instead of hardcoding them. ### Proposed solution Build a `gkg-proto` gem from this repo's CI pipeline and have Rails consume it through the Gemfile, same as Gitaly. ## Technical design ### 1. Proto gem build script Create `tools/protogem/build-proto-gem` in this repo, modeled on the [Gitaly script](https://gitlab.com/gitlab-org/gitaly/-/blob/master/tools/protogem/build-proto-gem): - Input: `crates/gkg-server/proto/gkg.proto` - Tool: `grpc_tools_ruby_protoc` (from `grpc-tools` gem) - Output: `_build/gkg-proto.gem` containing: - `ruby/proto/gkg/v1/gkg_pb.rb` (message definitions) - `ruby/proto/gkg/v1/gkg_services_pb.rb` (service stubs) - `ruby/proto/gkg.rb` (entry point, auto-requires service files) - `ruby/proto/gkg/version.rb` (version constant) ``` tools/ protogem/ build-proto-gem # Ruby build script Gemfile # grpc-tools dependency Gemfile.lock ``` The gem entry point (`gkg.rb`): ```ruby # Auto-generated by build-proto-gem require_relative 'gkg/version' require_relative 'gkg/v1/gkg_services_pb' ``` ### 2. Gemspec Dynamically generated during build: ```ruby Gem::Specification.new do |spec| spec.name = 'gkg-proto' spec.version = Gkg::VERSION spec.authors = ['GitLab Engineering'] spec.summary = 'Auto-generated gRPC client for the GitLab Knowledge Graph service' spec.files = Dir['ruby/proto/**/*.rb'] spec.require_paths = ['ruby/proto'] spec.add_dependency 'grpc', '~> 1.0' end ``` ### 3. Makefile targets ```makefile .PHONY: build-proto-gem build-proto-gem: ruby tools/protogem/build-proto-gem $(BUILD_GEM_OPTIONS) .PHONY: publish-proto-gem publish-proto-gem: build-proto-gem gem push _build/gkg-proto.gem ``` ### 4. CI pipeline jobs Add two jobs to `.gitlab-ci.yml`: ```yaml build-proto-gem: stage: publish image: ruby:${RUBY_VERSION} script: - make build-proto-gem BUILD_GEM_OPTIONS="--skip-verify-tag" artifacts: paths: - _build/gkg-proto.gem expire_in: 1 month rules: - when: manual allow_failure: true publish-proto-gem: stage: publish image: ruby:${RUBY_VERSION} script: - make publish-proto-gem rules: - if: $CI_COMMIT_TAG when: on_success ``` ### 5. Version management The gem version should track proto-breaking changes: - Patch bump: additive field changes, new RPCs - Minor bump: new services - Major bump: breaking changes (field removal, type changes) ### 6. Rails consumption (downstream changes) In `gitlab-org/gitlab`: 1. Add to `Gemfile`: ```ruby gem 'gkg-proto', '~> 0.1.0', feature_category: :knowledge_graph ``` 2. Remove committed proto files from `ee/lib/ai/knowledge_graph/proto/` 3. Remove the Zeitwerk exclusion from `config/initializers_before_autoloader/004_zeitwerk.rb` 4. Update `grpc_client.rb` to use gem-based loading: ```ruby require 'gkg' # loads all proto definitions from the gem ``` 5. Use dynamic stub resolution (following Gitaly): ```ruby def stub Gkg::V1::GkgService::Stub.new(endpoint, credentials, **channel_args) end ``` ### 7. Proto linting Add [`buf`](https://buf.build/) linting to CI to catch breaking changes before they ship: ```yaml lint-proto: stage: test image: bufbuild/buf:latest script: - buf lint crates/gkg-server/proto/ - buf breaking crates/gkg-server/proto/ --against '.git#branch=main' rules: - changes: - crates/gkg-server/proto/**/* ``` This requires a `buf.yaml` config file at the proto root. ## Comparison of approaches | Approach | Sync mechanism | Zeitwerk hack | Version control | Drift risk | Precedent | |----------|---------------|---------------|-----------------|------------|-----------| | Gem (proposed) | Bundler version pin | Not needed | Semver via gem | Low | Gitaly | | CI artifact download | Pipeline trigger | Still needed | Pipeline ID | Medium | None at GitLab | | Committed in Rails (current) | Manual copy | Required | None | High | None at GitLab | | Dedicated proto repo | Git submodule or gem | Depends | Git tags | Medium | buf.build pattern | The gem approach is the only one with existing precedent at GitLab. ## Implementation steps 1. Create `tools/protogem/` directory with build script and Gemfile 2. Add Makefile targets (`build-proto-gem`, `publish-proto-gem`) 3. Add CI jobs for building and publishing 4. Add `buf.yaml` and proto linting CI job 5. Publish initial `gkg-proto` gem version 6. Update Rails Gemfile to use the gem (separate MR in `gitlab-org/gitlab`) 7. Remove committed proto files and Zeitwerk exclusion from Rails (same MR) 8. Document the proto update workflow for contributors ## Key files | File | Purpose | |------|---------| | [`crates/gkg-server/proto/gkg.proto`](https://gitlab.com/gitlab-org/orbit/knowledge-graph/-/blob/main/crates/gkg-server/proto/gkg.proto) | GKG proto source of truth | | [`tools/protogem/build-proto-gem` (Gitaly)](https://gitlab.com/gitlab-org/gitaly/-/blob/master/tools/protogem/build-proto-gem) | Reference build script | | [`tools/protogem/Gemfile` (Gitaly)](https://gitlab.com/gitlab-org/gitaly/-/blob/master/tools/protogem/Gemfile) | Reference build dependencies | | [Gitaly Makefile](https://gitlab.com/gitlab-org/gitaly/-/blob/master/Makefile) | Reference Makefile targets | | [Gitaly `.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitaly/-/blob/master/.gitlab-ci.yml) | Reference CI jobs | | [`lib/gitlab/gitaly_client.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/gitaly_client.rb) | Reference Rails consumption pattern | | [`gitlab-org/gitlab!221417`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/221417) | Current MR with committed proto files | ## Open questions 1. Gem name: `gkg-proto` vs `gkg` vs `gitlab-knowledge-graph-proto`? Gitaly's gem is just called `gitaly`, but GKG is a less distinctive name. 2. RubyGems.org vs internal gem server: Gitaly publishes to public rubygems.org. Should GKG do the same, or use the GitLab package registry? 3. Mono-gem vs separate: if additional proto files are added later (e.g., for Siphon integration), should they go in the same gem or a separate one?