Skip to content

Backend work to return non-registered agent configuration directories

What is it about

We should expose an endpoint that will search through non-registered agents present in a project's default branch.

After fetching the list of existing agent-config-like directories from the repository, these need to be checked on the database against existing agents. The ones which don't have an associated agent are then sent back to the client in the response.

Tech details (investigating)

I think default branch is what we define as root_ref.

We also have the head_tree, which seems to return the commit from the root_ref. E.g:

[10] pry(main)> project.repository.head_tree
=> #<Tree:0x00007fa8c9ea9b28
 @entries=
  [#<Gitlab::Git::Tree:0x00007fa8c9e9b6e0
    @commit_id="9c65f8545a5dc7133e135819409c79f1186dc21c",
    @flat_path=".gitlab/agents/stress-test-1",
    @id="d910ec54edc1428f83b20d35e265123c5ad21f12",
    @mode="40000",
    @name=".gitlab",
    @path=".gitlab",
    @root_id="cd6b4d32afe1570b04b92219d032b44345207734",
    @type=:tree>,
   #<Gitlab::Git::Tree:0x00007fa8c9e9b398
    @commit_id="9c65f8545a5dc7133e135819409c79f1186dc21c",
    @flat_path="manifest.yaml",
    @id="0b83fa391681acdb0d80e245d6ef85c592a7ea7e",
    @mode="100644",
    @name="manifest.yaml",
    @path="manifest.yaml",
    @root_id="cd6b4d32afe1570b04b92219d032b44345207734",
    @type=:blob>],
 @path="/",
 @repository=#<Repository:@hashed/bb/66/bb668ca95563216088b98a62557fa1e26802563f3919ac78ae30533bb9ed422c>,
 @sha="9c65f8545a5dc7133e135819409c79f1186dc21c">

Repository#ls_files might also be helpful:

[6] pry(main)> r.ls_files('master')
  ApplicationSetting Load (1.7ms)  SELECT "application_settings".* FROM "application_settings" ORDER BY "application_settings"."id" DESC LIMIT 1 /*application:console,line:/app/models/concerns/cacheable_attributes.rb:19:in `current_without_cache'*/
=> [".gitlab/agents/stress-test-1/config.yaml", "manifest.yaml"]

Or even Repository#search_files_by_wildecard_path

[8] pry(main)> r.search_files_by_wildcard_path('.gitlab/agents/*')
=> [".gitlab/agents/stress-test-1/config.yaml"]

Points of attention

Do we need ReactiveCaching?

We need to check how files are currently read from the repository. If this needs to happen async, it's possible we might need to implement a solution with reactive caching for the FE to poll. It might be that we already cache the the default branch files. Hopefully that's the case and we're able query the config files from there. If I understand correctly Gitaly RPC calls are sync. The Repository model does not seem to include ReactiveCaching nor the methods seem to be wrapped in it. I've also drilled down the calls to the below methods without finding ReactiveCaching usage around them:

search_files_by_wildcard_path

Repository#search_files_by_wildcard_path -> Repository#raw_repository.search_files_by_regexp -> gitaly_repository_client.search_files_by_regexp -> GitalyClient.call(@storage, :repository_service, :search_files_by_name, request, timeout: GitalyClient.fast_timeout).flat_map(&:files)

The above is apparently used sync by API::Lint#post('/lint') as part of the chain: Gitlab::Ci::YamlProcessor.new(params[:content], user: current_user).execute -> Gitlab::Ci::Config#initialize -> Config::External::Processor.new -> External::Mapper.new(values, context).process -> .flat_map(&method(:expand_wildcard_paths)) -> context.project.repository.search_files_by_wildcard_path(location[:local], context.sha)

blob_at

Repository#blob_at -> Repository#raw_repository.blob_at -> Gitlab::Git::Blob.find(self, sha, path) -> Gitlab::Git::Blob.tree_entry(repository, sha, path, limit) -> Gitlab::GitalyClient::CommitService.new(repository).tree_entry(sha, path, req_limit) -> GitalyClient.call(@repository.storage, :commit_service, :tree_entry, request, timeout: GitalyClient.medium_timeout)

It's used by the Project::BlameController#show call made synchronously.

Dirs or Config files?

Do we need to validate directories or config files (trees or blobs)? Maybe directories is sufficient, since we already have the expected agent name.

Feature Flag

It might be a good idea to put this behind a FF, since the FE won't exist for it right away and there'll be several parts of FE to be implemented.

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

Edited by 🤖 GitLab Bot 🤖