Add Container Registry migration notification APIs
🏯 Context
We're currently migrating container images in the Container Registry. To keep things simple (this not accurate): the Container Registry team is adding support for a database and that database will host informations about the container images. Newly created container images will already be inserted in the database but existing ones need to be migrated.
In &7316 (closed), it has been described that rails will "drive" this migration. Put simply, using an API on the Container Registry, rails will start the migration for container images and will receive notifications about it.
The migration is basically two steps: pre_import
and import
. After each step, the Container Registry will notify the rails backend. Example: Hey for container image "my/awesome/image", step "pre_import" has completed.
.
This MR is implementing this API endpoint that will receive the migration notifications from the Containe Registry. That's issue #349742 (closed).
The notification payload is described here. We have basically, 4 possible statuses:
import_complete
pre_import_complete
import_failed
pre_import_failed
The endpoint behavior will be quite simple for now:
- Check that the target repository
migration_state
for coherence. Return an error if not coherent.- Rails manages a
migration_state
attribute on container images using a state machine, see !78499 (merged).
- Rails manages a
- For
pre_import_complete
, try to execute the proper transtition.- Return
bad_request
if the transition fails
- Return
- For
import_complete
, execute the proper transition if we're in the rightmigration_state
- Return
bad_request
if the transition fails
- Return
- For non successful statuses, abort the migration.
Note that (1.) - (3.) can leave the repository in incoherent migration_state
. This is done on purpose. We will have a background job whose responsibility is to check all the migration_state
s and fix incoherent ones. As such, it is not the responsibility of this endpoint to fix those statuses incoherences.
🔬 What does this MR do and why?
- Add a
PUT /api/:version/internal/registry/repositories/:path/migration/status
API endpoint- This endpoint is only accessible using the container registry notification secret.
- As such, this is not part of the public REST api = we will not documentation for it because this is an "internal" API
- Add additional helper functions in the container image model (
ContainerRepository
) - Add/Update the related spec
🖼 Screenshots or screen recordings
n / a
🔧 How to set up and validate locally
In a rails console:
- Take note of the container registry notification (you might need to set one in
gitlab.yml
:Gitlab.config.registry.notification_secret
import_complete
status
Receiving - Create a container repository in the importing
migration_state
and take note of the :FactoryBot.create(:container_repository, :importing, project: Project.first).path
- Send the migration notification:
$ curl -X PUT -H 'Content-Type: application/json' -H 'Authorization: <secret>' -d '{"status": "import_complete"}' http://gdk.test:8000/api/v4/internal/registry/repositories/<path>/migration/status 200
- Check the
migration_state
ContainerRepository.last.migration_state # "import_done"
pre_import_complete
status
Receiving :ok
.
- Create a container repository in the pre_importing
migration_state
and take note of the :FactoryBot.create(:container_repository, :pre_importing, project: Project.first).path
- Send the migration notification:
$ curl -X PUT -H 'Content-Type: application/json' -H 'Authorization: <secret>' -d '{"status": "pre_import_complete"}' http://gdk.test:8000/api/v4/registry/repositories/<path>/migration/status 200
- Check the
migration_state
ContainerRepository.last.migration_state # "importing"
anything
else
Receiving - Create a container repository in the pre_importing
migration_state
and take note of the :FactoryBot.create(:container_repository, :pre_importing, project: Project.first).path
- Send the migration notification:
$ curl -X PUT -H 'Content-Type: application/json' -H 'Authorization: <secret>' -d '{"status": "kaboom"}' http://gdk.test:8000/api/v4/registry/repositories/<path>/migration/status 200
- Check the
migration_state
ContainerRepository.last.migration_state # "import_aborted"
Additional cases
Play around with a wrong or an invalid . You will get 401 Unauthorized
and 404 Not found
respectively.
🚦 MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.