Refactor pull and push mirroring, and imports.
Currently, pull mirroring (an EE-only feature, also simply called 'mirroring') re-uses the onetime project import infrastructure (projects.import_url
and the project_import_data
table).
Push mirroring (another EE-only feature, also called 'remote mirroring') also uses projects.import_url
, but has separate tables (remote_mirrors
and project_mirror_data
).
Relevant parts of the schema:
create_table "project_import_data", force: :cascade do |t|
t.integer "project_id"
t.text "data"
t.text "encrypted_credentials"
t.string "encrypted_credentials_iv"
t.string "encrypted_credentials_salt"
end
add_index "project_import_data", ["project_id"], name: "index_project_import_data_on_project_id", using: :btree
create_table "project_mirror_data", force: :cascade do |t|
t.integer "project_id", null: false
t.string "status"
t.string "jid"
t.text "last_error"
end
add_index "project_mirror_data", ["jid"], name: "index_project_mirror_data_on_jid", using: :btree
add_index "project_mirror_data", ["project_id"], name: "index_project_mirror_data_on_project_id", unique: true, using: :btree
add_index "project_mirror_data", ["status"], name: "index_project_mirror_data_on_status", using: :btree
create_table "remote_mirrors", force: :cascade do |t|
t.integer "project_id"
t.string "url"
t.boolean "enabled", default: false
t.string "update_status"
t.datetime "last_update_at"
t.datetime "last_successful_update_at"
t.datetime "last_update_started_at"
t.string "last_error"
t.boolean "only_protected_branches", default: false, null: false
t.string "remote_name"
t.text "encrypted_credentials"
t.string "encrypted_credentials_iv"
t.string "encrypted_credentials_salt"
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
end
add_index "remote_mirrors", ["last_successful_update_at"], name: "index_remote_mirrors_on_last_successful_update_at", using: :btree
add_index "remote_mirrors", ["project_id"], name: "index_remote_mirrors_on_project_id", using: :btree
I'm currently implementing SSH public-key auth for pull repository mirroring in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2423 . Because pull and push mirrors are so separate, push mirrors won't benefit from this work. Implementing it naively will result in code duplication.
I think we can get some gains here by moving pull, push and onetime imports into a single table:
create_table :mirrors do |t|
t.string "url" # remove `import_url` from the `projects` table
t.string "type" # import, pull, push
t.text "encrypted_credentials"
# ...
end
I'm not clear on why This is actually for pull mirroring!project_mirror_data
is a separate table to remote_mirrors
. It may be possible to merge that table in too.
This will look a touch overengineered in CE, but EE will have two additional STI model classes (note that import
cannot inherit from pull
, due to the license, but perhaps pull could inherit from import). All three methods can then benefit from identical handling of credentials, UI, etc.
I'm not going to delay implementing SSH pull mirroring to do this refactor, but by the time we want to extend it to push mirroring, I think it should be done on these terms.
/cc @DouweM