Skip to content

Handle LFS objects for fork networks

Currently when the source of a fork(-network) is deleted, the forked projects lose track of the LFS objects.

After !17558 (merged) is merged, all projects in a fork network will have access to the LFS objects from any project in that LFS network. This is correct, since all forks can send merge requests to each other.

To avoid projects losing track of LFS objects uploaded to any other project in the fork network, we should also create a new LfsObjectsProject for each LFS object in the project being forked.

Question: Should we keep track of which LfsObjectProject was uploaded directly to the forked project and which one was "copied" during a fork? If so How?

Answer: We don't: #20042 (comment 257180920)

Create LfsObjectForkNetwork model

To avoid projects losing track of LFS objects uploaded to any other project in the fork network, we should also link the LFS objects to the fork network.

This would mean adding a new model:

class LfsObjectForkNetwork
  belongs_to :fork_network
  belongs_to :lfs_object
end

Then we could check that for accessibility if the LFSObject was not linked to the project directly.

When a project is forked, we would automatically need to create all the LfsObjectForkNetwork records in the Projects::ForkService.

If a project is part of a fork network, we need to create a LfsObjectForkNetwork when a new LFS object is created in the Projects::LfsStorageController and the Lfs::FileTransformer. In this case, we would no longer be creating an LfsObjectProject for the root of the fork network, but for the project it was created for. So no more Project#lfs_storage_project as all LFS objects should be stored on self.

We would need to backfill the LfsObjectForkNetwork records for all projects that are currently part of a fork network.

Original issue

We should figure out a way to handle this.

Possible solutions:

  • Add the LFS objects to the forked project immeadiatly after the fork. The objects will then be handled independently of the fork network and count towards the quota of the namespace the fork resides in.
  • Add the LFS objects to all the forked projects when the root project is deleted. This could be a lot of forks, but it's handled in the background.
  • Keep track of the LFS-objects on a fork network. But to which quota do they count?

Later created fork relations

Another issue was brought up in gitlab-ce!18104:

When a fork relation is created later on, the LFS objects of the fork might be somewhere else than the LFS objects of the parent. So at that point the objects of the fork become inaccessible, and the user should push the LFS objects again after creating the fork relation.

The deletion now happens in the Projects::ForkService#link_existing_project, that might have to be reconsidered when implementing this: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22226#note_108780553

Newly uploaded LFS objects

Right now, we're doing 2 different things:

  1. When a user pushes LFS objects through their git-lfs client, the uploads go through workhorse and in Projects::LfsStorageController#upload_finalize! and #link_project! which uses the LfsRequest#storage_project.
  2. When a user uploads through the web UI, this would be dealt with in this transformer, the LFS object would be linked to the current project and would become inaccessible.

This is being changed in !17558 (merged) to all upload to the root of the fork network, but make LFS objects linked to the project itself accessible as well.

We should decide on the right project to link to LFS objects uploaded for a fork.

Related issues

This relates to gitlab-foss#38883 (closed), and was brought up in gitlab-foss!15150 (comment 45541479)

/cc @DouweM @jramsay

Edited by Bob Van Landuyt