Re-architect job artifacts removal

What does this MR do?

Related to https://gitlab.com/gitlab-org/gitlab/-/issues/233939 and https://gitlab.com/gitlab-org/gitlab/-/issues/223034

This merge requests was extremely large and it was split into:

• !42095 (merged) - adds deduplicate :none strategy to allow parallel execution for jobs that have the same/no arguments
• !42237 (merged) - adds a framework that enqueues and executes a limited number of jobs in parallel
• !39464 (merged) - this merge request
• !42242 (merged) - will update DestroyExpiredJobArtifactsService to use the previous merge requests

Goal:

decouple database artifacts deletion from the object storage deletion. We need to delete records as fast as possible from the ci_job_artifacts table to update the storage quota for the users, but we have hit a bottleneck with the object storage. So we are copying the file attributes needed for object storage removal to a different table to allow slow paced deletes.

• Creates the new table which will store file details for the removed job artifacts
• Adds methods to import data from job artifacts into this new table
• Adds services and workers to consume the data from the new table.

Migration output

$ VERSION=20200813135558 bin/rake db:migrate:redo

== 20200813135558 CreateCiDeletedObjects: reverting ===========================
-- drop_table(:ci_deleted_objects)
   -> 1.5683s
== 20200813135558 CreateCiDeletedObjects: reverted (1.5683s) ==================

== 20200813135558 CreateCiDeletedObjects: migrating ===========================
-- create_table(:ci_deleted_objects, {:if_not_exists=>true})
   -> 0.0117s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0004s
-- execute("ALTER TABLE ci_deleted_objects\nADD CONSTRAINT check_5e151d6912\nCHECK ( char_length(store_dir) <= 1024 )\nNOT VALID;\n")
   -> 0.0008s
-- current_schema()
   -> 0.0003s
-- execute("SET statement_timeout TO 0")
   -> 0.0003s
-- execute("ALTER TABLE ci_deleted_objects VALIDATE CONSTRAINT check_5e151d6912;")
   -> 0.0007s
-- execute("RESET ALL")
   -> 0.0002s
== 20200813135558 CreateCiDeletedObjects: migrated (0.0276s) ==================

Screenshots

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• [-] Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• [-] Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team