## Why is this an epic? The restoration of the `ExpireBuildArtifactsWorker` and the re-implementation of the artifacts backlog expiration code is being implemented across three merge requests in two separate workers controlled by a total of five feature flags. This epic is to track progress in the deployment and overall target of removing expired artifacts from the `ci_job_artifacts` table. If anyone else needs to pick up some monitoring, iteration on any of these new designs, or work the feature flags a little bit, this epic should serve as an authoritative source on what exactly on earth is going on at any given time. ## Part 1: Stop the artifacts backlog growth :white_check_mark: **DONE** <details> ### Part 1 Merge Requests **DONE** :white_check_mark: 1. **DONE** Merge and deploy https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76504 - Restore optimized newly expired artifacts removal 2. **DONE** Merge and deploy https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/1393 Remove overriding cron worker schedule for ExpireBuildArtifactsWorker At this point, all of our feature flags are turned off. We won't be removing expired artifacts, but we'll see the `ExpireBuildArtifactsWorker` executing in [Kibana](https://log.gprd.gitlab.net/app/discover#/view/ba79eca0-3ce6-11ec-8c8e-ed83b5469915?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-15m%2Cto%3Anow))) successfully and reporting `destroyed_job_artifacts_count: 0`. At this point, we start hittin' the switches. ### Part 1 Feature Flags (3) **DONE** :white_check_mark: 1. **DONE** [Turn on `ci_destroy_all_expired_service`](https://gitlab.com/gitlab-org/gitlab/-/issues/348786), and watch the old three-table-join logic expire some artifacts at a slow pace, in the low thousands per execution. Verify database and dot-com stability. 2. **DONE** Turn on `ci_destroy_unlocked_job_artifacts` to use our modernized artifact expiration logic that ran in a performant and stable fashion during September and October. At first, we will see **1,000 artifacts removed per execution**, and execution time of the service should be quite fast. My gut says under 1 minute for the whole worker. Verify database and dot-com stability. 3. **DONE** Turn on `ci_artifact_fast_removal_large_loop_limit` to use the same modern, efficient expired artifact removal but at a pace of **10,000 artifacts removed per execution**. 4. **DONE** Given the very nice behavior and lack of DB performance change we're observing at 10k artifacts per execution, I've opened https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77603 to increase the 1k and 10k limits to 10k and 50k, respectively. This is necessary because our earlier estimation of artifacts creation seems as if it was low. A Sisense dashboard is still in the works (WILL UPDATE WHEN AVAILABLE), but at 10k artifacts per execution, we're still seeing the table and ObjectStorage grow at a rate of roughly 0.3% per weekday. We do a meaningful amount of catchup over the weekends, but given our expectation of reliably good performance, increasing the pace is definitely a prudent choice. Daily manual pulls from Sisense:  **Where are we now?** * At this point, we should be steadily, if not especially quickly, decreasing the number of rows in `ci_job_artifacts` with `expire_at < Time.current AND locked = 0` * The number of artifacts with `locked = 2` (unknown locked status) will remain constant. * We should see stable metrics from all the pages linked in the monitoring section. We should let this state continue to run until we see [Kibana](https://log.gprd.gitlab.net/app/discover#/view/ba79eca0-3ce6-11ec-8c8e-ed83b5469915?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-15m%2Cto%3Anow))) reporting that `ExpireBuildArtifactsWorker` is executing successfully and reporting `destroyed_job_artifacts_count` with **a number less than the execution limit, either 10,000 or 50,000**. This means that we've finished removing the backlog of expired artifacts that has accumulated since the [incident on November 22](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5952). At this point, we ~~will~~ have reached the backlog-stability that we originally had when we closed https://gitlab.com/gitlab-org/gitlab/-/issues/327281. **UPDATE January 18**: We have reached this state! Newly expired artifacts are now removed promptly. Given our 7-minute execution cycle and our (roughly) 1-minute execution time, I would confidently say that a artifact created and expired on gitlab.com today will be removed in less than ten minutes of it's expiration date. If a large number of artifacts are expired at once it could potentially create a small new backlog, but we'll catch up quickly. Here's the processing throughput dropping off on January 16th, when we finished the _newer_ backlog:  From here, we can decide how to deal with the old (pre 22 Nov 2021) backlog of unexpired artifacts, which is no longer growing. See **Part 2**. The other good news here is that The JobArtifact bulk-deletion API https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75488 can operate in a performant way. When new artifacts are "deleted" via that API call, they'll be enqueued for removal and deletion will begin immediately, and continue on the same 50k/300s cycle we're currently running. </details> ## Part 2: Remediation for accidentally expired artifacts on SM instances https://gitlab.com/groups/gitlab-org/-/epics/7097 :white_check_mark: DONE <details> Before we move forward with expiring the old (years old, prior to November 2021) backlog of expired, unremoved artifacts, we need to address this epic that discusses large swaths of artifacts that were accidentally given expiration dates that shouldn't have been. Due to the manually written expiration date, artifacts that are the last on their ref (e.g. tags) may have their artifacts cleared, and we can break workflows that depend on these artifacts. </details> ## Part 3: Process pre-existing backlog of expired artifacts waiting for removal ### Part 3 Merge Requests DONE :white_check_mark: <details> 1. **Introduce Ci::UpdateLockedUnknownArtifactsWorker** https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76509 :white_check_mark: This merge request introduces a completely separate worker that iterates through our backlog of `ci_job_artifacts` rows where `expire_at < Time.current AND locked = 0`. For each of those rows, the worker will execute a service to either update `locked = 1` (artifacts_locked), or remove the row if we determine that the artifacts are unlocked, expired, and therefore ready to be removed. Upon initial deployment, this worker will execute but do no work. </details> ### Part 3 Feature Flags DONE :white_check_mark: <details> 1. Turn on `ci_job_artifacts_backlog_work`, which allows the `Ci::UpdateLockedUnknownArtifactsWorker` to execute the `Ci::JobArtifacts::UpdateUnknownLockedStatusService`, which at first will either delete or write `locked = 1` to **10,000 artifacts per execution**, which is scheduled to **run every 15 minutes**. 2. Turn on `ci_job_artifacts_backlog_large_loop_limit`, to increase the sum of `removed_count` and `locked_count` reported by `Ci::UpdateLockedUnknownArtifactsWorker` to **50,000**. At this point, we should be observing all of the same metrics from the fourth feature flag, `ci_job_artifacts_backlog_work`, and confirm that we see no difference in application or database performance. We should then decide if we're comfortable leaving this worker at 50,000 records per execution, or if we want to turn it back down to 10,000. **At either 10,000 or 50,000 per execution of `Ci::UpdateLockedUnknownArtifactsWorker`, we are actively reducing the expired artifacts backlog and moving in the right direction. The only difference is how fast.** At this point, we should again check the monitoring section to make sure that we're not sacrificing any database peformance for the sake of this cleanup. Which can be moving a theoretical maximum of **600,000 artifacts per hour** When we have fully [Cleaned up old expired job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/322817) https://gitlab.com/gitlab-org/gitlab/-/issues/322817, both counts reported by `Ci::UpdateLockedUnknownArtifactsWorker` will drop to zero. This worker does not operate on `ci_job_artifacts` unless they are `locked = 2 AND expire_at < #{Time.current}`. The work of removing expired, unlocked artifacts is entirely handled by the `ExpireBuildArtifactsWorker`. When `removed_count` and `locked_count` both reach 0, we can schedule a complete removal of this worker, and discuss the best way to do so to account for self managed instances. </details> ### Monitoring DONE :white_check_mark: <details> * In [Kibana](https://log.gprd.gitlab.net/app/discover#/view/ba79eca0-3ce6-11ec-8c8e-ed83b5469915?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-15m%2Cto%3Anow))) we should see the `ExpireBuildArtifactsWorker` executing successfully and reporting `destroyed_job_artifacts_count` as either `10,000` or `50,000`. * We also have a Kibana [dashboard](https://log.gprd.gitlab.net/app/dashboards#/view/c3f71fd0-a606-11ec-bcd1-aba7259b6bf1?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-1w%2Cto%3Anow))) tracking artifacts removed from the table by the more performant ExpireBuildArtifactsWorker that handles the removal of expired artifacts on an ongoing basis. * In [Thanos](https://thanos-query.ops.gitlab.net/graph?g0.expr=sum(rate(pg_stat_user_indexes_idx_tup_read%7Benv%3D%22gprd%22%2C%20relname%3D%22ci_job_artifacts%22%2C%20fqdn%3D%22patroni-v12-03-db-gprd.c.gitlab-production.internal%22%7D%5B5m%5D))%20by%20(indexrelname)%20%3E%200&g0.tab=0&g0.stacked=0&g0.range_input=3h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.end_input=2021-11-22%2015%3A00%3A00&g0.moment_input=2021-11-22%2015%3A00%3A00&g1.expr=sum(rate(pg_stat_user_indexes_idx_tup_read%7Benv%3D%22gprd%22%2C%20relname%3D%22ci_job_artifacts%22%7D%5B5m%5D))%20by%20(indexrelname)&g1.tab=0&g1.stacked=0&g1.range_input=12h&g1.max_source_resolution=0s&g1.deduplicate=1&g1.partial_response=0&g1.store_matches=%5B%5D) we should see the dead tuples on `ci_job_artifacts` not growing especially quickly. A developer should confer with an SRE on this and have them gauge their comfort with the number of dead tuples, in the context of query performance across all `ci_job_artifacts` indexes and overall apdex. * Also in [Thanos](https://thanos.gitlab.net/graph?g0.expr=sum%20by%20(relname%2C%20indexrelname%2C%20fqdn)%20(rate(pg_stat_user_indexes_idx_tup_read%7Benv%3D%22gprd%22%2C%20relname%3D%22ci_job_artifacts%22%7D%5B1m%5D)%20and%20on%20(env%2C%20fqdn)%20(pg_replication_is_replica%20%3D%3D%201))%0A%2F%0A(%20sum%20by%20(relname%2C%20indexrelname%2C%20fqdn)%20(rate(pg_stat_user_indexes_idx_scan%7Benv%3D%22gprd%22%2C%20relname%3D%22ci_job_artifacts%22%7D%5B1m%5D)%20and%20on%20(env%2C%20fqdn)%20(pg_replication_is_replica%20%3D%3D%201))%20%3E%201)&g0.tab=0&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.end_input=2022-01-05%2019%3A30%3A00&g0.moment_input=2022-01-05%2019%3A30%3A00) we should see the rate of index tuple reads remain flat, hovering around 2.0. During the incident, this figure very quickly escalated [into the thousands](https://thanos.gitlab.net/graph?g0.expr=sum%20by%20(relname%2C%20indexrelname%2C%20fqdn)%20(rate(pg_stat_user_indexes_idx_tup_read%7Benv%3D%22gprd%22%2C%20relname%3D%22ci_job_artifacts%22%7D%5B1m%5D)%20and%20on%20(env%2C%20fqdn)%20(pg_replication_is_replica%20%3D%3D%201))%0A%2F%0A(%20sum%20by%20(relname%2C%20indexrelname%2C%20fqdn)%20(rate(pg_stat_user_indexes_idx_scan%7Benv%3D%22gprd%22%2C%20relname%3D%22ci_job_artifacts%22%7D%5B1m%5D)%20and%20on%20(env%2C%20fqdn)%20(pg_replication_is_replica%20%3D%3D%201))%20%3E%201)&g0.tab=0&g0.stacked=0&g0.range_input=12h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.end_input=2021-11-22%2019%3A30%3A00&g0.moment_input=2021-11-22%2019%3A30%3A00). </details> ## Cleanup #### (WE ARE HERE) 1. Convert to `ops` and/or remove the `loop_limit` feature flag https://gitlab.com/gitlab-org/gitlab/-/issues/356319 2. Write a background migration to finalize the `locked` value updates for self-managed instances https://gitlab.com/gitlab-org/gitlab/-/issues/439509 3. Remove the Worker and Service classes in %"17.0" https://gitlab.com/gitlab-org/gitlab/-/issues/439508 ## Remaining Technical Debt With No Existing Exit Strategy This can arguably be moved out of the epic after we finished the **Cleanup** issues, but until then I think it makes more sense to keep them all conceptually grouped here until the **Cleanup** is finished. 1. Figure out what to do with https://gitlab.com/gitlab-org/gitlab/-/issues/392990+ 2. Because until we do, we can't https://gitlab.com/gitlab-org/gitlab/-/issues/367884+ #### Why would we ever care to pay off this technical debt? 1. Reduce cognitive load. `unknown` locked status is a great example of implementation that you can only understand if you understand how and why it was implemented at the time. 2. It will reduce the difficulty of moving forward with https://gitlab.com/groups/gitlab-org/-/epics/7097, which has value for our broader Artifact maturity efforts (see linked epics through there)