<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "type::bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary Deleting an Achievement through the `achievementsDelete` GraphQL mutation removes the `achievements` row but does not remove the associated avatar `Upload` record or the stored avatar file.\ \ The deletion path calls: `achievement.delete`\ \ This bypasses the model and CarrierWave destruction callbacks responsible for removing the avatar, its `Upload` record, and the stored file.\ \ The behavior was reproduced using the normal product APIs: - a disposable group was created through the REST API; - an Achievement with an avatar was created through the `achievementsCreate` GraphQL mutation; - the same Achievement was deleted through the `achievementsDelete` mutation.\ \ After deletion: - the Achievement no longer existed; - its previous avatar URL returned HTTP 404; - the associated `Upload` record still existed; - the physical avatar file still existed on disk.\ \ The attached PoC waits five seconds before inspecting the post-deletion state. In a separate manual observation, the row and file were still present several minutes later while Sidekiq was running. No explicit Sidekiq drain was performed.\ \ This report does not demonstrate that the deleted avatar remains accessible through its former HTTP URL. The demonstrated issue is orphaned database and storage data after the owning Achievement has been deleted. ### Steps to reproduce 1\. Prepare an isolated Docker-based GitLab instance and a disposable API token that can: - create a top-level group; - administer Achievements in that group. 2. Download the attached PoC: [poc.py](/uploads/01faa625a9e61d2a53bbc4d0ad41ba77/poc.py)\ \ 3. Ensure that Python 3 and the `requests` package are installed:\ \ `python3 -m pip install requests`\ \ \ 4. Export the disposable GitLab API token:\ \ `export GITLAB_POC_TOKEN='<disposable_api_token>'` \ \ 5. Run the PoC with the name of the Omnibus GitLab container: ``` python3 poc.py \ --instance http://127.0.0.1:8090 \ --gitlab-container gitlab \ --confirm-delete-resources \ --cleanup ``` \ Adjust the instance URL and container name for the isolated deployment. \ \ 6. The PoC performs the following sequence: - verifies that `gitlab-rails runner` works in the supplied container before creating any resources; - verifies the supplied API token; - creates a randomized disposable private group; - creates an Achievement with a one-pixel GIF avatar using the multipart `achievementsCreate` GraphQL mutation; - confirms through the Rails application model that the Achievement, `Upload` record, and stored file all exist; - verifies that the avatar URL initially returns HTTP 200; - deletes the Achievement using the `achievementsDelete` mutation; - waits five seconds; - checks the Achievement, `Upload`, and file state again; - requests the former avatar URL; - optionally removes the orphaned Upload, stored file, and disposable group. \ \ 7. The creation mutation uses the verified Group global ID format: ``` mutation($avatar: Upload!) { achievementsCreate( input: { namespaceId: "gid://gitlab/Group/<group_id>" name: "<randomized_name>" avatar: $avatar } ) { achievement { id name avatarUrl } errors } } ``` \ \ \ 8. The measured deletion uses: ``` mutation { achievementsDelete( input: { achievementId: "gid://gitlab/Achievements::Achievement/<achievement_id>" } ) { achievement { id name } errors } } ``` \ 9. After the mutation succeeds, native inspection shows that the Achievement is gone but its avatar Upload and stored file remain.\ \ The PoC prints its machine-readable result to stdout and status messages to stderr. It returns: - exit code `0` only when the deletion miss is reproduced; - exit code `2` on refusal, setup failure, or an unexpected result.\ \ The `--confirm-delete-resources` flag is required. A missing or incorrect container is rejected during preflight before the PoC creates any HTTP resources.\ \ With `--cleanup`, the PoC explicitly removes the orphaned file and Upload row and then deletes its randomized disposable group. ### Example Project No public example project is available because the reproduction creates and deletes disposable data on an isolated local GitLab instance. The attached PoC creates its own randomized private group and Achievement and does not require an existing project. ### What is the current _bug_ behavior? `Achievements::DestroyService` deletes the Achievement using:\ \ `achievement.delete`\ \ \ The Achievement row is removed, but the avatar cleanup lifecycle is bypassed.\ \ The observed result is: - the Achievement row is removed; - the former avatar URL returns HTTP 404; - the avatar’s `Upload` record remains with: - `model_type: "Achievements::Achievement"`; - `model_id` referencing the deleted Achievement; - `uploader: "AvatarUploader"`; - the physical avatar file remains in GitLab’s uploads storage.\ \ In a separate comparison on the same isolated instance, deleting an equivalent Achievement with `destroy!` removed: - the Achievement; - its avatar Upload row; - the stored avatar file.\ \ This confirms that the cleanup behavior exists but is bypassed by the `delete` call.\ \ A separate group-deletion comparison also confirmed that deleting the containing group later removes the orphaned Upload row through the `uploads.namespace_id` foreign key with `ON DELETE CASCADE`. However, that database cascade does not remove the stored file, leaving an untracked file on disk. ### What is the expected _correct_ behavior? Deleting an Achievement should also remove data exclusively owned by that Achievement, including: - its avatar `Upload` record; - any associated upload-state record; - the stored avatar file.\ \ The former avatar URL may continue returning HTTP 404, but the underlying database and storage data should not remain orphaned.\ \ Deletion should follow the same cleanup lifecycle demonstrated by destroying the Achievement with callbacks. ### Relevant logs and/or screenshots **Verified PoC result** The finalized attached PoC produced the following output. The randomized filename has been redacted, but the keys, IDs, and observed results are otherwise unchanged: ``` { "group_id": 10, "achievement_id": 2, "upload_id": 3, "avatar_http_before_delete": 200, "achievement_deleted": true, "upload_row_still_exists": true, "stored_file_still_exists": true, "avatar_http_after_delete": 404, "upload_path": "uploads/-/system/achievements/achievement/avatar/2/<redacted>.gif" } ``` \ \ The PoC exited with status `0` and reported: ``` BUG REPRODUCED: deleting the Achievement left its Upload row and stored avatar file behind. ``` \ \ Cleanup completed successfully: ``` Cleanup result: {"remaining_upload": false, "remaining_group": false} ``` \ \ The final targeted cleanup check returned: ``` { "groups": 0, "uploads": 0, "validation_tokens": 0 } ``` **Native confirmation**\ \ A separate native inspection confirmed that an Achievement had been deleted while its associated Upload remained: ``` Achievement: absent Upload: present model_type: Achievements::Achievement uploader: AvatarUploader Stored file: present ``` \ **Source-level cause**\ \ The Achievement model includes `Avatarable`: ``` include Avatarable ``` \ \ `Avatarable` mounts the avatar uploader: ``` mount_uploader :avatar, AvatarUploader ``` \ \ `AvatarUploader` includes `RecordsUploads::Concern`, which declares an upload-removal callback: ``` before :remove, :destroy_upload ``` \ \ That callback removes the corresponding Upload record for the stored path.\ \ However, the Achievement deletion service calls: ``` achievement.delete ``` \ \ Active Record `delete` removes the row without running the destruction callbacks used by the mounted uploader.\ \ The isolated `destroy!` comparison removed both the Upload row and stored file, supporting this causal explanation. **Sidekiq observation**\ \ Sidekiq was running during the reproduction: ``` Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1 ``` \ The attached PoC does not explicitly drain Sidekiq. The orphaned row and file were also observed several minutes after deletion while Sidekiq remained running. \ \ **Related issue**\ \ A related issue,[#554288\](https://gitlab.com/gitlab-org/gitlab/-/issues/554288), documented orphaned `Upload` records for another model type causing Geo verification and checksum failures.\ \ That issue concerned `Projects::ImportExport::RelationExportUpload`, while this report concerns Achievement avatars and `AvatarUploader`.\ \ This report demonstrates the same general orphaned-upload pattern but does not independently demonstrate Geo impact. #### Results of GitLab environment info Command: ``` docker exec <gitlab_container_name> gitlab-rake gitlab:env:info ``` <details> <summary>Expand for output related to GitLab environment info</summary> <pre> \\\\\\\ \\\`System information\\\ System:\\\ Proxy: no\\\ Current User: git\\\ Using RVM: no\\\ Ruby Version: 3.3.11\\\ Gem Version: 3.7.1\\\ Bundler Version: 4.0.10\\\ Rake Version: 13.4.2\\\ Redis Version: 7.2.14\\\ Sidekiq Version: 7.3.9\\\ Go Version: unknown\\\ \\\ GitLab information\\\ Version: 19.2.2-ee\\\ Revision: b6b3c0efcda\\\ Directory: /opt/gitlab/embedded/service/gitlab-rails\\\ DB Adapter: PostgreSQL\\\ DB Version: 17.10\\\ URL: http://localhost:8090\\\ HTTP Clone URL: http://localhost:8090/some-group/some-project.git\\\ SSH Clone URL: ssh://git@localhost:2223/some-group/some-project.git\\\ Elasticsearch: no\\\ Geo: no\\\ Using LDAP: no\\\ Using Omniauth: yes\\\ Omniauth Providers:\\\ \\\ GitLab Shell\\\ Version: 14.56.1\\\ Repository storages:\\\ - default: unix:/var/opt/gitlab/gitaly/gitaly.socket\\\ GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell\\\ \\\ Gitaly\\\ - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket\\\ - default Version: 19.2.2\\\ - default Git Version: 2.55.0-rc1.g02bb39c\\\` </pre> </details> #### Results of GitLab application Check Command: ``` docker exec <gitlab_container_name> gitlab-rake gitlab:check SANITIZE=true ``` <details> <summary>Expand for output related to the GitLab application check</summary> <pre> \\\`Checking GitLab subtasks ...\\\ \\\ Checking GitLab Shell ...\\\ \\\ GitLab Shell: ... GitLab Shell version \\\>= 14.56.1 ? ... OK (14.56.1)\\\ Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check\\\ Internal API available: OK\\\ Redis available via internal API: OK\\\ gitlab-shell self-check successful\\\ \\\ Checking GitLab Shell ... Finished\\\ \\\ Checking Gitaly ...\\\ \\\ Gitaly: ... default ... OK\\\ \\\ Checking Gitaly ... Finished\\\ \\\ Checking Sidekiq ...\\\ \\\ Sidekiq: ... Running? ... yes\\\ Number of Sidekiq processes (cluster/worker) ... 1/1\\\ \\\ Checking Sidekiq ... Finished\\\ \\\ Checking Incoming Email ...\\\ \\\ Incoming Email: ... Reply by email is disabled in config/gitlab.yml\\\ \\\ Checking Incoming Email ... Finished\\\ \\\ Checking LDAP ...\\\ \\\ LDAP: ... LDAP is disabled in config/gitlab.yml\\\ \\\ Checking LDAP ... Finished\\\ \\\ Checking GitLab App ...\\\ \\\ Database config exists? ... yes\\\ Tables are truncated? ... skipped\\\ All migrations up? ... yes\\\ Database contains orphaned GroupMembers? ... no\\\ GitLab config exists? ... yes\\\ GitLab config up to date? ... yes\\\ Cable config exists? ... yes\\\ Resque config exists? ... yes\\\ Log directory writable? ... yes\\\ Tmp directory writable? ... yes\\\ Uploads directory exists? ... yes\\\ Uploads directory has correct permissions? ... yes\\\ Uploads directory tmp has correct permissions? ... yes\\\ Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)\\\ Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither systemd units nor init scripts)\\\ Projects have namespace: ... can't check, you have no projects\\\ Redis version \\\>= 6.2.14? ... yes\\\ Ruby version \\\>= 3.0.6 ? ... yes (3.3.11)\\\ Git user has default SSH configuration? ... yes\\\ Active users: ... 3\\\ Is authorized keys file accessible? ... yes\\\ GitLab configured to store new projects in hashed storage? ... yes\\\ All projects are in hashed storage? ... yes\\\ Elasticsearch version 7.x-9.x or OpenSearch version 1.x-3.x (9.x recommended for production) ... skipped (advanced search is disabled)\\\ All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)\\\ \\\ Checking GitLab App ... Finished\\\ \\\ Checking GitLab subtasks ... Finished\\\` </pre> </details> ### Possible fixes The smallest fix appears to be changing the Achievement deletion service to use the normal destruction lifecycle: `achievement.destroy!` instead of: `achievement.delete` This approach was tested on an equivalent Achievement with an avatar and removed: - the Achievement; - its Upload record; - its stored avatar file.\ \ If `delete` is required for another reason, the service should explicitly remove the mounted avatar before deleting the Achievement.\ \ Existing orphaned Upload rows and files may also require a targeted cleanup mechanism. The cleanup should account for both: - orphaned Upload database records; - files whose Upload rows may already have been removed by a namespace database cascade.\ \ Regression coverage could verify that: 1. an Achievement with an avatar is created through the normal service or GraphQL mutation; 2. an Upload row and stored avatar file exist; 3. `Achievements::DestroyService` deletes the Achievement; 4. the Upload row is removed; 5. the stored file is removed; 6. awarded or revoked user-achievement records still follow their existing foreign-key behavior; 7. deleting one Achievement does not remove another Achievement’s avatar; 8. cleanup works with both local and object storage configurations. <!--If you don't have /label privileges, follow up with an issue comment of `@gitlab-bot label ~"type::bug"`-->