Job artifacts without expire_in wouldn't be expired with instance wide expire policy if they were create when instance-wide expire policy wasn't set up
Summary
Job artifacts without expire_in wouldn't be expired with instance wide expire policy if artefact were created when instance-wide expire policy wasn't set up. (Summarize the bug encountered concisely)
Steps to reproduce
- Disable instance-wide artifacts expire policy (https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#default-artifacts-expiration-core-only)
- Run some jobs that produce artifacts without set up
expire_in
property - Enable instance-wide artifacts expire policy
Example Project
(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)
(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)
What is the current bug behavior?
Artifacts wouldn't deleted.
What is the expected correct behavior?
Artifacts deleted with instance-wide artifacts expire policy
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: CentOS 7.6.1810 Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 3.0.6 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.10 Git Version: 2.22.1 Sidekiq Version:5.2.7 Go Version: unknownGitLab information Version: 12.2.4 Revision: Unknown Directory: /home/git/gitlab DB Adapter: PostgreSQL DB Version: 11.5 URL: https://git.wrke.in HTTP Clone URL: https://git.wrke.in/some-group/some-project.git SSH Clone URL: git@git.wrke.in:some-group/some-project.git Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 9.3.0 Repository storage paths:
- default: /home/git/repositories GitLab Shell path: /home/git/gitlab-shell Git: /opt/3rd/git/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0) Running /home/git/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /home/git/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain not verifying SSL hostname of LDAPS server 'directory.team.wrike.com:636' LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... no Try fixing it: Install the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Init script up-to-date? ... can't check because of previous errors Projects have namespace: ...
... Some logs removed ...
Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.1) Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1568385120 sudo mv /home/git/.ssh/id_repush ~/gitlab-check-backup-1568385120 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. Active users: ... 388
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
If I right understood currently if job haven't got expire_in
property and instance-wide then expire_at
field of ci_jobs_artifacts
table is set to NULL
.
New destroy expired artifacts job uses expired
scope to get list of expired artifacts.
expired
scope selects only job artifacts with expired_at
field less then current time, but it doesn't interacts with artifacts that have NULL
in expired_at
column.
I think there should be something like code below in case of default expire time not 0.
scope :expired, -> (limit) { where('(expire_at < ?) OR (file <> \'job.log\' AND expire_at IS NULL AND created_at < ?)', *[Time.now, Time.now - ChronicDuration.parse(Gitlab::CurrentSettings.current_application_settings.default_artifacts_expire_in)]).limit(limit) }
Here's my MR -> !16702 (closed)