Job artifacts without expire_in wouldn't be expired with instance wide expire policy if they were create when instance-wide expire policy wasn't set up

Summary

Job artifacts without expire_in wouldn't be expired with instance wide expire policy if artefact were created when instance-wide expire policy wasn't set up. (Summarize the bug encountered concisely)

Steps to reproduce

  1. Disable instance-wide artifacts expire policy (https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#default-artifacts-expiration-core-only)
  2. Run some jobs that produce artifacts without set up expire_in property
  3. Enable instance-wide artifacts expire policy

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)

What is the current bug behavior?

Artifacts wouldn't deleted.

What is the expected correct behavior?

Artifacts deleted with instance-wide artifacts expire policy

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		CentOS 7.6.1810
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	3.0.6
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.10
Git Version:	2.22.1
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.2.4
Revision:	Unknown
Directory:	/home/git/gitlab
DB Adapter:	PostgreSQL
DB Version:	11.5
URL:		https://git.wrke.in
HTTP Clone URL:	https://git.wrke.in/some-group/some-project.git
SSH Clone URL:	git@git.wrke.in:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	no


GitLab Shell
Version:	9.3.0
Repository storage paths:
  • default: /home/git/repositories GitLab Shell path: /home/git/gitlab-shell Git: /opt/3rd/git/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /home/git/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
not verifying SSL hostname of LDAPS server 'directory.team.wrike.com:636'
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... no
Try fixing it:
Install the init script
For more information see:
doc/install/installation.md in section "Install Init Script"
Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...


... Some logs removed ...


Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... yes (2.22.1)
Git user has default SSH configuration? ... no
Try fixing it:
mkdir ~/gitlab-check-backup-1568385120
sudo mv /home/git/.ssh/id_repush ~/gitlab-check-backup-1568385120
For more information see:
doc/ssh/README.md in section "SSH on the GitLab server"
Please fix the error above and rerun the checks.
Active users: ... 388


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

If I right understood currently if job haven't got expire_in property and instance-wide then expire_at field of ci_jobs_artifacts table is set to NULL.

New destroy expired artifacts job uses expired scope to get list of expired artifacts.

expired scope selects only job artifacts with expired_at field less then current time, but it doesn't interacts with artifacts that have NULL in expired_at column.

I think there should be something like code below in case of default expire time not 0.

scope :expired, -> (limit) { where('(expire_at < ?) OR (file <> \'job.log\' AND expire_at IS NULL AND created_at < ?)', *[Time.now, Time.now - ChronicDuration.parse(Gitlab::CurrentSettings.current_application_settings.default_artifacts_expire_in)]).limit(limit) }

Here's my MR -> !16702 (closed)

Edited by James Heimbuck