Error during project transfer - db related

Summary

During transfer of project from one namespace to another via project configuration tab, error 404 appears instead of successful transfer of repo. This is a regression after one of updates in last 3 months (we did this before without any issues). I've found similar issue in omnibus-gitlab#5671 (closed) - but recommended mitigation was not successful, as there was no entries in future. Dates on both db server and gitlab instance are set in same way (UTC + NTP).

Steps to reproduce

Very hard to describe, as it may be a cornercase:

• have a long-running gitlab instance (upgraded regularly from 9.x upwards)
• create two namespaces (A and B)
• create project in namespace A
• try to transfer it from A to B
• our effect - 404 and log that is below

What is the current bug behavior?

Simply put - it does not allow to transfer projects between namespaces.

What is the expected correct behavior?

Allowing to transfer project between namespaces without issues.

Relevant logs and/or screenshots

{"severity":"ERROR","time":"2021-01-20T10:06:10.186Z","correlation_id":"<removed>","tags.correlation_id":"<removed>","tags.locale":"en","user.id":1,"user.email":"","user.username":"",
"extra.audit_event_type":"AuditEvent","extra.sql":"INSERT INTO \"audit_events\" (\"author_id\", \"entity_id\", \"entity_type\", \"details\", \"created_at\", \"author_name\", \"target_details\", \"target_type\", \"target_id\") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING \"id\"","exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::CheckViolation: ERROR:  no partition of relation \"audit_events_part_5fc467ac26\" found for row\nDETAIL: 
Partition key of the failing row contains (created_at) = (2021-01-20 10:06:10.182101).\nCONTEXT:  SQL statement \"INSERT INTO audit_events_part_5fc467ac26 (id,\n    author_id,\n    entity_id,\n    entity_type,\n    details,\n    ip_address,\n    author_name,\n    entity_path,\n    target_details,\n    target_type,\n    target_id,\n    created_at)\n  VALUES (NEW.id,\n    NEW.author_id,\n    NEW.entity_id,\n    NEW.entity_type,\n    NEW.details,\n    NEW.ip_address,\n    NEW.author_name,\n    NEW.entity_path,\n    NEW.target_details,\n    NEW.target_type,\n    NEW.target_id,\n    NEW.created_at)\"\nPL/pgSQL function table_sync_function_2be879775d() line 20 at SQL statement\n",
"exception.backtrace":["app/services/audit_event_service.rb:133:in `save_or_track'","app/services/audit_event_service.rb:118:in `log_security_event_to_database'","app/services/audit_event_service.rb:46:in `security_event'","ee/app/services/ee/audit_event_service.rb:123:in `security_event'","ee/lib/audit/changes.rb:70:in `audit_event'","ee/lib/audit/changes.rb:25:in `audit_changes'","ee/lib/ee/audit/project_changes_auditor.rb:10:in `execute'","ee/app/services/ee/projects/transfer_service.rb:14:in `execute_system_hooks'","app/services/projects/transfer_service.rb:96:in `block in proceed_to_transfer'","app/services/projects/transfer_service.rb:71:in `proceed_to_transfer'","app/services/projects/transfer_service.rb:62:in `transfer'","app/services/projects/transfer_service.rb:27:in `execute'","app/controllers/projects_controller.rb:112:in `transfer'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'","app/controllers/application_controller.rb:494:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:485:in `set_session_storage'","lib/gitlab/i18n.rb:73:in `with_locale'","lib/gitlab/i18n.rb:79:in `with_user_locale'","app/controllers/application_controller.rb:479:in `set_locale'","lib/gitlab/error_tracking.rb:52:in `with_context'","app/controllers/application_controller.rb:544:in `sentry_context'","app/controllers/application_controller.rb:472:in `block in set_current_context'","lib/gitlab/application_context.rb:54:in `block in use'","lib/gitlab/application_context.rb:54:in `use'","lib/gitlab/application_context.rb:21:in `with_context'","app/controllers/application_controller.rb:464:in `set_current_context'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:234:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:23:in `call'","config/initializers/fix_local_cache_middleware.rb:9:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"]}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

```
System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.1
Redis Version:  5.0.9
Git Version:    2.29.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.7.3-ee
Revision:       925686b2c52
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.3
URL:            
HTTP Clone URL: 
SSH Clone URL:  
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.14.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git
```

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.14.0 ? ... OK (13.14.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
158/13 ... yes
6/35 ... yes
6/40 ... yes
158/51 ... yes
37/78 ... yes
6/80 ... yes
37/83 ... yes
27/84 ... yes
6/87 ... yes
6/90 ... yes
27/92 ... yes
11/94 ... yes
6/106 ... yes
177/107 ... yes
158/115 ... yes
158/121 ... yes
176/122 ... yes
158/123 ... yes
31/127 ... yes
27/132 ... yes
15/139 ... yes
158/140 ... yes
36/147 ... yes
142/158 ... yes
27/159 ... yes
158/170 ... yes
158/175 ... yes
6/176 ... yes
158/178 ... yes
6/179 ... yes
176/182 ... yes
120/189 ... yes
177/197 ... yes
158/198 ... yes
177/205 ... yes
47/207 ... yes
98/208 ... yes
158/209 ... yes
98/210 ... yes
133/211 ... yes
98/212 ... yes
54/220 ... yes
15/223 ... yes
15/224 ... yes
15/231 ... yes
15/238 ... yes
47/251 ... yes
66/263 ... yes
64/268 ... yes
64/269 ... yes
31/272 ... yes
109/273 ... yes
109/277 ... yes
176/283 ... yes
176/286 ... yes
37/287 ... yes
109/290 ... yes
63/292 ... yes
47/293 ... yes
63/300 ... yes
64/301 ... yes
64/307 ... yes
12/309 ... yes
52/315 ... yes
72/322 ... yes
12/323 ... yes
72/326 ... yes
37/327 ... yes
15/329 ... yes
37/337 ... yes
177/338 ... yes
176/343 ... yes
64/349 ... yes
47/352 ... yes
71/353 ... yes
109/357 ... yes
176/358 ... yes
176/359 ... yes
47/362 ... yes
37/363 ... yes
109/367 ... yes
158/373 ... yes
47/378 ... yes
6/383 ... yes
98/388 ... yes
64/394 ... yes
133/395 ... yes
84/396 ... yes
98/400 ... yes
75/417 ... yes
84/418 ... yes
158/421 ... yes
98/428 ... yes
47/429 ... yes
134/430 ... yes
64/431 ... yes
219/432 ... yes
36/433 ... yes
204/434 ... yes
37/435 ... yes
36/436 ... yes
106/437 ... yes
108/440 ... yes
108/441 ... yes
108/442 ... yes
64/443 ... yes
6/444 ... yes
12/447 ... yes
108/448 ... yes
55/449 ... yes
115/452 ... yes
109/454 ... yes
37/458 ... yes
98/465 ... yes
104/466 ... yes
109/467 ... yes
98/468 ... yes
54/469 ... yes
120/470 ... yes
83/471 ... yes
6/472 ... yes
106/473 ... yes
120/474 ... yes
98/476 ... yes
98/477 ... yes
64/478 ... yes
98/479 ... yes
98/480 ... yes
98/481 ... yes
120/482 ... yes
98/483 ... yes
31/484 ... yes
133/486 ... yes
98/487 ... yes
98/489 ... yes
158/490 ... yes
98/491 ... yes
98/492 ... yes
98/493 ... yes
98/494 ... yes
98/495 ... yes
108/496 ... yes
108/497 ... yes
37/498 ... yes
158/499 ... yes
106/503 ... yes
176/504 ... yes
130/505 ... yes
176/506 ... yes
109/507 ... yes
122/508 ... yes
133/512 ... yes
176/513 ... yes
6/514 ... yes
109/515 ... yes
130/517 ... yes
130/518 ... yes
37/519 ... yes
131/520 ... yes
158/521 ... yes
276/522 ... yes
176/523 ... yes
177/524 ... yes
134/525 ... yes
138/526 ... yes
138/528 ... yes
131/531 ... yes
131/534 ... yes
142/537 ... yes
98/538 ... yes
138/539 ... yes
113/540 ... yes
150/541 ... yes
150/542 ... yes
256/545 ... yes
6/547 ... yes
109/549 ... yes
131/550 ... yes
133/551 ... yes
158/552 ... yes
138/553 ... yes
12/554 ... yes
109/555 ... yes
109/557 ... yes
146/558 ... yes
158/559 ... yes
120/560 ... yes
109/561 ... yes
109/562 ... yes
109/563 ... yes
109/564 ... yes
109/565 ... yes
158/566 ... yes
64/567 ... yes
106/569 ... yes
55/571 ... yes
91/572 ... yes
119/573 ... yes
121/574 ... yes
162/575 ... yes
83/576 ... yes
188/577 ... yes
138/578 ... yes
107/579 ... yes
37/581 ... yes
82/586 ... yes
15/587 ... yes
98/590 ... yes
165/591 ... yes
131/592 ... yes
166/593 ... yes
166/594 ... yes
146/596 ... yes
158/597 ... yes
37/600 ... yes
98/601 ... yes
138/602 ... yes
131/604 ... yes
15/605 ... yes
15/606 ... yes
55/607 ... yes
55/608 ... yes
109/609 ... yes
107/611 ... yes
158/612 ... yes
177/613 ... yes
172/615 ... yes
158/618 ... yes
98/619 ... yes
173/620 ... yes
109/621 ... yes
165/622 ... yes
98/623 ... yes
165/624 ... yes
197/626 ... yes
109/627 ... yes
37/629 ... yes
217/630 ... yes
131/631 ... yes
158/634 ... yes
158/635 ... yes
109/636 ... yes
182/637 ... yes
98/638 ... yes
109/639 ... yes
98/640 ... yes
109/641 ... yes
131/642 ... yes
27/643 ... yes
165/644 ... yes
98/645 ... yes
158/647 ... yes
109/648 ... yes
177/649 ... yes
109/650 ... yes
142/651 ... yes
98/652 ... yes
188/655 ... yes
113/656 ... yes
98/657 ... yes
37/658 ... yes
138/659 ... yes
131/661 ... yes
98/666 ... yes
197/667 ... yes
197/668 ... yes
197/670 ... yes
142/671 ... yes
197/672 ... yes
133/673 ... yes
199/676 ... yes
173/678 ... yes
109/679 ... yes
109/680 ... yes
276/681 ... yes
138/683 ... yes
204/684 ... yes
98/685 ... yes
98/686 ... yes
207/688 ... yes
75/689 ... yes
177/691 ... yes
197/692 ... yes
207/693 ... yes
109/696 ... yes
138/697 ... yes
98/698 ... yes
197/699 ... yes
197/700 ... yes
203/701 ... yes
98/702 ... yes
98/703 ... yes
142/704 ... yes
205/705 ... yes
6/706 ... yes
197/707 ... yes
217/709 ... yes
204/710 ... yes
98/712 ... yes
98/713 ... yes
217/714 ... yes
98/715 ... yes
217/720 ... yes
98/722 ... yes
75/723 ... yes
197/724 ... yes
162/725 ... yes
217/726 ... yes
204/727 ... yes
220/728 ... yes
220/729 ... yes
220/730 ... yes
217/731 ... yes
197/733 ... yes
197/734 ... yes
197/735 ... yes
6/736 ... yes
202/737 ... yes
197/738 ... yes
133/739 ... yes
197/740 ... yes
27/741 ... yes
133/742 ... yes
98/743 ... yes
142/744 ... yes
108/746 ... yes
66/747 ... yes
217/749 ... yes
220/751 ... yes
108/753 ... yes
98/754 ... yes
199/756 ... yes
228/758 ... yes
228/759 ... yes
37/760 ... yes
205/761 ... yes
185/762 ... yes
109/763 ... yes
205/764 ... yes
98/766 ... yes
228/767 ... yes
109/769 ... yes
205/770 ... yes
217/772 ... yes
217/773 ... yes
236/774 ... yes
15/775 ... yes
205/776 ... yes
121/777 ... yes
217/778 ... yes
109/779 ... yes
37/780 ... yes
228/781 ... yes
217/782 ... yes
98/785 ... yes
37/786 ... yes
130/787 ... yes
204/789 ... yes
158/790 ... yes
27/791 ... yes
185/792 ... yes
195/793 ... yes
197/794 ... yes
146/795 ... yes
197/796 ... yes
197/797 ... yes
197/798 ... yes
197/799 ... yes
214/800 ... yes
196/801 ... yes
205/802 ... yes
98/803 ... yes
197/804 ... yes
197/805 ... yes
223/807 ... yes
109/808 ... yes
230/809 ... yes
197/810 ... yes
177/811 ... yes
6/812 ... yes
230/813 ... yes
162/814 ... yes
228/815 ... yes
205/816 ... yes
242/817 ... yes
146/821 ... yes
177/822 ... yes
194/823 ... yes
98/825 ... yes
195/826 ... yes
221/827 ... yes
221/829 ... yes
230/831 ... yes
177/833 ... yes
109/834 ... yes
221/835 ... yes
276/836 ... yes
109/837 ... yes
158/838 ... yes
253/839 ... yes
247/840 ... yes
197/841 ... yes
204/842 ... yes
98/843 ... yes
113/844 ... yes
230/845 ... yes
230/846 ... yes
255/847 ... yes
98/848 ... yes
138/849 ... yes
217/850 ... yes
75/851 ... yes
194/852 ... yes
161/853 ... yes
221/854 ... yes
161/855 ... yes
245/856 ... yes
158/858 ... yes
185/859 ... yes
245/862 ... yes
245/863 ... yes
98/865 ... yes
109/866 ... yes
55/868 ... yes
98/870 ... yes
37/871 ... yes
109/872 ... yes
138/873 ... yes
266/874 ... yes
217/875 ... yes
230/876 ... yes
158/877 ... yes
276/878 ... yes
109/879 ... yes
217/881 ... yes
177/882 ... yes
205/884 ... yes
98/885 ... yes
228/886 ... yes
205/888 ... yes
205/889 ... yes
259/890 ... yes
177/891 ... yes
37/893 ... yes
37/894 ... yes
98/895 ... yes
109/896 ... yes
98/898 ... yes
197/900 ... yes
197/901 ... yes
197/902 ... yes
197/903 ... yes
269/912 ... yes
98/913 ... yes
98/915 ... yes
270/916 ... yes
98/918 ... yes
109/920 ... yes
276/921 ... yes
270/922 ... yes
109/923 ... yes
109/924 ... yes
274/925 ... yes
274/926 ... yes
109/927 ... yes
109/928 ... yes
205/929 ... yes
98/930 ... yes
269/931 ... yes
220/932 ... yes
230/933 ... yes
205/934 ... yes
37/935 ... yes
205/936 ... yes
177/937 ... yes
252/940 ... yes
98/941 ... yes
205/942 ... yes
205/943 ... yes
158/944 ... yes
109/945 ... yes
98/946 ... yes
109/947 ... yes
55/948 ... yes
98/949 ... yes
267/950 ... yes
220/951 ... yes
158/952 ... yes
230/953 ... yes
220/954 ... yes
109/955 ... yes
37/956 ... yes
109/957 ... yes
37/958 ... yes
282/959 ... yes
247/960 ... yes
197/961 ... yes
158/962 ... yes
282/963 ... yes
282/964 ... yes
284/965 ... yes
109/966 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 65
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes