Geo: Namespaces cannot be deleted due to "Invalid argument for string field"
Sentry error: https://sentry.gitlap.com/gitlab/gitlabcom/issues/190450/
TypeError: Invalid argument for string field.
from gitlab/gitaly_client/namespace_service.rb:21:in `initialize'
from gitlab/gitaly_client/namespace_service.rb:21:in `new'
from gitlab/gitaly_client/namespace_service.rb:21:in `remove'
from gitlab/shell.rb:322:in `block in rm_namespace'
from gitlab/gitaly_client.rb:268:in `block (2 levels) in migrate'
from gitlab/gitaly_client.rb:312:in `allow_n_plus_1_calls'
from gitlab/gitaly_client.rb:261:in `block in migrate'
from gitlab/metrics/influx_db.rb:103:in `measure'
from gitlab/gitaly_client.rb:259:in `migrate'
from gitlab/shell.rb:319:in `rm_namespace'
from gitlab/metrics/instrumentation.rb:159:in `block in rm_directory'
from gitlab/metrics/method_call.rb:36:in `measure'
from gitlab/metrics/instrumentation.rb:159:in `rm_directory'
from gitlab/shell.rb:186:in `remove_repository'
from gitlab/metrics/instrumentation.rb:159:in `block in remove_repository'
from gitlab/metrics/method_call.rb:36:in `measure'
from gitlab/metrics/instrumentation.rb:159:in `remove_repository'
from gitlab_shell_worker.rb:6:in `perform'
from sidekiq/processor.rb:188:in `execute_job'
from sidekiq/processor.rb:170:in `block (2 levels) in process'
from sidekiq/middleware/chain.rb:128:in `block in invoke'
from gitlab/sidekiq_status/server_middleware.rb:5:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from gitlab/sidekiq_middleware/shutdown.rb:52:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from gitlab/sidekiq_middleware/arguments_logger.rb:6:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from gitlab/metrics/sidekiq_middleware.rb:13:in `block in call'
from gitlab/metrics/transaction.rb:53:in `run'
from gitlab/metrics/sidekiq_middleware.rb:13:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from sidekiq/middleware/server/active_record.rb:15:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from raven/integrations/sidekiq.rb:9:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from sidekiq/middleware/chain.rb:133:in `invoke'
from sidekiq/processor.rb:169:in `block in process'
from sidekiq/processor.rb:141:in `block (6 levels) in dispatch'
from sidekiq/job_retry.rb:97:in `local'
from sidekiq/processor.rb:140:in `block (5 levels) in dispatch'
from sidekiq.rb:36:in `block in <module:Sidekiq>'
from sidekiq/processor.rb:136:in `block (4 levels) in dispatch'
from sidekiq/processor.rb:204:in `stats'
from sidekiq/processor.rb:131:in `block (3 levels) in dispatch'
from sidekiq/job_logger.rb:7:in `call'
from sidekiq/processor.rb:130:in `block (2 levels) in dispatch'
from sidekiq/job_retry.rb:72:in `global'
from sidekiq/processor.rb:129:in `block in dispatch'
from sidekiq/logging.rb:44:in `with_context'
from sidekiq/logging.rb:38:in `with_job_hash_context'
from sidekiq/processor.rb:128:in `dispatch'
from sidekiq/processor.rb:168:in `process'
from sidekiq/processor.rb:85:in `process_one'
from sidekiq/processor.rb:73:in `run'
from sidekiq/util.rb:16:in `watchdog'
from sidekiq/util.rb:25:in `block in safe_thread'
It's possible Gitaly needs a restart:
file-20-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 8486) 144027s; run: log: (pid 1055) 147110s
file-10-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 484) 143645s; run: log: (pid 1572) 1564768s
file-04-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 16266) 143469s; run: log: (pid 30955) 6951919s
file-03-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 15858) 143390s; run: log: (pid 30975) 6951890s
file-02-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 16162) 142598s; run: log: (pid 1525) 4330717s
file-16-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 21112) 143478s; run: log: (pid 30947) 6951959s
file-01-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 11689) 143982s; run: log: (pid 31002) 6951980s
file-08-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 20405) 142131s; run: log: (pid 31138) 6951772s
file-12-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 5970) 143561s; run: log: (pid 30965) 6951991s
file-09-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 10303) 142317s; run: log: (pid 1902) 4760677s
file-07-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 4145) 142414s; run: log: (pid 1589) 4765011s
file-11-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 16168) 142846s; run: log: (pid 30993) 6951843s
file-05-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 6878) 143580s; run: log: (pid 1436) 720446s
file-17-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 8495) 144036s; run: log: (pid 963) 147102s
file-06-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 18223) 144015s; run: log: (pid 30978) 6951873s
file-13-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 10493) 142890s; run: log: (pid 1710) 665587s
file-14-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 27023) 142484s; run: log: (pid 1694) 4314038s
file-15-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 15129) 143099s; run: log: (pid 1474) 6395724s
file-18-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 8405) 144032s; run: log: (pid 984) 147104s
file-19-stor-gprd.c.gitlab-production.internal run: gitaly: (pid 8371) 144034s; run: log: (pid 982) 147100s