ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server

https://sentry.gitlab.net/gitlab/gitlabcom/issues/1154250/

PG::UnableToSend: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

  active_record/connection_adapters/postgresql_adapter.rb:611:in `async_exec_params'
    @connection.exec_params(sql, type_casted_binds)
  active_record/connection_adapters/postgresql_adapter.rb:611:in `block (2 levels) in exec_no_cache'
    @connection.exec_params(sql, type_casted_binds)
  active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    yield
  active_support/concurrency/share_lock.rb:187:in `yield_shares'
    yield
  active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    @lock.yield_shares(compatible: [:load]) do
...
(110 additional frame(s) were not displayed)

ActiveRecord::StatementInvalid: PG::UnableToSend: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
: DELETE FROM "uploads" WHERE "uploads"."uploader" = 'ImportExportUploader' AND "uploads"."model_type" = 'ImportExportUpload' AND "uploads"."model_id" = 34935 AND "uploads"."path" = 'import_export_upload/export_file/34935/2020-01-09_03-03-954_rhapsody.global_rhapsody_engin_export.tar.gz'
PG::UnableToSend: no connection to the server

  active_record/connection_adapters/postgresql/database_statements.rb:75:in `async_exec'
    @connection.async_exec(sql)
  active_record/connection_adapters/postgresql/database_statements.rb:75:in `block (2 levels) in execute'
    @connection.async_exec(sql)
  active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    yield
  active_support/concurrency/share_lock.rb:187:in `yield_shares'
    yield
  active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    @lock.yield_shares(compatible: [:load]) do
...
(87 additional frame(s) were not displayed)

ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server
: ROLLBACK
PG::UnableToSend: no connection to the server

  active_record/connection_adapters/postgresql_adapter.rb:611:in `async_exec_params'
    @connection.exec_params(sql, type_casted_binds)
  active_record/connection_adapters/postgresql_adapter.rb:611:in `block (2 levels) in exec_no_cache'
    @connection.exec_params(sql, type_casted_binds)
  active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    yield
  active_support/concurrency/share_lock.rb:187:in `yield_shares'
    yield
  active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    @lock.yield_shares(compatible: [:load]) do
...
(89 additional frame(s) were not displayed)

ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server
: SELECT  "project_mirror_data".* FROM "project_mirror_data" WHERE "project_mirror_data"."project_id" = 9174072 LIMIT 1

PG::UnableToSend: no connection to the server
: SELECT  "project_mirror_data".* FROM "project_mirror_data" WHERE "project_mirror_data"."project_id" = 9174072 LIMIT 1

added typebug label

Silver customer encountered this error while trying to export one of their projects. Customer does nightly backups of their projects (see ticket for details)

Internal links

• https://gitlab.zendesk.com/agent/tickets/143374
• https://log.gitlab.net/goto/8c02e98be7feff9a192421e659fbbfa0
• https://sentry.gitlab.net/gitlab/gitlabcom/issues/1154250/events/23448534/

@lmcandrew can we get someone to take a quick look and at least see if we have any kind of workaround for the customer? If not, we can add the needs investigation and leave it to the PM to prioritize

added customer label

added DEPRECATED_project export devopsmanage + 1 deleted label

mentioned in issue #196463 (closed)

This is a high severity issue for my company. We need to take backups of our code bases for legal reasons and it currently fails every time.

mentioned in issue gitlab-com/gl-infra/production#1564 (moved)

marked this issue as related to gitlab-com/gl-infra/production#1564 (moved)

Investigation: gitlab-com/gl-infra/production#1564 (moved)

possible cause: #34091 (closed)

mentioned in issue #197797 (closed)

other possibly related issues:

• #12107 (closed)
• #22081 (closed)
• #32613 (closed)

Silver SMB customer encountering export issues. Error message is different, but looks to be similar to other reports: https://gitlab.zendesk.com/agent/tickets/143939

PG::QueryCanceled: ERROR: canceling statement due to statement timeout : SELECT "snippets"."id" FROM "snippets" WHERE "snippets"."type" IN ('ProjectSnippet') AND "snippets"."project_id" = 4659683 ORDER BY "snippets"."id" ASC LIMIT 100

Bronze SMB customer encountering similar error message: https://gitlab.zendesk.com/agent/tickets/144326

mentioned in issue gitlab-org/quality/performance#174 (closed)

mentioned in issue #198979 (closed)

mentioned in issue #201628 (closed)

@georgekoltsov my initial thoughts are this is likely due to contention and scalability concerns involving the import process on .com?

Looking at the logs I can see that the exception is raised around export upload. Exception backtrace also shows use of transaction. Most likely what's happening is upload process, which is wrapped in a db transaction (which it shouldn't be), takes longer than db transaction idle_in_transaction_session_timeout.

This might be similar to upload issues that we've seen, that are also wrapped in a transaction.

#34091 (closed)

Also Stan's issue re: project exports is very similar to this one. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9011

mentioned in issue #204651 (closed)

marked this issue as related to #34091 (closed)

mentioned in issue #206814 (closed)

mentioned in issue #207751 (closed)

@georgekoltsov @lmcandrew What is our next step here?

See my previous comment #195981 (comment 287578361)

It looks like connection to db dropped due to hitting idle_in_transaction_session_timeout. One of the possible reasons why is use of upload parameter, which uploads the file within a db transaction, which causes this error.

The suggestion is to upgrade carrierwave, but that might bring other problems https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9011#note_275602494 WIP MR: !23002 (closed)

@georgekoltsov

upload process, which is wrapped in a db transaction (which it shouldn't be)

Would you please help me understand what's involved in unwrapping the upload process from the DB transaction?

The suggestion is to upgrade carrierwave

Could you please help me understand what this means (risk/effort) and if this is something that ~"group::import" can do, or if it involves other teams?

Carrierwave is used in every part of GitLab that deals with uploads (lfs, ci artifacts, attachments, import/export). Impact and risks are significant. Upgrading carrierwave is more of a project that would have to be split into multiple issues (e.g. removing bandaid fixes, upgrading gem, fixing any bubbled up issues, fixing any follow up hidden issues).

One thing that we could do is investigate this comment and see why we still hit transaction timeout, as we moved to upload using workhorse, which should have made an improvement in this area. But I am not sure if there is anything that could be done, since this issue is still happening after transition to upload using workhorse.

mentioned in issue #12107 (closed)

mentioned in issue #208613 (closed)

mentioned in issue #209697 (closed)

mentioned in issue #211273 (closed)

mentioned in issue #212097 (closed)

marked this issue as related to #12107 (closed)

added to epic &2959

changed milestone to %Backlog

added Category:Importers label

Silver customer still experiencing this issue: https://gitlab.zendesk.com/agent/tickets/150986 (Internal)

Are they utilizing upload param when doing export? That might be causing connection failure see https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9011#note_275602490

added [deprecated] Accepting merge requests label

changed epic to &3054

marked this issue as related to gitlab-com/gl-infra/production#2360 (moved)

https://sentry.gitlab.net/gitlab/gitlabcom/issues/1737190/?referrer=gitlab_plugin

Trial customer is experiencing this issue when attempting to export. ZD Ticket (Internal): https://gitlab.zendesk.com/agent/tickets/165105

Silver customer experiencing this issue: ZD Ticket (Internal): https://gitlab.zendesk.com/agent/tickets/170223

Applying severity2 label as there is no acceptable known workaround. While impact is only on specific projects, it does prevent a customer from doing exports / backups.

added severity2 label

added sectiondev label

ZD Ticket (Internal): https://gitlab.zendesk.com/agent/tickets/171696

added reproduced on GitLab.com label

Another customer who is doing regular exports seems to be hit by this (on GitLab.com).

In the past 7 days, this customer triggered 12 export jobs for 3 different projects and 4 of them failed (one project had 2 failures, the other 2 projects had 1 failure each).

Zendesk ticket: https://gitlab.zendesk.com/agent/tickets/174272

Kibana: https://log.gprd.gitlab.net/goto/202e9f72a4da57e79437dce262182a58

Sentry:

1. https://sentry.gitlab.net/gitlab/gitlabcom/issues/1737190/events/34329161/
2. https://sentry.gitlab.net/gitlab/gitlabcom/issues/1737190/events/34367930/

Note: I didn't manage to find 2 more entries in sentry, but they do exist in Kibana and seem identical.

added priority2 label

ZD Ticket: https://gitlab.zendesk.com/agent/tickets/177270

Export of 2 projects at about 4GB each returns this failure.

added SLOMissed label

mentioned in issue gitlab-com/support/support-team-meta#2920 (closed)

@hdelalic Would we be able to bring this out of the backlog to Next XX-XX release of some sort?

based on gitlab-com/support/support-team-meta#2920 (comment 451813167)

[This issue] seems to impact a lot of exports:

• Any repo that is beyond a few GB will typically run into this bug. This issue is 10 months old and is currently backlogged. This is probably the most prominent issue with exporting.

Prioritizing this alone could reduce the need for any other involvement.

@cynthia I am now trying to understand what's involved in fixing this so that I can give you a more accurate reply.

Thank you for your patience.

added internal customer label

mentioned in issue #289853 (closed)

marked this issue as related to #289853 (closed)

mentioned in issue #282503 (closed)

ZD Ticket: https://gitlab.zendesk.com/agent/tickets/190252

added infradev label

marked this issue as related to #282503 (closed)

Removing the infradev label because #282503 (closed) is what we need right now.

removed infradev label

mentioned in issue gitlab-com/support/support-team-meta#3096 (closed)

Adding Support Priority . See #282503 (comment 527397312) for more info

added Support Priority label

Another premium customer is facing the same issue (Internal link)

ZD ticket: https://gitlab.zendesk.com/agent/tickets/207227

added Importer:Project Export/Import label

mentioned in merge request !63350 (merged)

I think !63350 (merged) will solve this.

assigned to @stanhu

changed milestone to %14.0

mentioned in commit 2c641bb3

removed [deprecated] Accepting merge requests label

mentioned in issue gitlab-com/support/support-team-meta#3543 (closed)

mentioned in commit 9a5d932e

mentioned in commit fa4e7efb

Assuming this issue was caused by project exports, !63350 (merged) should hopefully solve this issue. There may be other places that idle in transaction too long and cause this issue, but we should open a new issue targeting that cause if this happens again. https://log.gprd.gitlab.net/goto/1d683425f3124fd295ecf8e399ee7d75 has a query of PG::UnableToSend over the last few days; it only popped up twice over the last few days.

closed

mentioned in issue #339608 (closed)

marked this issue as related to #339608 (closed)

added groupimport and integrate label and removed 1 deleted label