Skip to content

Remove DB transaction from Rebase Gitaly operation

Oswaldo Ferreira requested to merge osw-remove-transaction-while-rebasing into master

What does this MR do?

It removes a transaction that was enclosing an expensive Gitaly operation (Rebase). I was able to reproduce locally, and we were seeing this when it surpassed the idle_in_transaction_session_timeout DB setting:

2019-11-25_19:30:48.04046 rails-background-jobs : 2019-11-25T19:30:48.040Z 84652 TID-ovz5hpums RebaseWorker JID-289d814f770bd2faad9a4159 INFO: start
2019-11-25_19:30:48.23688 rails-background-jobs : 2019-11-25T19:30:48.236Z 84652 TID-ovz5hpums RebaseWorker JID-289d814f770bd2faad9a4159 INFO: arguments: [37,1]
2019-11-25_19:31:03.59492 rails-background-jobs : FATAL:  terminating connection due to idle-in-transaction timeout
2019-11-25_19:31:03.60105 rails-background-jobs : 2019-11-25T19:31:03.600Z 84652 TID-ovz5hpums RebaseWorker JID-289d814f770bd2faad9a4159 INFO: fail: 15.561 sec
2019-11-25_19:31:03.60109 rails-background-jobs : 2019-11-25T19:31:03.601Z 84652 TID-ovz5hpums WARN: {"context":"Job raised exception","job":{"class":"RebaseWorker","args":[37,1],"retry":3,"queue":"rebase","backtrace":true,"jid":"289d814f770bd2faad9a4159","created_at":1574710248.0318432,"correlation_id":"u2Ri7Qp0K52","uber-trace-id":"5ee8f57aacd4796:33d77a7051b6a180:394388d47db97b8:1","enqueued_at":1574710248.0388129},"jobstr":"{\"class\":\"RebaseWorker\",\"args\":[37,1],\"retry\":3,\"queue\":\"rebase\",\"backtrace\":true,\"jid\":\"289d814f770bd2faad9a4159\",\"created_at\":1574710248.0318432,\"correlation_id\":\"u2Ri7Qp0K52\",\"uber-trace-id\":\"5ee8f57aacd4796:33d77a7051b6a180:394388d47db97b8:1\",\"enqueued_at\":1574710248.0388129}"}
2019-11-25_19:31:03.60110 rails-background-jobs : 2019-11-25T19:31:03.601Z 84652 TID-ovz5hpums WARN: ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server
2019-11-25_19:31:03.60112 rails-background-jobs : : UPDATE "merge_requests" SET "rebase_jid" = $1 WHERE "merge_requests"."id" = $2
2019-11-25_19:31:03.60113 rails-background-jobs : 2019-11-25T19:31:03.601Z 84652 TID-ovz5hpums WARN: /Users/osw/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/postgresql_adapter.rb:611:in `async_exec_params'

Instead, now we rescue from possible Gitaly errors, clean the rebase_commit_sha column then raise the error back.

More information can be found at #33943 (closed).

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Oswaldo Ferreira

Merge request reports