Remove DB transaction from Rebase Gitaly operation
What does this MR do?
It removes a transaction that was enclosing an expensive Gitaly operation (Rebase). I was able to reproduce locally, and we were seeing this when it surpassed the idle_in_transaction_session_timeout
DB setting:
2019-11-25_19:30:48.04046 rails-background-jobs : 2019-11-25T19:30:48.040Z 84652 TID-ovz5hpums RebaseWorker JID-289d814f770bd2faad9a4159 INFO: start
2019-11-25_19:30:48.23688 rails-background-jobs : 2019-11-25T19:30:48.236Z 84652 TID-ovz5hpums RebaseWorker JID-289d814f770bd2faad9a4159 INFO: arguments: [37,1]
2019-11-25_19:31:03.59492 rails-background-jobs : FATAL: terminating connection due to idle-in-transaction timeout
2019-11-25_19:31:03.60105 rails-background-jobs : 2019-11-25T19:31:03.600Z 84652 TID-ovz5hpums RebaseWorker JID-289d814f770bd2faad9a4159 INFO: fail: 15.561 sec
2019-11-25_19:31:03.60109 rails-background-jobs : 2019-11-25T19:31:03.601Z 84652 TID-ovz5hpums WARN: {"context":"Job raised exception","job":{"class":"RebaseWorker","args":[37,1],"retry":3,"queue":"rebase","backtrace":true,"jid":"289d814f770bd2faad9a4159","created_at":1574710248.0318432,"correlation_id":"u2Ri7Qp0K52","uber-trace-id":"5ee8f57aacd4796:33d77a7051b6a180:394388d47db97b8:1","enqueued_at":1574710248.0388129},"jobstr":"{\"class\":\"RebaseWorker\",\"args\":[37,1],\"retry\":3,\"queue\":\"rebase\",\"backtrace\":true,\"jid\":\"289d814f770bd2faad9a4159\",\"created_at\":1574710248.0318432,\"correlation_id\":\"u2Ri7Qp0K52\",\"uber-trace-id\":\"5ee8f57aacd4796:33d77a7051b6a180:394388d47db97b8:1\",\"enqueued_at\":1574710248.0388129}"}
2019-11-25_19:31:03.60110 rails-background-jobs : 2019-11-25T19:31:03.601Z 84652 TID-ovz5hpums WARN: ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server
2019-11-25_19:31:03.60112 rails-background-jobs : : UPDATE "merge_requests" SET "rebase_jid" = $1 WHERE "merge_requests"."id" = $2
2019-11-25_19:31:03.60113 rails-background-jobs : 2019-11-25T19:31:03.601Z 84652 TID-ovz5hpums WARN: /Users/osw/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/postgresql_adapter.rb:611:in `async_exec_params'
Instead, now we rescue from possible Gitaly errors, clean the rebase_commit_sha
column then raise the error back.
More information can be found at #33943 (closed).
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Edited by Oswaldo Ferreira