RepositoryImportWorker holds transactions open for extended periods exceeding 11m

RepositoryImportWorker can hold database transactions open for extended periods.

Alert: https://gitlab.slack.com/archives/C101F3796/p1619100446100900

image

https://dashboards.gitlab.net/d/alerts-long_running_transactions/alerts-long-running-transactions?orgId=1&from=now-6h%2Fm&to=now-1m%2Fm&var-environment=gprd

This interferes with our ability to scale the throughput of transactions on database.

If possible, this worker needs to break these long running transactions down into smaller transactions.

Example

We see a transaction being held open between 13h56 and 14h07

image

https://thanos-query.ops.gitlab.net/graph?g0.range_input=2h&g0.end_input=2021-04-22%2014%3A30&g0.max_source_resolution=0s&g0.expr=topk(10%2C%0A%20%20max%20by%20(application%2C%20endpoint)%20(%0A%20%20%20%20max_over_time(%0A%20%20%20%20%20%20pg_stat_activity_marginalia_sampler_max_tx_age_in_seconds%7B%0A%20%20%20%20%20%20%20%20type%3D%22patroni%22%2C%0A%20%20%20%20%20%20%20%20environment%3D%22gprd%22%2C%0A%20%20%20%20%20%20%20%20command!%3D%22autovacuum%22%2C%0A%20%20%20%20%20%20%20%20command!~%22%5BaA%5D%5BnN%5D%5BaA%5D%5BlL%5D%5ByY%5D%5BzZ%5D%5BeE%5D%22%2C%0A%20%20%20%20%20%20%7D%5B1m%5D)%0A%20%20%20%20and%20on(instance%2C%20job)%0A%20%20%20%20pg_replication_is_replica%20%3D%3D%200%0A%20%20)%0A)%0A&g0.tab=0

Edited Apr 22, 2021 by Andrew Newdigate
Assignee Loading
Time tracking Loading