Find a good value for maximum timeout for Shared Runners
With GitLab 10.7, we're releasing the Per runner maximum job timeout feature (gitlab-org/gitlab-ce!17221). We should do some calculations and specify a good value that we could set for:
- gitlab-shared-runners-manager-X - these runners should be used only for GitLab CE/EE jobs (from main projects and forks), so we should find out the longest time for a job and add few minutes for a safety margin;
- shared-runners-manager-X - these are a general purpose runners available for all users. We should check what are the times of jobs currently running on Shared Runners, what timeouts are set in projects that are using the Shared Runners. Ideally would be to get a histogram of such timings. Having this we could decide what value of maximum timeout is a reasonable compromise.
Activity
I've prepared a code snippet to get data from production DB:
# Based on: # https://stackoverflow.com/questions/11784843/calculate-95th-percentile-in-ruby#answer-11785414 # def percentile(values, percentile) return nil if values.empty? return values[0] if values.size < 2 values_sorted = values.sort k = (percentile * (values_sorted.size - 1) + 1).floor - 1 f = (percentile * (values_sorted.size - 1) + 1).modulo(1) values_sorted[k] + (f * (values_sorted[k+1] - values_sorted[k])) end def avg(values) values.inject(0.0) { |sum, el| sum + el } / values.size end def summary(title, dry_run) raise "No block given" unless block_given? data = yield puts puts "### #{title}" puts if dry_run puts " SQL: #{data.to_sql}" puts return end durations = data.map(&:duration) if durations.empty? puts " No data returned" puts return end puts " count: #{durations.size}" puts " avg: #{avg(durations)}" puts " min: #{durations.min}" puts " max: #{durations.max}" puts " mean: #{percentile(durations, 0.5)}" puts " 95th: #{percentile(durations, 0.95)}" puts end def projects_summary(dry_run) projects_stats = Project.all.select('avg(build_timeout) as avg', 'max(build_timeout) as max', 'min(build_timeout) as min') puts puts "### Projects stats" puts if dry_run puts " SQL: #{projects_stats.to_sql}" puts return end if projects_stats.empty? puts " No data returned" puts return end puts " avg: #{projects_stats[0]['avg']}" puts " min: #{projects_stats[0]['min']}" puts " max: #{projects_stats[0]['max']}" puts end DRY_RUN = true GITLAB_CE_EE_TIMEFRAME = 1.week GENERAL_TIMEFRAME = 2.days GITLAB_CE_EE_PROJECTS = [13083, 278964] USED_SHARED_RUNNERS = [40786, 40788, 44028, 44949] def ce_ee_jobs(status) projects = Project.where(id: GITLAB_CE_EE_PROJECTS) Ci::Build. where(project: projects). where('finished_at > ?', GITLAB_CE_EE_TIMEFRAME.ago.midnight). where('status = ?', status) end def other_jobs_on_shared_runners Ci::Build.joins(:runner). where(ci_runners: { is_shared: true, id: USED_SHARED_RUNNERS }). where('project_id NOT IN (?)', GITLAB_CE_EE_PROJECTS). where('finished_at > ?', GENERAL_TIMEFRAME.ago.midnight) end summary("GitLab CE/EE succeeded jobs", DRY_RUN) { ce_ee_jobs(:success) } summary("GitLab CE/EE failed jobs", DRY_RUN) { ce_ee_jobs(:failed) } summary("GitLab CE/EE cancelled jobs", DRY_RUN) { ce_ee_jobs(:cancelled) } summary("Non GitLab CE/EE jobs on Shared Runners", DRY_RUN) { other_jobs_on_shared_runners } projects_summary(DRY_RUN)I've executed this in dry run mode on production DB to see what SQL queries will be generated:
### GitLab CE/EE succeeded jobs SQL: SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."project_id" IN (SELECT "projects"."id" FROM "projects" WHERE "projects"."id" IN (13083, 278964)) AND (finished_at > '2018-04-12 00:00:00.000000') AND (status = 'success') ### GitLab CE/EE failed jobs SQL: SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."project_id" IN (SELECT "projects"."id" FROM "projects" WHERE "projects"."id" IN (13083, 278964)) AND (finished_at > '2018-04-12 00:00:00.000000') AND (status = 'failed') ### GitLab CE/EE cancelled jobs SQL: SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."project_id" IN (SELECT "projects"."id" FROM "projects" WHERE "projects"."id" IN (13083, 278964)) AND (finished_at > '2018-04-12 00:00:00.000000') AND (status = 'cancelled') ### Non GitLab CE/EE jobs on Shared Runners SQL: SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "ci_runners" ON "ci_runners"."id" = "ci_builds"."runner_id" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_runners"."is_shared" = 't' AND "ci_runners"."id" IN (40786, 40788, 44028, 44949) AND (project_id NOT IN (13083,278964)) AND (finished_at > '2018-04-17 00:00:00.000000') ### Projects stats SQL: SELECT avg(build_timeout) as avg, max(build_timeout) as max, min(build_timeout) as min FROM "projects"I wanted to check how heavy these queries could be, but
EXPLAIN ANALYZEis failing with/app/lib/chatops/database/read_only_connection.rb:43:in 'exec': ERROR: canceling statement due to statement timeout (PG::QueryCanceled), when used with our chatops integration in Slack.Edited by Tomasz MaczukinI've asked our DB team for some suggestions, and this is what I've got:
- Using plain SQL should be better and faster than through Rails console.
- Moving calculations to PostgreSQL should also make the query more performant (and the
ntile()window function was proposed). - The last query, about Project's
build_timeshould be limited, since currently it will be targeting ~1M of records.
Query for GitLab CE/EE jobs:
SELECT status, quantile, MIN(duration), MAX(duration), AVG(duration) FROM ( SELECT status, duration, ntile(10) OVER (ORDER BY duration) AS quantile FROM ( SELECT status, (finished_at - started_at) AS duration FROM ci_builds WHERE status IN ('success', 'failed', 'canceled') AND type IN ('Ci::Build') AND runner_id IN (37397, 37398, 157328, 157329) AND project_id IN (13083, 278964) AND finished_at >= '2018-04-12 00:00' ) AS source ) AS stats GROUP BY stats.status, stats.quantile ORDER BY stats.status, stats.quantile;Query for non GitLab CE/EE jobs:
SELECT status, quantile, MIN(duration), MAX(duration), AVG(duration) FROM ( SELECT status, duration, ntile(10) OVER (ORDER BY duration) AS quantile FROM ( SELECT status, (finished_at - started_at) AS duration FROM ci_builds WHERE status IN ('success', 'failed', 'canceled') AND type IN ('Ci::Build') AND runner_id IN (40786, 40788, 44028, 44949) AND project_id NOT IN (13083, 278964) AND finished_at >= '2018-04-12 00:00' ) AS source ) AS stats GROUP BY stats.status, stats.quantile ORDER BY stats.status, stats.quantile;Query for projects:
SELECT quantile, MIN(build_timeout), MAX(build_timeout), AVG(build_timeout) FROM ( SELECT build_timeout::integer, ntile(10) OVER (ORDER BY build_timeout::integer) AS quantile FROM ( SELECT build_timeout FROM projects WHERE shared_runners_enabled = 't' AND last_activity_at >= '2018-04-01 00:00' ) AS source ) AS stats GROUP BY stats.quantile ORDER BY stats.quantile;cc @ayufan
The last one still does a sequential scan across all projects:
QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------- Sort (cost=318832.66..318833.16 rows=200 width=44) Sort Key: (ntile(10) OVER (?)) -> HashAggregate (cost=318822.52..318825.02 rows=200 width=44) Group Key: ntile(10) OVER (?) -> WindowAgg (cost=301028.92..309332.60 rows=474496 width=8) -> Sort (cost=301028.92..302215.16 rows=474496 width=4) Sort Key: projects.build_timeout -> Seq Scan on projects (cost=0.00..256293.35 rows=474496 width=4) Filter: (shared_runners_enabled AND (last_activity_at >= '2018-04-01 00:00:00+00'::timestamp with time zone)) (9 rows)If I disable seq scans Postgres thinks it'll be slightly slower:
QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------- Sort (cost=347219.31..347219.81 rows=200 width=44) Sort Key: (ntile(10) OVER (?)) -> HashAggregate (cost=347209.17..347211.67 rows=200 width=44) Group Key: ntile(10) OVER (?) -> WindowAgg (cost=329444.11..337734.47 rows=473735 width=8) -> Sort (cost=329444.11..330628.45 rows=473735 width=4) Sort Key: projects.build_timeout -> Bitmap Heap Scan on projects (cost=80714.10..284785.77 rows=473735 width=4) Recheck Cond: (last_activity_at >= '2018-04-01 00:00:00+00'::timestamp with time zone) Filter: shared_runners_enabled -> Bitmap Index Scan on index_projects_on_last_activity_at (cost=0.00..80595.66 rows=481014 width=0) Index Cond: (last_activity_at >= '2018-04-01 00:00:00+00'::timestamp with time zone) (12 rows)How frequently is this query going to be run? If it's once a day it's not really a problem but if it's on every job or every five minutes this isn't the direction we want to go in...
First one:
status | quantile | min | max | avg ----------+----------+-----------------+-----------------+----------------- canceled | 1 | 00:00:00.259499 | 00:01:42.327692 | 00:00:19.63005 canceled | 2 | 00:02:03.61914 | 00:04:21.335233 | 00:03:29.182399 canceled | 3 | 00:04:21.650826 | 00:08:14.203294 | 00:05:38.61694 canceled | 4 | 00:08:16.529618 | 00:10:52.365723 | 00:09:41.405737 canceled | 5 | 00:11:07.963884 | 00:12:41.438367 | 00:11:42.092037 canceled | 6 | 00:13:03.393954 | 00:14:01.46036 | 00:13:17.151306 canceled | 7 | 00:14:05.297523 | 00:15:08.119094 | 00:14:48.523573 canceled | 8 | 00:15:21.515979 | 00:16:08.861099 | 00:15:49.330231 canceled | 9 | 00:16:47.194247 | 00:18:34.951585 | 00:17:05.661985 canceled | 10 | 00:20:00.952057 | 01:32:26.856662 | 00:38:43.185581 failed | 1 | 00:00:06.624557 | 00:01:54.901296 | 00:00:51.555143 failed | 2 | 00:01:54.915941 | 00:04:21.633016 | 00:02:52.175026 failed | 3 | 00:04:21.756064 | 00:08:15.014438 | 00:06:14.26231 failed | 4 | 00:08:16.534186 | 00:10:57.096118 | 00:09:39.247175 failed | 5 | 00:10:57.336277 | 00:12:44.258396 | 00:11:51.376575 failed | 6 | 00:12:44.752982 | 00:14:02.545107 | 00:13:23.051694 failed | 7 | 00:14:02.804232 | 00:15:17.489946 | 00:14:40.966322 failed | 8 | 00:15:17.803545 | 00:16:45.738384 | 00:16:02.053077 failed | 9 | 00:16:45.910374 | 00:19:08.242918 | 00:17:50.666574 failed | 10 | 00:19:08.379777 | 04:28:32.830972 | 00:53:56.205041 success | 1 | 00:00:14.709001 | 00:01:54.870505 | 00:01:05.228565 success | 2 | 00:01:54.903427 | 00:04:21.634607 | 00:02:59.99545 success | 3 | 00:04:21.644138 | 00:08:15.375582 | 00:06:15.013375 success | 4 | 00:08:15.386642 | 00:10:57.324141 | 00:09:43.880651 success | 5 | 00:10:57.33254 | 00:12:44.588685 | 00:11:54.486895 success | 6 | 00:12:44.615901 | 00:14:02.772724 | 00:13:24.455637 success | 7 | 00:14:02.777371 | 00:15:17.702108 | 00:14:39.84815 success | 8 | 00:15:17.708 | 00:16:45.798038 | 00:15:59.59518 success | 9 | 00:16:45.800803 | 00:19:08.351051 | 00:17:46.42274 success | 10 | 00:19:08.387096 | 01:35:37.139052 | 00:23:00.255037 (30 rows)How frequently is this query going to be run? If it's once a day it's not really a problem but if it's on every job or every five minutes this isn't the direction we want to go in...
@_stark It's one-time query. We need to get data to decide what is the best value for a timeout that we will set on Shared Runners :)
-
The first two have a similar problem.
QUERY PLAN --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- GroupAggregate (cost=732181.52..732307.04 rows=772 width=59) Group Key: stats.status, stats.quantile -> Sort (cost=732181.52..732200.83 rows=7725 width=27) Sort Key: stats.status, stats.quantile -> Subquery Scan on stats (cost=731450.92..731682.67 rows=7725 width=27) -> WindowAgg (cost=731450.92..731605.42 rows=7725 width=27) -> Sort (cost=731450.92..731470.23 rows=7725 width=23) Sort Key: ((ci_builds.finished_at - ci_builds.started_at)) -> Bitmap Heap Scan on ci_builds (cost=182716.13..730952.06 rows=7725 width=23) Recheck Cond: (((status)::text = ANY ('{success,failed,canceled}'::text[])) AND ((type)::text = 'Ci::Build'::text) AND (runner_id = ANY ('{37397,37398,157328,157329}'::integer[])) AND (project_id = ANY ('{13083,278964}'::integer[]))) Filter: (finished_at >= '2018-04-12 00:00:00'::timestamp without time zone) -> BitmapAnd (cost=182716.13..182716.13 rows=397517 width=0) -> Bitmap Index Scan on index_ci_builds_on_status_and_type_and_runner_id (cost=0.00..88063.98 rows=3599615 width=0) Index Cond: (((status)::text = ANY ('{success,failed,canceled}'::text[])) AND ((type)::text = 'Ci::Build'::text) AND (runner_id = ANY ('{37397,37398,157328,157329}'::integer[]))) -> Bitmap Index Scan on index_ci_builds_on_project_id_and_id (cost=0.00..94648.04 rows=7023388 width=0) Index Cond: (project_id = ANY ('{13083,278964}'::integer[]))They're using the index on
project_idas well as the index onstatus,type,runner_idhowever as they're separate indexes it has to do scan them separately and then intersect the results. This is compounded by the fact you're accessing multiple statuses and multiple runner_id so the scan on that index is also not really as effective as it could be.If you could restrict it to a single status, perhaps get separate statuses for each status that might be better. And you might try doing the two projects separately too.
We may have to create an index on
project_id,runner_id where status in (success,failed,canceled) and type = Ci::Build. That would let you do each project indivdiually quite efficiently. -
second:
status | quantile | min | max | avg ----------+----------+-----------------+-------------------------+----------------- canceled | 1 | 00:00:00.104069 | 00:00:45.365008 | 00:00:20.214189 canceled | 2 | 00:00:45.391074 | 00:01:02.518067 | 00:00:53.35322 canceled | 3 | 00:01:02.555798 | 00:01:18.293899 | 00:01:09.77623 canceled | 4 | 00:01:18.37465 | 00:01:35.817575 | 00:01:27.156628 canceled | 5 | 00:01:35.869565 | 00:01:56.944536 | 00:01:46.05738 canceled | 6 | 00:01:56.997785 | 00:02:24.108494 | 00:02:09.355273 canceled | 7 | 00:02:24.170565 | 00:03:04.302432 | 00:02:42.63674 canceled | 8 | 00:03:04.435084 | 00:04:13.656893 | 00:03:36.266939 canceled | 9 | 00:04:14.127394 | 00:06:55.906372 | 00:05:23.473336 canceled | 10 | 00:06:57.205487 | 8 days 21:23:59.218075 | 00:42:42.510219 failed | 1 | 00:00:05.707861 | 00:00:45.379507 | 00:00:32.240905 failed | 2 | 00:00:45.381809 | 00:01:02.527653 | 00:00:54.170289 failed | 3 | 00:01:02.528944 | 00:01:18.294873 | 00:01:10.085978 failed | 4 | 00:01:18.29595 | 00:01:35.838549 | 00:01:26.772339 failed | 5 | 00:01:35.839295 | 00:01:56.955287 | 00:01:46.016552 failed | 6 | 00:01:56.960686 | 00:02:24.134105 | 00:02:09.782676 failed | 7 | 00:02:24.136412 | 00:03:04.360174 | 00:02:42.850915 failed | 8 | 00:03:04.374173 | 00:04:13.815233 | 00:03:34.16081 failed | 9 | 00:04:13.837559 | 00:06:56.936154 | 00:05:19.954595 failed | 10 | 00:06:56.946901 | 22 days 23:01:22.201237 | 00:50:30.303678 success | 1 | 00:00:13.193739 | 00:00:45.381284 | 00:00:33.885618 success | 2 | 00:00:45.381453 | 00:01:02.527802 | 00:00:54.475098 success | 3 | 00:01:02.529694 | 00:01:18.295858 | 00:01:10.318173 success | 4 | 00:01:18.297385 | 00:01:35.838434 | 00:01:26.962822 success | 5 | 00:01:35.83863 | 00:01:56.958071 | 00:01:46.035108 success | 6 | 00:01:56.958474 | 00:02:24.135127 | 00:02:10.001001 success | 7 | 00:02:24.135432 | 00:03:04.357623 | 00:02:42.592974 success | 8 | 00:03:04.36062 | 00:04:13.827971 | 00:03:35.817938 success | 9 | 00:04:13.83141 | 00:06:56.938878 | 00:05:19.834479 success | 10 | 00:06:56.939247 | 09:16:58.070374 | 00:18:16.150249 (30 rows) -
third:
quantile | min | max | avg ----------+------+----------+----------------------- 1 | 0 | 3600 | 3576.6058031275422580 2 | 3600 | 3600 | 3600.0000000000000000 3 | 3600 | 3600 | 3600.0000000000000000 4 | 3600 | 3600 | 3600.0000000000000000 5 | 3600 | 3600 | 3600.0000000000000000 6 | 3600 | 3600 | 3600.0000000000000000 7 | 3600 | 3600 | 3600.0000000000000000 8 | 3600 | 3600 | 3600.0000000000000000 9 | 3600 | 3600 | 3600.0000000000000000 10 | 3600 | 60000000 | 6634.1416013197441866 (10 rows)third was done in less than a second, check if everything is alright with it :)
Oh, the third one is interesting. Looking on the quantiles it seems that almost everyone use the default 3600 timeout.
Also these three rows from second query (non GitLab CE/EE jobs on Shared Runners) are interesting:
canceled | 10 | 00:06:57.205487 | 8 days 21:23:59.218075 | 00:42:42.510219 failed | 10 | 00:06:56.946901 | 22 days 23:01:22.201237 | 00:50:30.303678 success | 10 | 00:06:56.939247 | 09:16:58.070374 | 00:18:16.150249If I haven't messed anything in the queries, it looks like more than at least 90% of jobs on GitLab.com's Shared Runners are executed in less than 6 minutes!
I think I'll add standard deviation fields to these queries to check what are the extremes in each of the quantiles.
Queries for GitLab CE/EE jobs:
SELECT status, percentile, MIN(duration), MAX(duration), AVG(duration), STDDEV_SAMP(EXTRACT(epoch FROM duration))/60 AS std_dev_in_minutes FROM (SELECT status, duration, ntile(100) OVER (ORDER BY duration) AS percentile FROM (SELECT status, (finished_at - started_at) AS duration FROM ci_builds WHERE status = 'success' AND type IN ('Ci::Build') AND runner_id IN (37397, 37398, 157328, 157329) AND project_id IN (13083, 278964) AND finished_at >= '2018-04-01 00:00') AS source) AS stats GROUP BY stats.status, stats.percentile ORDER BY stats.percentile; SELECT status, percentile, MIN(duration), MAX(duration), AVG(duration), STDDEV_SAMP(EXTRACT(epoch FROM duration))/60 AS std_dev_in_minutes FROM (SELECT status, duration, ntile(100) OVER (ORDER BY duration) AS percentile FROM (SELECT status, (finished_at - started_at) AS duration FROM ci_builds WHERE status = 'failed' AND type IN ('Ci::Build') AND runner_id IN (37397, 37398, 157328, 157329) AND project_id IN (13083, 278964) AND finished_at >= '2018-04-01 00:00') AS source) AS stats GROUP BY stats.status, stats.percentile ORDER BY stats.percentile; SELECT status, percentile, MIN(duration), MAX(duration), AVG(duration), STDDEV_SAMP(EXTRACT(epoch FROM duration))/60 AS std_dev_in_minutes FROM (SELECT status, duration, ntile(100) OVER (ORDER BY duration) AS percentile FROM (SELECT status, (finished_at - started_at) AS duration FROM ci_builds WHERE status = 'canceled' AND type IN ('Ci::Build') AND runner_id IN (37397, 37398, 157328, 157329) AND project_id IN (13083, 278964) AND finished_at >= '2018-04-01 00:00') AS source) AS stats GROUP BY stats.status, stats.percentile ORDER BY stats.percentile;Queries for non GitLab CE/EE jobs on Shared Runners:
SELECT status, percentile, MIN(duration), MAX(duration), AVG(duration), STDDEV_SAMP(EXTRACT(epoch FROM duration))/60 AS std_dev_in_minutes FROM (SELECT status, duration, ntile(100) OVER (ORDER BY duration) AS percentile FROM (SELECT status, (finished_at - started_at) AS duration FROM ci_builds WHERE status = 'success' AND type IN ('Ci::Build') AND runner_id IN (40786, 40788, 44028, 44949) AND project_id NOT IN (13083, 278964) AND finished_at >= '2018-04-01 00:00') AS source) AS stats GROUP BY stats.status, stats.percentile ORDER BY stats.percentile; SELECT status, percentile, MIN(duration), MAX(duration), AVG(duration), STDDEV_SAMP(EXTRACT(epoch FROM duration))/60 AS std_dev_in_minutes FROM (SELECT status, duration, ntile(100) OVER (ORDER BY duration) AS percentile FROM (SELECT status, (finished_at - started_at) AS duration FROM ci_builds WHERE status = 'failed' AND type IN ('Ci::Build') AND runner_id IN (40786, 40788, 44028, 44949) AND project_id NOT IN (13083, 278964) AND finished_at >= '2018-04-01 00:00') AS source) AS stats GROUP BY stats.status, stats.percentile ORDER BY stats.percentile; SELECT status, percentile, MIN(duration), MAX(duration), AVG(duration), STDDEV_SAMP(EXTRACT(epoch FROM duration))/60 AS std_dev_in_minutes FROM (SELECT status, duration, ntile(100) OVER (ORDER BY duration) AS percentile FROM (SELECT status, (finished_at - started_at) AS duration FROM ci_builds WHERE status = 'canceled' AND type IN ('Ci::Build') AND runner_id IN (40786, 40788, 44028, 44949) AND project_id NOT IN (13083, 278964) AND finished_at >= '2018-04-01 00:00') AS source) AS stats GROUP BY stats.status, stats.percentile ORDER BY stats.percentile;Query for project build_timeouts:
SELECT percentile, MIN(build_timeout), MAX(build_timeout), AVG(build_timeout), STDDEV_SAMP(build_timeout) AS std_dev_in_seconds FROM (SELECT build_timeout::integer, ntile(100) OVER (ORDER BY build_timeout::integer) AS percentile FROM (SELECT build_timeout FROM projects WHERE shared_runners_enabled = 't' AND last_activity_at >= '2018-04-01 00:00') AS source) AS stats GROUP BY stats.percentile ORDER BY stats.percentile;Edited by Tomasz MaczukinI've needed to update the
Query for project build_timeoutssince most of the projects are using default 3600! I've additionally checked the histogram for projects withbuild_timeout < 3600and executed another query forbuild_timeout > 3600. After this I've executed these three queries once again, but changed date requirement tolast_activity_at >= '2018-01-01 00:00'Results:
- gitlab-ce-ee.results
- non-gitlab-ce-ee-shared.results
- projects-active-this-year.results
- projects.results
Most interested metrics:
-
succeeded (so most probably the longest) GitLab CE/EE jobs (I've counted only gitlab-org/gitlab-ce and gitlab-org/gitlab-ee)
status | percentile | min | max | avg | std_dev_in_minutes ---------+------------+-----------------+-----------------+-----------------+-------------------- success | 100 | 00:28:47.945143 | 01:40:49.397007 | 00:38:45.633882 | 11.6862573662018Which means that 99% of our jobs are succeeding in less than 29 minutes!
-
succeeded (so most probably the longest) non GitLab CE/EE jobs on Shared Runners
status | percentile | min | max | avg | std_dev_in_minutes ---------+------------+-----------------+-----------------+-----------------+--------------------- success | 100 | 00:36:22.062229 | 18:32:35.319479 | 01:03:00.312742 | 37.5597534592895Which means that most of the jobs on Shared Runners are executed under 0.5-2 hours!
-
Projects active this year:
percentile | min | max | avg | std_dev_in_seconds ------------+------+-----------+-----------------------+-------------------- 99 | 3600 | 3600 | 3600.0000000000000000 | 0 100 | 3600 | 180000000 | 44583.406113537118 | 1960224.3296So 99% of projects that are active since 2018-01-01 are using a
<= 1htimeout! Looking on the non-default ones that use timeout bigger than 1h:percentile | min | max | avg | std_dev_in_seconds ------------+----------+-----------+------------------------+--------------------- 50 | 10800 | 10800 | 10800.0000000000000000 | 0 90 | 36000 | 43200 | 38190.000000000000 | 3416.050350916977 95 | 96000 | 180000 | 122000.000000000000 | 29799.32885150 99 | 3000000 | 5999940 | 4699970.000000000000 | 1440800.6111 100 | 48000000 | 180000000 | 77999980.000000000000 | 50199610.1977which means:
- 50% of this 1% of projects (which means 0.5% of all active projects) uses a
<= 3htimeout - 95% of this 1% of projects (which means 0.95% of all active projects) uses a
<= 50htimeout - the last percentile (minimal timeout at level of ~555 days!) are probably abusers.
- 50% of this 1% of projects (which means 0.5% of all active projects) uses a
-
Whoa what an awesome research @tmaczukin!
Let's look again on the data:
percentile | min | max | avg | std_dev_in_seconds ------------+------+-----------+-----------------------+-------------------- 1 | 0 | 3600 | 3448.1637426900584795 | 632.399014407278 100 | 3600 | 180000000 | 44583.406113537118 | 1960224.3296 percentile | min | max | avg | std_dev_in_seconds ------------+----------+-----------+------------------------+--------------------- 1 | 3900 | 4800 | 4542.8571428571428571 | 364.495737776374 2 | 4800 | 5400 | 5314.2857142857142857 | 226.778683805536 8 | 5940 | 6000 | 5991.4285714285714286 | 22.6778683805536336 9 | 6000 | 7200 | 6574.2857142857142857 | 603.844823964379 40 | 7200 | 8400 | 7525.7142857142857143 | 463.280229913355 41 | 9000 | 9600 | 9085.7142857142857143 | 226.778683805536 42 | 9600 | 10800 | 10542.8571428571428571 | 440.519174221898 54 | 10800 | 12000 | 11485.714285714286 | 641.426980589819 55 | 12000 | 14400 | 14022.857142857143 | 896.469264870326 61 | 14400 | 16800 | 15342.857142857143 | 838.365757206926 62 | 17880 | 18000 | 17982.857142857143 | 45.3557367611072673 77 | 18600 | 21600 | 20140.000000000000 | 1469.040503185668 81 | 21600 | 22800 | 21800.000000000000 | 489.897948556636 82 | 24000 | 28800 | 27200.000000000000 | 2478.709341572747 83 | 28800 | 30000 | 29200.000000000000 | 619.677335393187 84 | 30000 | 36000 | 31000.000000000000 | 2449.489742783178 90 | 36000 | 43200 | 38190.000000000000 | 3416.050350916977 91 | 43200 | 57600 | 45600.000000000000 | 5878.775382679627 92 | 59940 | 72000 | 66790.000000000000 | 5973.591884285367 93 | 72000 | 86400 | 83970.000000000000 | 5864.520440752168 96 | 180000 | 360000 | 258000.000000000000 | 80229.67032214 97 | 360000 | 604800 | 532780.000000000000 | 108384.38706751 98 | 1728000 | 3000000 | 2788000.000000000000 | 519291.82547003 99 | 3000000 | 5999940 | 4699970.000000000000 | 1440800.6111 100 | 48000000 | 180000000 | 77999980.000000000000 | 50199610.1977First table counts all projects with
shared_runners_enabled = tand active since 2018-01-01. Something between 99% and 100% of such projects uses a timeout that is lower or equal than 1h.The second table is the histogram that counts the same projects, but filtered with
build_timeout > 3600, so the last 1% of the previous histogram. Between 54% and 55% of such projects uses a timeout that is lower than 3h. This means that something between 99.54-99.55% of all active projects are using a timeout lower than 3h.After a discussion we've decided that 3h timeout for Shared Runners would be a good compromise. It's 3 times more than most of currently active projects are using and still feels to be reasonable.
To apply this we should:
- find which projects will be affected when we'll set the 3h timeout,
- notify owners of such projects, specifying when we'll update the timeout,
- apply the timeout to Shared Runners.
I'll create a separated issue to track this (done: #4070 (closed)).
Edited by Tomasz MaczukinAs for GitLab CE/EE jobs, 99% of them succeeds in less than 30 minutes. The maximum duration of a succeeded job (I've counted jobs since 2018-04-01) was 1h40min.
We should check:
-
why some of the jobs were so much longer than most of them,
-
what causes 95% of failed jobs to run for longer than 1h
failed | 95 | 01:18:42.481679 | 01:40:04.849246 | 01:33:12.416284 | 7.46966188731685 -
decide what is the best timeout. As for now 45min seems to be reasonable, yet have still some additional free space for jobs that are longer than the usual <30min.
@stanhu @rymai Since you've been involved in making the pipelines faster, with our high-cpu machines history and other improvements in
.gitlab-ci.ymlconfiguration, I'd like to hear your voice about such timeout :)-
mentioned in issue #4070 (closed)
@tmaczukin Awesome data. 45 minutes seems like it doesn't give a lot of leeway if on average most jobs finish in 38 minutes. Most of the time it's a bit of a mystery why the job took so long: it could be a slow VM, slow clone on GitLab.com, many failed retries in the specs, etc.
Also, isn't it better to give a job 15 more minutes rather than retry and fail again? Another words, would we be more comfortable setting it at 60 minutes to start?
@tmaczukin Given the data, I guess 45 minutes should be reasonable, but I don't have a strong opinion regarding 45 vs 60 minutes to start with. This is a two-way door decision anyway, so we can just pick one and look back to the data again in a few weeks.
