Make builds on www-gitlab-com faster
Build time breakdown
- Docker image is downloaded:
1.15GB
normally about 2min - Clone into the repository: 5s
- Download cache: 10s
- Bundle install: instant if no gems updated and cache used.
-
bundle exec rake build pdfs
-
Create
/direction
and/release-list
Direction page build time: 7.115994490999981 Release list page build time: 5.233266092000008
-
Rest of the website and blog is built: approx. 2min
-
PDFs generated: approx. 2.5sec each
Generating public/terms/print/githost_terms.pdf public/terms/print/githost_terms.pdf build time: 4.311765053000045 Generating public/terms/print/gitlab_com_terms.pdf public/terms/print/gitlab_com_terms.pdf build time: 2.461660427999959 Generating public/terms/print/gitlab_consultancy_terms.pdf public/terms/print/gitlab_consultancy_terms.pdf build time: 2.0289885680000452 Generating public/terms/print/gitlab_subscription_terms.pdf public/terms/print/gitlab_subscription_terms.pdf build time: 2.5152828930000624 Generating public/terms/print/gitlab_subscription_terms_sig.pdf public/terms/print/gitlab_subscription_terms_sig.pdf build time: 2.542115881999962 Generating public/high-availability/gitlab-ha.pdf public/high-availability/gitlab-ha.pdf build time: 2.3923460889999433 Generating public/features/gitlab-features.pdf public/features/gitlab-features.pdf build time: 2.159292478999987 Generating public/pdfs/the-eleven-rules-of-gitlab-flow.pdf public/pdfs/the-eleven-rules-of-gitlab-flow.pdf build time: 2.2655011069999773 ```
-
- Upload artifacts and rsync for deploy: approx 1m20s
Total time until deploy: approx. 7-8min
Solutions
There are a couple of ways to make builds on this repo run faster:
-
DONE Use smaller images. We currently use
registry.gitlab.com/gitlab-org/gitlab-build-images:www-gitlab-com
from https://gitlab.com/gitlab-org/gitlab-build-images/container_registry which is 1.15GB. Issue in gitlab-org/gitlab-build-images#4 (closed). - Find another way to generate the PDFs. We need LaTeX and pandoc to convert some pages to PDFs, and these two packages add almost ~400MB of additional space in our Docker image. One thought @ayufan had was to convert pdf_template.tex into HTML and then use phantomjs to convert the needed pages to PDFs. There are many tools that do the job: node-html-pdf, phantom-html-to-pdf, etc.
- ~~Do not generate the direction and release pages on every push. Right now we generate the direction page and the release-list with every build. This is done by 2 scripts, generators/direction.rb which talks to GitLab's API and searches for issues attached to milestones (~12s), and generators/release_list.rb which scrapes our blog for releases starting from 7.8 up to the latest (~32s). One idea would be to decouple this from the normal build and have it triggered once a day with cron using a build trigger with a specific variable.~~ It's taking only 10s in total during build time. Locally it took me 44s, so it has to do with the network I guess.
-
DONE Update to Middleman 4.1 There are performance issues according to https://forum.middlemanapp.com/t/performance-middleman-4-vs-middleman-3. MR !2781 (merged). Along with this, we updated to Ruby 2.3 and we now use
therubyracer
to compile the assets.