Integrate Google Lighthouse into our CI to evaluate frontend code performance

Here's a link to the Chrome extension that you can try out now: https://developers.google.com/web/tools/lighthouse/

@mikegreiling how much effort would it be to simply set this up for a handful of URLs, I'd nominate the same ones we currently track in our Prometheus-fed blackbox monitoring.

Hmmm... not sure this is the right tool for us. I just ran it on gitlab.com/gitlab-org/gitlab-ce/issues/1, and because it chooses to work with a 3G network speed, it took 17 seconds. Full report attached. It does point out that we should specify a network speed for our user-side performance testing.Lighthouse_Report.pdf

@mikegreiling so as you read that report, how does it teach us where we can get bang for the buck of invested time in speeding things up?

Here's how I read this report:

You can safely ignore most of the suggestions in the "progressive web app" section. These are targets for websites intended to run as installable apps on android devices, and it's a buzz word google has been pushing recently – twitter.com for instance is a PWA. Most of these metrics are based around assumptions about mobile-first websites with offline-modes and service workers... it doesn't pertain to us.
The "performance" section is very helpful. It displays our "time to interactive" and "first meaningful paint" metrics. As you said this is on a simulated 3G connection, but we could select a different simulated network for our purposes.

These numbers importantly allow us to quantify the improvements we make to our frontend codebase, and we could even write rules around them in our pipelines. For instance we could make a CI pipeline fail if the time to "first meaningful paint" increases past some threshold that we set for ourselves. The first meaningful paint corresponds to the time frame of section 1 and 2 of the graph in #33391 (closed) and the time to interactive is roughly the time encompassing steps 1-4. Lighthouse also allows us to write our own rules and capture our own custom metrics.

Further down in this section, it also reports some changes that can be made to the assets we serve in order to speed up page performance. For instance it cites image sizes and compression techniques here, recommendations to limit render-blocking scripts, and it notes uncompressed assets. Almost all of these suggestions are things noted in #33391 (closed) and #33501 (closed), but even after we address them it is worth having an automated tool like this check them to prevent us from unintentionally regressing in the future.
The "accessibility" section is also valuable as we target improvements to be made for accessibility in gitlab... this is pretty self-explanatory.
Lastly the "best practices" section provides some useful suggestions for security or browser compatibility. For instance I believe we have an issue open regarding the rel="noopener" suggestion.

also, although targeting performance on 3G networks or slow mobile CPUs may not be our primary use case, it is still something worth considering.

for instance #30674 (closed) is an issue opened by a community member comparing our poor mobile performance to that of github. anecdotally, I tend to use the site on mobile devices quite a bit as well

mentioned in issue brycepj/gitlab-frontend-monitor#2 (closed)

Just FYI. @brycepj has noted some overlap between what we want for performance monitoring and what lighthouse offers. brycepj/gitlab-frontend-monitor#2 (closed)

IMO, instead of continuing to build our own thing, integrating lighthouse into CI could be a much nicer solution. As I said, @brycepj had a few really good points on this...

The Chrome team is behind it and there's growing enthusiasm for tooling around it (e.g. big-rig)

Lighthouse's metrics are UX-centric metrics, whereas performance.timings represent more neutral events as they occur during page load. tldr: less work for us to get meaningful, actionable insight

Now that we can see lighthouse log warnings in development, it would be cool to have some parity between what we're monitoring and what we're seeing in the console day-to-day.

We're already going to need to use the chrome debugging protocol to gather feature-specific timings, and lighthouse is built on top of it. Seems smart to be using common APIs across aspects of our monitoring

Taking this approach turns this from something that we could work on for months and would maintain indefinitely to something that can be scheduled for a single milestone and maintains itself (yet still allows us to contribute).

Hmmm.. Doesn't CI feel like a bit of a mismatch for this?

For example, how will we determine which pages to run lighthouse on for a given revision? If we run it for a static set of pages (say, our 10 most used), that's a lot of wasted testing for changes that don't actually affect performance on those pages.

Also, I'm not sure it's a good idea to use it to actually enforce performance with CI either, since there will be pretty wide variance on timings returned in build environments. Can you imagine the transient failures?

Another significant issue is build performance: in the screenshot in this issue's description, you'll see that a couple of fields returned Audit Errors. This happened when I was scripting with Lighthouse locally too. It happens because the tool requires several seconds to collect those metrics. I had to manually tack on a 5 second timeout (in addition to a 10-15 second load time) to give those metrics time to run. Doing this on a large number of pages would be a huge drain on build performance.

In my mind, remote scraping and monitoring (with Prometheus) is still a better fit for pageload type insights that Lighthouse gives than CI. We'd regularly scrape pages we really care about performance for, and export the raw results lighthouse returns into buckets that Prometheus can monitor and alert us in when something goes wrong.

I could see CI being one way of bringing performance closer to our day-to-day work. But now that we can see Lighthouse warnings in our browser consoles, I think encouraging devs to open issues when they see significant warnings would ensure they're getting insights about code they're actually working on.

For example, last week I was working on the MR page, and noticed my log was filling up with warnings.

That's from initTimeagoTimeout being run every second. I've already been pretty skeptical of how we run that code, but now I have a visual representation of the performance impact.

I'll also have the console warnings when I start to fix it, to track whether my changes are improving the problem or not.

All in all, I'm having a hard time seeing the real, actionable value we'd get from adding this to CI that we couldn't achieve with other approaches.

mentioned in issue #33958 (moved)

Doesn't CI feel like a bit of a mismatch for this?

@brycepj Ahh yes, I feel we also had this conversation about gitlab-frontend-monitor. I agree.

In my mind, remote scraping and monitoring (with Prometheus) is still a better fit for pageload type insights that Lighthouse gives than CI. We'd regularly scrape pages we really care about performance for, and export the raw results lighthouse returns into buckets that Prometheus can monitor and alert us in when something goes wrong.

I agree, having something like Lighthouse run as an external service to ping GitLab.com for performance scores is something we should definitely do, but I don't think both approaches are mutually exclusive. If this were reliable and deterministic, and if this were able to capture useful metrics that would catch regressions in performance best practices or accessibility, I think you'd agree that doing this in CI would have some real value.

The Lighthouse devs have specifically talked about how to run it in CI so it seems that at least some people have found value in it. I think it is at least worth exploring. If the CI job takes excessive time or resources, or is so non-deterministic that it produces wildly varying performance scores that are of no value, then I agree it isn't worth pursuing but we won't know until we try.

For example, how will we determine which pages to run lighthouse on for a given revision? If we run it for a static set of pages (say, our 10 most used), that's a lot of wasted testing for changes that don't actually affect performance on those pages.

Well that's also a general problem with our pipeline config at the moment... why do we run unit tests on our ruby code or test database migrations when we've only made changes to javascript or css? I think we ought to work on something like #26619 (moved) to address that particular concern.

We could also make this a manual build step similar to how we user the "build" CI step currently. That way it would only be triggered on-demand.

Also, I'm not sure it's a good idea to use it to actually enforce performance with CI either, since there will be pretty wide variance on timings returned in build environments. Can you imagine the transient failures?

If the variance is too high to be useful, then I agree there's not any point in doing this. I think there may be ways to mitigate this though. If we can get all of our runners to have the same hardware specs and available resources, it might be fairly reliable.

Another significant issue is build performance: in the screenshot in this issue's description, you'll see that a couple of fields returned Audit Errors. This happened when I was scripting with Lighthouse locally too. It happens because the tool requires several seconds to collect those metrics. I had to manually tack on a 5 second timeout (in addition to a 10-15 second load time) to give those metrics time to run. Doing this on a large number of pages would be a huge drain on build performance.

Some of this is due to the fact that we currently score very low on their performance metrics because our JS bundles are so gigantic and it simulates a 3G data connection while performing these audits. Hopefully when we get our code in a better state this won't be so bad.

@timzallmann and I also discussed the possibility of automating this locally. We could set up a Dockerfile and some scripts similar to our GitLab QA process and allow devs to manually generate lighthouse reports on their local machines in an automated way. That would take care of some of the unpredictability of our CI machines, but still cut down on the complexity of manually running lighthouse locally.

@mikegreiling We're in agreement

Here's an approach that runs lighthouse against a staging server to generate reports:

https://github.com/ebidel/lighthouse-ci

Also we should remember that, if we had the tooling to support it, we can use lighthouse in CI where we want it to fail for some defined rules and can let it pass for anything perf related. For example, there is plenty basic web spec and accessibility enforcement.

In gitlab-com/infrastructure#2106, it is discussed that we should focus on sitespeed.io before working with Lighthouse. My assumption is that the benefits you're expecting to see from using Lighthouse in CI can be had from sitespeed.io as well... true or false?

@ernstvn Correct, sitespeed will actually better suit our performance needs I think.

@tauriedavis I'm not sure if this needs to be in the a11y epic. It's about making it easier to test a11y stuff automatically, but it's not itself an a11y problem.

mentioned in issue #46516 (moved)

I'll add it in order to increase visibility, thanks @wispfox! As we break out the epic into smaller epics, we may find we should have one dedicated towards testing a11y

added to epic &31

please add customer interest here (me)

added rebuild in GitLab label

This is a really nice feature that can be added especially because a lot of open source project's websites have terrible performance problems, poor ui etc and this can really help keep the standards up!

added 1 deleted label

the basic thing about this is again generic html reports per build pipeline ... like jenkins has it since years as publisher plugin. pages is a nice step, but i would prefer to have multiple pages ...

https://github.com/GoogleChrome/lighthouse/issues/7149

So, I found this article: https://blog.toukopeltomaa.com/run-simple-scheduled-lighthouse-tests-on-git-lab-ci Unfortunately, the Docker image used for GitLab runner wasn't working in my case, so I prepared my own: https://cloud.docker.com/u/massimofilippi/repository/docker/massimofilippi/gitlab-runner

It works, the report is saved as artefacts archive. Please, any clue how to add comments directly to pipeline job output? Coz now you have to download artefact unarchive, unzip, open in browser... it would be really nice to see the output directly on pipelines page, something like here (Github + Travis): https://github.com/GoogleChromeLabs/lighthousebot

Any clue how to do this? Cannot find how :( Thanx!

Here is the testing config:

image: massimofilippi/gitlab-runner
#image: markhobson/node-chrome

lighthouse:
    stage: deploy
    environment: Production
    only:
        - master
    script:
        # Run Lighthouse test
        - lighthouse --chrome-flags="--headless --no-sandbox" https://www.google.com/ --output html --output-path ./report.html
    artifacts:
        paths:
            - ./report.html
        expire_in: 1 week

you can store the report as gitlab page
you can use gitlab api to add infos in the mr

added DogfoodingRebuild in GitLab devopsmonitor labels and removed rebuild in GitLab label

removed 1 deleted label

added [deprecated] Accepting merge requests label

removed [deprecated] Accepting merge requests label

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue

Integrate Google Lighthouse into our CI to evaluate frontend code performance

Designs

Child items ...

Activity

Frequently Asked Questions