## Status We are moving to the Bring-Your-Own-Tool direction for Code Quality as discussed in this epic. To learn more about the change, please see the following public resources: - [Deprecation announcement](https://docs.gitlab.com/ee/update/deprecations.html#codeclimate-based-code-quality-scanning-will-be-removed) - [Restructured Code Quality documentation](https://docs.gitlab.com/ee/ci/testing/code_quality.html#scan-code-for-quality-violations) explaining the options for scanning - [List of documented integrations](https://docs.gitlab.com/ee/ci/testing/code_quality.html#integrate-common-tools-with-code-quality) (more MRs welcome!) - [Migration cross-reference](https://docs.gitlab.com/ee/ci/testing/code_quality.html#migrate-from-codeclimate-based-scanning) in documentation - [Category direction page](https://about.gitlab.com/direction/application_security_testing/static-analysis/code-quality/) Team members have access to additional resources, including enablement training, in [the Field Guide for this change, on Highspot](https://gitlab.highspot.com/items/673ce2479dee95f62d5b1f2b?lfrm=srp.0#1). (The epic was structured to support a decision process that has completed. As such we will not be deleting its contents.) <details><summary>Historical contents - Click to expand</summary> ## Problem The current Code Quality scanning system is a serious barrier to adoption of the Code Quality feature category and to related categories like SAST. Based on a large number of specific reports from customers and the wider community, we believe this is primarily due to a variety of problems with how Code Quality scanning works—not because of a lack of interest in code quality assessment. The problems with Code Quality scanning are very well validated; they relate primarily to slow performance, out-of-date scanning plugins, and difficult/insecure operational requirements. Code Quality users may not be fully aware of the existing option to bring their own tools; they may assess that implementing a custom tool would be unreasonably difficult; or they may be blocked on this by another limitation, like the inability to consistently use more than one report per pipeline. (Multiple reports is a blocker for at least two internal dogfooding use cases.) It would be easy to view Code Quality as an all-or-nothing proposition—the [Bring-Your-Own-Tool option](https://docs.gitlab.com/ee/ci/testing/code_quality.html#implementing-a-custom-tool) that exists today is just one section in documentation, while the automatic scanning is included in Auto DevOps and otherwise framed as the preferred option. (We do sometimes hear questions about custom tools, but questions much more often are about issues with the existing scanning system.) ## Scope of this effort The Code Quality feature area can be thought of as a few component parts: 1. A **scanning system** to identify problems in source code. This is currently provided by the GitLab packaging of the [CodeClimate open source project](https://github.com/codeclimate/codeclimate), deployed through a GitLab-managed CI/CD template or [through Auto DevOps](https://docs.gitlab.com/ee/topics/autodevops/stages.html#auto-code-quality). 1. A **reporting system** to ingest those findings and show them in appropriate places. This is currently implemented in the Rails platform. 1. **Workflow features** to control and manage quality issues on an ongoing basis. This is currently *not implemented*. This effort is primarily about removing barriers to adoption in area **\#1 (scanning)**, but will almost certainly affect how **\#2 (reporting)** works. We will **not** tackle **\#3 (workflow)** as part of this effort. ## The status quo Before assessing a new solution, it is worth being clear about where we are today with scanning. - GitLab-managed scanning entirely relies on the [CodeClimate open source project](https://github.com/codeclimate/codeclimate) and its plugins. - The [available plugins](https://docs.codeclimate.com/docs/list-of-engines) wrap open-source scanning tools like ESLint, SonarJava, or PHP Code Sniffer. - Many of these plugins are multiple years out of date. - Some of these plugins overlap with other feature areas in GitLab, like License Compliance, SAST, or Secret Detection. - The set of plugins used is determined by a CodeClimate configuration file (directly defined by CodeClimate upstream) that is committed into each project. - When a pipeline runs Code Quality, the job runs a shell script. - The shell script invokes `docker` to run a "parent" `codeclimate` job. - The parent job runs separate containers for each "child" `plugin` job, then gathers the results together into a single report JSON file. - The report is saved as a [special artifact type](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscodequality) that the Rails platform knows how to process. - You can provide a report in a CodeClimate-defined format instead of using GitLab scanning. - Support for providing more than one artifact per pipeline is currently [incomplete](https://gitlab.com/gitlab-org/gitlab/-/issues/358759). - The report files are processed on-demand when results are requested. - This leads to performance issues that have blocked rollout of support for multiple reports and contribute to failures of various related APIs. ## What we have learned This is a summary of what we've learned from customers, users, and internal stakeholders. Many of the findings were identified or confirmed by Problem Validation UX research (https://gitlab.com/gitlab-org/ux-research/-/issues/1886). - Developers typically adopt a set of quality tools, enable a ruleset (usually the default), and then don't mess with it too much. - All interview subjects used lint-style rules to maintain quality of their codebases. - Some (not all) developers and managers also value complexity metrics. They use them to identify opportunities for refactoring, to better focus human review on "hot spots", or quantify efforts to proactively improve maintainability and quality. - Teams often use quality tools to "hold the line" on quality—they decide on a ruleset and then enforce compliance thereafter. If needed, they may disable a rule or allow an exception, but they often expressed a preference for resolving all errors. - Some organizations manage rules centrally, but many delegate customization or exceptions to individual teams and do not require higher-level approval for quality or linting exceptions. - People's current workflows often center around jobs, and reading job logs. We have an opportunity to substantially advance many users' experience if we make it easier to process these findings. - Motivated users and tool authors are often open to implementing new output formats in order to feed their results into tools. We already have more than one example of tools adding CodeClimate-report-format output to integrate with GitLab Code Quality. This suggests that there is potential for community-driven connections into the feature. ## What we want to achieve _Ambitious vision:_ Code Quality is the single best way in GitLab to tell an MR author and reviewer about a quality or linting problem in the diff they've submitted. This epic proposes to first shape Code Quality into a flexible, open, community-oriented feature area. Adopting a more "pluggable" model should help us: - Meet teams where they are instead of requiring a one-size-fits-all approach. This includes respecting team preferences about the tools they wish to use, including tools that may be organization-specific or may not even be open-source. - Encourage individual users and teams to experiment and take the initiative to move their teams forward. We want users to be able to build their workflows around GitLab Code Quality freely without relying on first getting patches included in GitLab or an upstream project like CodeClimate. - Maximize compatibility with different project organization structures and requirements, including when projects need particular tool versions or must be executed in certain ways. - Enable reporting on a wider variety of languages—and even non-code content like technical documentation—than a single GitLab team could support. This approach is similar to the way that other products take standard coverage file formats. Within GitLab, similar examples include accepting [the JUnit XML format](https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html#how-it-works) for test reporting, and accepting [cyclonedx reports](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscyclonedx) for Software Bill of Materials information. We would *not*, in this proposal, immediately try to independently match the functionality of a broad set of open-source analyzers. Rather, we would first make it simpler for users to integrate the tools they already use without trying to either replace those tools or take over maintenance of them. ## Dogfooding potential Within GitLab, we have a number of possible avenues for dogfooding: - The existing Code Quality feature is already included in gitlab-org/gitlab builds. - eslint, rubocop, and similar tools are used for quality checking, separate from Code Quality. - Technical Writing uses the Vale linter to check documentation, along with other dedicated types of linters that output information to job logs. - Tracking Pajamas migrations (and flagging use of out-of-date components) was identified as an internal use case. - Danger Bot checks various other issues, some of which may be able to roll into Code Quality. We could begin channeling those features' output into the Code Quality processing system, while leaving existing comments and job logs available to reduce risk. However, most or all of these are blocked by support for multiple reports per pipeline. One way to dogfood faster could be targeting smaller projects that would only need one report at a time. The benefits to internal users could be substantial—features like MR diff annotation are quite a bit more convenient than reading job logs. ## How we plan to do it We haven't yet designed system changes to reflect this vision. When we do, a summary and link to the architecture blueprint should be included here. Possible solution "shapes" include these, or a mix of them: - Upstream contributions to popular tools to enable output to the required format. - A small binary (or container image) that you can easily drop into a pipeline to convert results to the required format; this would support a defined set of popular formats. - New `artifacts` format options (see the existing [codequality report type](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscodequality)). - Other ideas? In all cases above we would document the required format in sufficient detail so that anyone could contribute a new report parser or simply implement one for themselves. This way, everyone can contribute to their team's efforts to improve quality. ## Possible iterations We can advance along different axes to define increments that help us learn as we go while also providing value earlier. 1. Programming language coverage. We can begin with the most widely used languages or those used by customers/users that are specifically interested in collaborating as we develop the feature further. - To inform which languages come first, we have data on [GitLab.com programming language usage (team members only)](https://app.periscopedata.com/app/gitlab/635927/Programming-Languages-on-GitLab.com), image pull metrics for CodeClimate plugins on Docker Hub, and (currently unstructured) information on which languages are used by the most interested customers. We also know that Ruby, Go, and JavaScript would be immediately usable within GitLab projects. 1. Availability level. We can begin the feature in one of the [pre-GA availability levels](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) if we feel it necessary. 1. Level of "magic". - Code Quality is currently available in Auto DevOps, though the set of plugins that runs is not determined automatically. - If we have a reliable way of doing the right thing most of the time, we should explore such an option. For instance, if we identify an `eslint` configuration file, we could automatically run `yarn install` or `npm install`, run `eslint`, and process its findings. - But, this does not need to be part of the first set of changes; we could support it in a limited set of cases; or we could provide example templates but clearly note that some customization is expected. 1. Types of problems that can be addressed natively. For instance, we can begin with linting rules, and move on to complexity assessment (possibly through GitLab-managed tools) later. 1. Language version support. We can start with a smaller support matrix (recent language versions only) if needed, and expand when we know we need to. ## Open questions 1. Should we take this opportunity to change the format in which we ingest quality reports? - There are existing open-source integrations that output CodeClimate report JSON as a way to integrate with GitLab. - Any breaking change would have to go through the deprecation and removal process, leading to a removal in 16.0. Hence, we would still have to support the existing format for some time even if we introduced a new format option in the meantime. - But, we have limited usage today, so changing now (before pushing hard to gain additional adoption) could limit the impact of the change. 1. Could/should we handle problems that aren't attributable to a specific line in a specific file? Think, for instance, of commit message guidelines or other things that Danger Bot remarks upon. 1. Is it feasible to offer an Auto DevOps version of Code Quality scanning? 1. Is the reporting system ready to handle an increased amount of load? What data would we need to collect to accurately assess this? What changes to the report processing system would we want to make before pushing to substantially increase usage of Code Quality features? ## References - [Meeting notes](https://docs.google.com/document/d/1QV5QhqRyX8p6c1_ynKYaKSzMv2j9is316vWf2R2EoLo/edit#heading=h.6upuyp25d0wm) ## Index of discussion threads - Scope of Code Quality category - Interaction with code coverage and other areas https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1116727238 - Workflows outside of individual MRs - Reporting on code quality statistics (including at levels above an individual project) https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1116727238 - Drilldown, faceting, navigating to code https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1116698480 - Interactions with/comparisons to Vulnerability Report https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1116698480, https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1116942989 - Tradeoffs related to level of automation/magic https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1116693549 - Differences from Security workflows https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1116634345, - What are we leaning into with this effort? - Emphasis on scanning? Experience? Workflow? https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1111004419 - Are we changing scanning or reporting? https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1111013092 - One DevOps Platform advantage https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1111039520 - Inline comments https://gitlab.com/groups/gitlab-org/-/epics/8790#note_1118300285 </details> <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.* <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->