Skip to content

Split Secure common library into Go modules

Summary

The common library is a collection of Go packages used by all SAST, Dependency Scanning, and Container Scanning analyzers. Currently this library is a single Go module and all the Go packages share the same versions and changelog,. This is not flexible enough, and is expensive to maintain:

  • When upgrading the common dependency in the analyzer projects, it's not clear why we are updating, and what the benefit is.
  • We cannot introduce a breaking change in any of the Go packages common is made of without upgrading the major number of common itself, which is expensive (extra maintenance in the analyzer projects using the common library). Actually, the analyzer projects might very well upgrade common to the next major version w/o any API change on their side, which is confusing.
  • We cannot remove any of these Go packages without upgrading the major version of common, even if this Go package isn't used by any analyzer - that's the case for the orchestrator package.

The solution is to split common into multiple Go modules. Each Go module has its own versions and its own changelog. A module can introduce a breaking change or can be removed without any side-effect in the analyzer projects that don't use it.

Technically each Go module has its own git tags. See documentation on Go modules.

Improvements

Better communication, reduced maintenance, increased flexibility.

Risks

Broken build in the analyzer projects.

Involved components

Optional: Intended side effects

Key Takeaways

  1. There are 3 types of modules we are dealing with:
    1. framework cli modules (command.go)
    2. group-specific library modules (ruleset.go)
    3. shared library modules (issue.go, logutil.go)
  2. Any modules we plan to split out should have a single owner.
  3. Group-specific modules should belong to their own project. The most immediate candidate for this is ruleset.go package, which is owned by SAST.
  4. Framework cli modules should be split out and provided by each group individually. The clearest example of this is command.go, used by DS and SAST. This is likely to occur anyway as DS deprecates its use, but removal from common removes dead-code.
  5. Shared library modules will be part of an ongoing effort to reduce conflicts in ownership. There are 2 groups, based primarily around number of depending groups
    1. Utilities; i.e. logutil.go cacert.go, pathfilter.go. Most of these are quite stable and do not require updates. If they do, they should be considered for splitting out. An owner should be determined for these and either split the modules into separate projects or simply use CODEOWNERS until changes are needed.
    2. Interfaces; i.e. issue.go. Issue module is the highest touch but the most difficult to split. The main benefit for splitting our common is forced version upgrades since common is inextricably tied to this module. The best direction here is to refactor the module to reflect our schemas more closely using embedded structs and a "base issue" (struct SastIssue { Issue }). We can then use CODEOWNERS to segment ownership of the individual structs.
  6. There's no immediate need for git submodules, but they can be a useful addition if we need to relink any dependencies in the future

Actions

  1. Clean up easy-wins to reduce surface area
    1. Split common/template into separate project
    2. Remove common/table (unused)
  2. Split group-owned modules into separate project
    1. ruleset.go (owned by SAST)
  3. Refactor issue.go into Category-specific file-separated embedded structs with relevant CODEOWNERS. This should mirror the security-report-schemas composition model. This could be done as a separate project to help with versioning - thus requiring opt-in. common/issue.go is put in maintenance-mode, future updates will require teams to use the new structs.
  4. [Consider] Placing common in maintenance mode for stable packages with future updates encouraging separation of modules.
Edited by Lucas Charles