Use Protocol Buffers to standardize Security Products exchange format

Problem to solve

We don't have a clean way to open and share the format used in our Security Products. There are some issues to document it (at least), but it doesn't make it easy to consume. This issue is to evaluate the introduction of Protocol Buffers in our libraries.

Intended users

• Sasha (Software Developer)

Further details

From the official website: "Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages."

It's a contract between entities (services, libraries, etc.) where the exchange format is documented, versioned, and enforced. It's a great fit for our need and would allow developers to create new analyzers without requiring Go (the only current way). The format is compiled and can be shared easily.

It's much more common than json schemas

Proposal

We already use protocol buffers at GitLab, for example in Gitaly. We currently use a go api to ensure the right fields are provided, but it's:

• restricted to Go usage only (ProtoBuf is language agnostic)
• Hard to version (and we sometimes forget to update the version, creating broken ones)
• Not automatically documented

Permissions and Security

No change in permissions.

Documentation

• We should have a dedicated section for this in https://docs.gitlab.com/ee/user/application_security/ (or at least, a page to explain how to create custom analyzers)

Testing

We can add a sample analyzer that would consume the protocol.

What does success look like, and how can we measure that?

• All analyzers implement the provided protocol
• We have clean format versions
• Users can create analyzers in the language of their choice

What is the type of buyer?

• Ultimate for now, Core when &440 (closed) is done.

Links / references

/cc @gl-secure