Skip to content

Add step schema separate from step proto

Joe Burnett requested to merge jburnett/schema into main

This change adds a data model in which users will write their steps. This model will be used in jobs under the run keyword and inside step.yml where steps are defined. It is compiled (one-way) to the current proto-based model on which step-runner continues to operate. The model source is a package of Go types in /schema/step.go. A JSON schema is generated from the struct tags and Go comments, stored in /schema/spec.json and /schema/definition.json. The schema is an artifact consumable by the UI (and other tooling) for validating the structure of steps YAML and JSON documents.

The /pkg/step/ package has been divided into reading and compiling functions. There are read methods for both YAML step schema files and step proto files. And schema objects can be created directly in Go using the types defined in step.go. Step schema types are compiled into step proto types with the Compile function. Compilation has been arranged by attaching a sequence of mutations to each schema data type. This allows new, convenient mutations to be added without overwhelming the package with lots of named helper functions. I tried a few approaches ranging from lots of helper functions to a single, flat list of mutations using Go generics. This arrangement seemed the most readable and therefore maintainable.

The stated objective of this change is to add a single script keyword to the step invocation syntax, a shorthand for invoking a script step with a string input. This forced the creation of a second data model which paves the way for more syntactic sugar. I also added a top-level script keyword as a proof-of-concept for how we can easily make the schema model easier for users.

E.g. this is a valid step definition:

{}
---
script: echo hello world

Which compiles to this proto step:

spec:
    inputs: {}
---
type: steps
steps:
    - name: "run a script"
      step: "https://gitlab.com/josephburnett/script@v1" # until we create the canonical step repository
      inputs:
          script: echo hello world

Next steps will be to create a canonical steps repository where we can provide GitLab authored and maintained steps. And then write the script step itself.

Fixes gitlab#440470 (closed)

Merge request reports