Feature: declarative pipelines

[FrodeHus]

I've started on a feature that others might find useful in Brigade. It's very basic right now, but it works for simple stuff.

Use case

Defining components that can be re-used and composed into pipelines - all in YAML.
The pipelines themselves can be composed of other pipelines in addition to components.
This allows for the creation of a library of Brigade components and pipelines ready for use without the need to write brigade scripts.

For example, someone can create a security scan pipeline and you can then refer to that pipeline as a step in your own pipeline without having to create and maintain it yourself.

These declarative pipelines should also make it easy to create nice visualizations of the build process.
The end result of the pipelines is the generation of a brigade.js that is attached to the worker just as default script is today.

Sample pipeline
Feature fork

Would this be interesting as a PR?

[technosophos]

See also #666, which is a request to do GitHub HCL pipelines. We're talking about whether we can do this in "user space" (like pre-converting from one to another). But we're open to ideas at this point.

[hharnisc]

Some things that would be great to see:

Keeping the pipeline configuration in code

• Allows for code reviews
• Configuration is state you pass to something that runs the pipeline

Reference secrets in pipeline configuration

• Allows public projects to have pipeline configuration without exposing sensitive information
• Could use Brigade Secrets to plug into a configuration template

A nice to have might be some sort of compatibility with GithubActions -- it sounds like there's a push to build a community around actions.

[radu-matei]

Hi, @FrodeHus!
First of all, thanks for opening the issue! Your fork looks really interesting, and we're looking at adding multiple declarative ways for defining pipelines - and while we haven't yet discussed any implementation details, we think we want to support HCL and YAML.

Would you care to open a Work in Progress PR with your work so we can review it in detail and start discussing the specific implementation?

cc @adamreese

[FrodeHus]

Of course!
I havent had time to focus much on this lately, but if its something that is interesting for Brigade I will make the time :)

Pull request #777 ready

[krancour]

@FrodeHus you might want to look at some of the work going on in https://github.com/lovethedrake/brigdrake.

[krancour]

Closing due to similarity with #1024.