README.md 16.7 KB
Newer Older
Laurent B's avatar
Laurent B committed
1
2
PowerStencil
============
Laurent B's avatar
Laurent B committed
3

Laurent B's avatar
Laurent B committed
4
[![Gem Version](https://badge.fury.io/rb/power_stencil.svg)](https://rubygems.org/gems/power_stencil) [![Pipeline status](https://gitlab.com/tools4devops/power_stencil/badges/master/pipeline.svg)](https://gitlab.com/tools4devops/power_stencil/commits/master)
Laurent Briais's avatar
Laurent Briais committed
5

Laurent Briais's avatar
Laurent Briais committed
6
`Power Stencil` is the Swiss-army knife templating workflow for developers and ops.
Laurent B's avatar
Laurent B committed
7

Laurent B's avatar
Laurent B committed
8
9
10
See [official website][PowerStencil site].


Laurent B's avatar
Laurent B committed
11
12
<!-- TOC -->

Laurent Briais's avatar
Laurent Briais committed
13
- [Overview](#overview)
Laurent Briais's avatar
Doc    
Laurent Briais committed
14
- [Pfff, yet another templating engine...?](#pfff-yet-another-templating-engine)
Laurent B's avatar
Laurent B committed
15
16
- [Installation](#installation)
- [Usage](#usage)
Laurent B's avatar
Laurent B committed
17
    - [Help](#help)
Laurent B's avatar
Laurent B committed
18
    - [Getting started](#getting-started)
Laurent Briais's avatar
Laurent Briais committed
19
20
21
        - [Creating a `PowerStencil` project](#creating-a-powerstencil-project)
        - [`PowerStencil` project structure overview](#powerstencil-project-structure-overview)
        - [Command line auto-completion](#command-line-auto-completion)
Laurent B's avatar
Doc    
Laurent B committed
22
23
24
25
        - [Entities](#entities)
        - [Templates](#templates)
        - [Builds](#builds)
        - [Plugins](#plugins)
26
        - [Git integration](#git-integration)
Laurent B's avatar
Laurent B committed
27
- [Project status](#project-status)
Laurent B's avatar
Laurent B committed
28
29
30
31
32
33
- [Contributing](#contributing)
    - [License](#license)
    - [Code of Conduct](#code-of-conduct)

<!-- /TOC -->

Laurent Briais's avatar
Laurent Briais committed
34
# Overview
Laurent B's avatar
Laurent B committed
35

Laurent B's avatar
Laurent B committed
36
`PowerStencil` proposes a radical approach on how to manage your shared configuration.
Laurent B's avatar
Laurent B committed
37

Laurent Briais's avatar
Laurent Briais committed
38
**Configuration** is one of the most complex things to maintain, and anyone who participated in a large scale piece of software, be it for code, tools configuration or even documentation, knows how complex it is to maintain, **keep consistent and avoid duplication** on the long run, across the various projects that may share this configuration.
Laurent Briais's avatar
Laurent Briais committed
39

Laurent Briais's avatar
Laurent Briais committed
40
From a very high level point of view, the workfow `PowerStencil` proposes looks like:
Laurent Briais's avatar
Laurent Briais committed
41

Laurent Briais's avatar
Laurent Briais committed
42
![simple-flow-image]
Laurent B's avatar
Laurent B committed
43
44
45

`PowerStencil` provides **development and operations friendly workflows to fully manage the maintenance of a complex shared config** in order to generate anything you may want like _documentation_, _static sites_, _code_, _configuration_ for multiple tools..., while **avoiding duplication, and manage it like code** !

Laurent Briais's avatar
Doc    
Laurent Briais committed
46
It is a common pattern to introduce a database to manage shared configuration for CI/CD processes, and it is a very bad idea (see [F.A.Q.][DB in F.A.Q.]) because it implies extra tooling, and specific developments that de-facto become single point of failure at the heart of your development process. Why would you do that when something as versatile and ubiquitous as Git exists that will integrate smoothly within your existing development and operations workflows, builds, CI/CD processes, while benefiting from all the goodness provided by Git, like branches, decentralization, versioning, tags, code reviews, traceability, simplified fallbacks etc... ?
Laurent B's avatar
Laurent B committed
47

Laurent Briais's avatar
Laurent Briais committed
48
A `PowerStencil` project is composed of a data repository and a pretty standard templating flow to generate whatever is needed by your project or organization and **with `PowerStencil` everything is maintained in a git repository**: data, templates, specific code **without sacrificing on data integrity and relational features**. No more config loss, no more change applied without a full lineage.
Laurent B's avatar
Laurent B committed
49

50
**:information_source: With `PowerStencil` data can either be managed as documents to edit (Yaml) or fully as code within a powerful shell !!**. Choose the method that fits the most to what you are doing.
Laurent B's avatar
Laurent B committed
51

Laurent B's avatar
Laurent B committed
52
53
The `PowerStencil` gem provides the `power_stencil` executable which is a pluggable CLI to:

Laurent B's avatar
Doc    
Laurent B committed
54
55
56
57
- Manage your project **relational-integrity-checked data** in a versioned, git diff friendly yet flexible format.
- Visualize your **data relations with graphviz**. Customize generated graphs easily.
- Easily **manage templates** to generate whatever your project/organization requires.
- Define builds that you can easily **embed in your CI/CD process** or use them standalone.
Laurent Briais's avatar
Laurent Briais committed
58
59
- Propose mechanism to temporarily override some data to test changes impact without polluting the official repository data.
- Ensure **re-usability** with a versatile extension mechanism, including plugins.
Laurent B's avatar
Laurent B committed
60

Laurent Briais's avatar
Doc    
Laurent Briais committed
61
# Pfff, yet another templating engine...?
Laurent B's avatar
Laurent B committed
62

Laurent B's avatar
Doc    
Laurent B committed
63
64
**Actually `PowerStencil` is _not_ a templating engine**. It uses already existing and proven templating engines.

Laurent Briais's avatar
Laurent Briais committed
65
`PowerStenil` currently embeds two templating engines:
Laurent B's avatar
Doc    
Laurent B committed
66

Laurent Briais's avatar
Laurent Briais committed
67
68
* **[ERB] is the default**. This is a pretty standard and very well known engine in the [Ruby] world and especially for people who already used the [Rails] framework. This is a generic templating engine which can handle any use-case. Each template will be processed by default using the [ERB] engine.
* When it comes to generate mark-up documents like XML or HTML, many people prefer to use [Haml], as it designed towards generating these kinds of documents and tends to end-up with much clearer templates. Any template whose names ends by `.haml` will be processed by the [Haml] engine by default instead of [ERB].
Laurent Briais's avatar
Doc    
Laurent Briais committed
69

Laurent Briais's avatar
Laurent Briais committed
70
So `PowerStencil` is definitely not a templating engine, it is actually much more than that. It is a framework to manage your templates, design the relational data you can use in your templates, manage the versioning, define the way you build things etc... Read this documentation (and especially the [getting started](#getting-started) section which is a tutorial in itself).
Laurent B's avatar
Laurent B committed
71
72


Laurent B's avatar
Laurent B committed
73
# Installation
Laurent B's avatar
Laurent B committed
74

Laurent Briais's avatar
minor    
Laurent Briais committed
75
76
77
You need a working install of the [Ruby] language (>= 2.3). You should have `bundler` installed (`gem install bundler`). This is not added as an automatic dependency as updating `bundler` may not be what you want. Yet any working install of Ruby should consider having `bundler` installed.

:hand: It is **strongly** advised to have `git` installed on your system, as `PowerStencil` is integrated with `git` to provide a better user experience.
Laurent B's avatar
Laurent B committed
78

Laurent Briais's avatar
Laurent Briais committed
79
Then you can install the `PowerStencil` gem by issuing the usual:
Laurent B's avatar
Laurent B committed
80
81
82

    $ gem install power_stencil

83
If you want to create [Graphviz] graphs, you probably may want to install it for your system. If you are using an _apt-based_ system like Ubuntu or Debian it may probably be as simple as:
Laurent B's avatar
Doc    
Laurent B committed
84
85
86

    $ sudo apt install graphviz

87
88
If you use Windows, install [Graphviz] using the stable version for Windows and add the `bin` directory to the `%PATH%`.

Laurent B's avatar
Laurent B committed
89
# Usage
Laurent B's avatar
Laurent B committed
90

Laurent B's avatar
Laurent B committed
91
## Help
Laurent B's avatar
Laurent B committed
92

Laurent Briais's avatar
Laurent Briais committed
93
The `power_stencil` CLI provides a contextual help. You can begin with:
Laurent B's avatar
Laurent B committed
94
95
96
97
98

    $ power_stencil --help

That will display a basic help:

Laurent B's avatar
Laurent B committed
99
```shell
Laurent Briais's avatar
Laurent Briais committed
100
PowerStencil is the Swiss-army knife templating workflow for developers and ops.
Laurent B's avatar
Laurent B committed
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
-- Options ---------------------------------------------------------------------
    -v, --verbose  Displays extra runtime information.
    -h, --help  Displays this help.
    --program-version, -V, --version  Displays program version.
    --simulate  Will not perform actual actions
    --debug  Debug mode
    --debug-on-err, --debug-on-stderr  Sends debugging to SDTERR
    --log-level  Defines the level of logging (0 to 5)
    --log-file  Specifies a file to log into
    --truncate-log-file  Truncates the log file (appends by default)
    --project-path  Specifies a startup path to use instead of '.'
    --auto  Bypasses command-line confirmations to the user

--------------------------------------------------------------------------------
 Following subcommands exist too:
 For more information you can always issue sub_command_name --help...
--------------------------------------------------------------------------------
 * init: Initializes a PowerStencil repository ...
 * info: Generic information about the repository ...
120
 * plugin: Manipulates plugins ...
121
 * get: Query entities from repository ...
Laurent B's avatar
Laurent B committed
122
 * shell: Opens a shell to interact with entities ...
123
 * adm: Administrative commands ...
124
125
126
127
 * check: Check repository entities consistency ...
 * create: Creates entities in the repository ...
 * edit: Edit entities from repository ...
 * delete: Delete entities from repository ...
Laurent Briais's avatar
Laurent Briais committed
128
 * describe: Detailed information about entity types ...
Laurent B's avatar
Laurent B committed
129
130
131
132
133
134
135
 * build: Builds entities ...
```

The program uses the standard paradigm of sub-commands (à-la-git), and each can have its own options that you can check using:

    $ power_stencil <subcommand> --help

Laurent B's avatar
Laurent B committed
136
[Plugins] may bring extra subcommands and/or options, so depending on the project you are working in, the output of `--help` may differ...
Laurent B's avatar
Laurent B committed
137

138
139
140
141




Laurent Briais's avatar
Laurent Briais committed
142
143
144
## Getting started

See the following topics as a kind of tutorial on `PowerStencil`. You should read them in order, as each of them assumes you already read the previous one.
145

Laurent Briais's avatar
Laurent Briais committed
146
147

### Creating a `PowerStencil` project
Laurent B's avatar
Laurent B committed
148

Laurent Briais's avatar
Laurent Briais committed
149
To create a new project, use the `init` sub-command. It works as you would expect. If you run it in an existing directory it will create the project here but you can specify the directory where you want the project to be created (the path will be created provided you have enough rights on the filesystem):
Laurent B's avatar
Laurent B committed
150

151
    $ power_stencil init /where/you/want/your/project
Laurent B's avatar
Laurent B committed
152

153
Once the project created, if you are anywhere within the project tree, you don't need to specify the project path anymore (if needed, any `power_stencil` sub-command supports a `--project-path` option). If you're used to the Git command line, it works the same way, and you should not feel in uncharted territory...
Laurent B's avatar
Laurent B committed
154

Laurent Briais's avatar
Laurent Briais committed
155
**:information_source: The rest of this documentation will assume you are at the root of this created project.**
Laurent Briais's avatar
Laurent Briais committed
156

157
**:information_source: If you have `git` installed on your system, **the repository of the newly created project has been automatically turned into a git repository. And any action done through the `power_stencil` command-line will be automatically tracked by `git`**. Only things you will do outside of the `power_stencil` command-line (adding or modifying templates, creating or modifying entity types... any manual action) will require a user action to make `git` take it in account. You can completely de-activate this behaviour if you want to fully manage things by yourself by adding `:no-git: true` in the `.ps_project/versioned-config.yaml`, yet there is no good reason for that... Unless you know what you are doing, you should keep the default settings. See [Git integration] for further information.
Laurent B's avatar
Laurent B committed
158

Laurent Briais's avatar
Laurent Briais committed
159
### `PowerStencil` project structure overview
Laurent B's avatar
Laurent B committed
160

Laurent B's avatar
Laurent B committed
161
162
163
164
165
166
167
168
169
170
171
The structure of a brand new `PowerStencil` project is the following:

```
├── .gitignore
└── .ps_project
    ├── entities
    │   └── README.md
    ├── entity_definitions
    │   └── README.md
    ├── personal-config.yaml
    ├── plugins
Laurent B's avatar
Laurent B committed
172
173
    ├── templates-templates
    │   └── README.md
Laurent B's avatar
Laurent B committed
174
175
176
177
    ├── user_entities
    │   └── README.md
    └── versioned-config.yaml
```
Laurent Briais's avatar
Doc    
Laurent Briais committed
178
179

Other directories will appear once you start working with the project, perform some [builds] or when you will define [templates], but this is the initial state.
Laurent B's avatar
Laurent B committed
180

Laurent B's avatar
Doc    
Laurent B committed
181
Not all the files in this directory should be versioned, and this the reason why there is a preconfigured `.gitignore` at the root of the project. It is prefilled so that unless you want to specifically avoid a file to be versioned, you don't need to worry regarding `PowerStencil` project files. **By default everything that must be versioned actually is and everything that shouldn't isn't**.
Laurent B's avatar
Laurent B committed
182

Laurent B's avatar
Doc    
Laurent B committed
183
The project has two distinct config files.
Laurent Briais's avatar
Doc    
Laurent Briais committed
184

Laurent Briais's avatar
Laurent Briais committed
185
186
- Keep the official project config in the `.ps_project/versioned-config.yaml` which, as its name suggest, will be git versioned.
- But any developer who wants to temporarily override some configuration should do it in the `.ps_project/personal-config.yaml`, which is not to be versioned. The content defined in the latter will override the content of the versioned one.
Laurent B's avatar
Laurent B committed
187

Laurent B's avatar
Doc    
Laurent B committed
188
**This is a clean way for anyone who wants to test something to safely do it locally without the risk to pollute the central repo.**
Laurent B's avatar
Laurent B committed
189

Laurent Briais's avatar
Laurent Briais committed
190
There is the same mechanism between `.ps_project/entities` (where the project data is stored and which is versioned) and `.ps_project/user_entities` (where local data is stored and not versioned), but as opposed to the two aforementioned config files, you should not edit anything directly there, but instead use the CLI or the shell. More information on how to manage entities [here][entities].
Laurent B's avatar
Doc    
Laurent B committed
191
192

The `plugins` directory can optionally contain project specific plugins. Plugins are a great way to extend `PowerStencil` for the needs of a project or organization.
Laurent B's avatar
Laurent B committed
193

Laurent Briais's avatar
Laurent Briais committed
194
195
196
197
198
199
200
201
202
203
204
205
206
### Command line auto-completion

`PowerStencil` comes with a powerful command-line auto-completion feature for `zsh`. (It is not yet available for `bash`).

You can install it by running:

    $ power_stencil adm --zsh-completion

It provides a very useful auto-completion for sub-commands, options as well as options parameters...

`PowerStencil` uses a _two-stages_ auto-completion mechanism:

* Project independent completion is generated in `~/.zsh/completion/_power_stencil`, whereas completion related to new commands or options, coming from plugins used in a project, is generated (and versioned) within the project itself, in `.ps_project/.zsh_project_completion`.
Laurent B's avatar
Laurent B committed
207

Laurent Briais's avatar
minor    
Laurent Briais committed
208
* Meaning that if you want to benefit from the "_full_" completion from within a project (ie including extra command line options brought by the project itself), you may have to re-run `power_stencil add --zsh-completion` from within any project where you use extra plugins, in order to (re)generate the `.ps_project/.zsh_project_completion` file. If you don't, it is not a big deal, but then you will only have access to the "_generic_" completion, ie completion not taking in account any of the extra commands or options you are using in this project brought by plugins you may use...
Laurent B's avatar
Doc    
Laurent B committed
209
210

### Entities
Laurent B's avatar
Laurent B committed
211

Laurent B's avatar
Doc    
Laurent B committed
212
The **core of the system** is the **[entity][entities]**, so you should start by having a look at what entities are, how they are easily managed using the `PowerStencil` CLI and shell.
Laurent B's avatar
Laurent B committed
213

Laurent B's avatar
Doc    
Laurent B committed
214
### Templates
Laurent B's avatar
Laurent B committed
215

Laurent B's avatar
Doc    
Laurent B committed
216
Then it is important to understand how to use entities within **[templates]**. Templates are the way to **generate whatever you need** (doc, code, descriptors, sites, whatever...) from the data that the entities represent.
Laurent B's avatar
Laurent B committed
217

Laurent B's avatar
Doc    
Laurent B committed
218
### Builds
Laurent B's avatar
Laurent B committed
219

Laurent B's avatar
Doc    
Laurent B committed
220
The mechanism that **combines entities and templates** is called a **[build][builds]**. A build is always attached to an entity (you build something). The result of a build is a set of files. Optionally an action can be triggered after the files are generated (could be as simple as calling a script that has been generated).
Laurent B's avatar
Laurent B committed
221

Laurent B's avatar
Doc    
Laurent B committed
222
### Plugins
Laurent B's avatar
Laurent B committed
223

Laurent B's avatar
Doc    
Laurent B committed
224
225
`PowerStencil` could stop there, and you would be able to do whatever you want, but there is a whole world beyond. **[Plugins]** provide a way to completely **extend `PowerStencil` and ease cross-projects share and re-usability**, further control relations between entities, implement complex post-build actions, add CLI sub-commands and options. 
Plugins can be local to the project or coming in the form of standard Ruby Gems ! The icing on the cake...
Laurent B's avatar
Laurent B committed
226

227
228
229
230
### Git integration

Git is closely integrated at the heart of `PowerStencil`. See how works the [Git integration].

Laurent B's avatar
Doc    
Laurent B committed
231
<br>
Laurent Briais's avatar
Laurent Briais committed
232

Laurent Briais's avatar
Laurent Briais committed
233
And of course, you may want to **read the [F.A.Q.]**
Laurent Briais's avatar
Laurent Briais committed
234

Laurent B's avatar
Laurent B committed
235
236
# Project status

Laurent B's avatar
Laurent B committed
237
Check `PowerStencil` [status in the F.A.Q.].
Laurent Briais's avatar
Laurent Briais committed
238

Laurent B's avatar
Laurent B committed
239
# Contributing
Laurent B's avatar
Laurent B committed
240

Laurent B's avatar
Laurent B committed
241
Bug reports and pull requests are welcome on Gitlab at https://gitlab.com/tools4devops/power_stencil. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
Laurent B's avatar
Laurent B committed
242
243
244

## License

Laurent B's avatar
Laurent B committed
245
The gem is available as open source under the terms of the [MIT License].
Laurent B's avatar
Laurent B committed
246
247
248

## Code of Conduct

Laurent B's avatar
Laurent B committed
249
Everyone interacting in the PowerStencil project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct].
Laurent B's avatar
Laurent B committed
250

Laurent B's avatar
Laurent B committed
251
<!-- End of Document -->
Laurent B's avatar
Laurent B committed
252
253

<!-- Pages -->
Laurent B's avatar
Laurent B committed
254
255
256
257
[templates]: doc/templates.md "Templates in PowerStencil"
[entities]: doc/entities.md "Entities in PowerStencil"
[builds]: doc/builds.md "Builds in PowerStencil"
[plugins]: doc/plugins.md "Plugins in PowerStencil"
Laurent B's avatar
Laurent B committed
258
[example use cases]: doc/example_use_cases.md "Example uses cases using PowerStencil"
259
[Git integration]: doc/git_integration.md "See of git is integrated with PowerStencil"
Laurent Briais's avatar
Laurent Briais committed
260
[F.A.Q.]: doc/faq.md "PowerStencil F.A.Q."
Laurent B's avatar
Laurent B committed
261
[status in the F.A.Q.]: doc/faq.md#what-is-the-status-of-powerstencil- "PowerStencil project status"
Laurent Briais's avatar
Doc    
Laurent Briais committed
262
[DB in F.A.Q.]: doc/faq.md#why-is-a-real-database-a-bad-idea-as-configuration-store-for-cicd- "Databases in build processes"
Laurent B's avatar
Laurent B committed
263
[code of conduct]: CODE_OF_CONDUCT.md
Laurent B's avatar
Laurent B committed
264

Laurent B's avatar
Laurent B committed
265
<!-- Code links -->
Laurent B's avatar
Laurent B committed
266
[ERB implementation]: lib/power_stencil/engine/renderers/erb.rb "Mostly the only module you need to clone to implement another templating engine"
Laurent B's avatar
Laurent B committed
267

Laurent B's avatar
Laurent B committed
268
<!-- Illustrations -->
Laurent B's avatar
Laurent B committed
269
270
[simple-flow-image]: doc/images/power-stencil-simple-flow.svg

Laurent B's avatar
Laurent B committed
271
<!-- External links -->
Laurent B's avatar
Laurent B committed
272
[MIT License]: http://opensource.org/licenses/MIT "The MIT license"
Laurent B's avatar
Laurent B committed
273
[ERB]: https://ruby-doc.org/stdlib-2.6.3/libdoc/erb/rdoc/ERB.html "Quick ERB description"
Laurent B's avatar
Laurent B committed
274
[Haml]: http://haml.info/ "The templating engine for XML-like documents"
275
[Ruby]: https://www.ruby-lang.org "The powerful Ruby language"
Laurent B's avatar
Laurent B committed
276
[Rails]: https://rubyonrails.org/ "The Ruby on Rails framework"
277
[PowerStencil site]: https://powerstencil.brizone.org "Official PowerStencil website"
Laurent Briais's avatar
Minor    
Laurent Briais committed
278
[Graphviz]: https://www.graphviz.org/ "Graph Visualization Software"