Verified Commit fa7701b5 authored by doshitan's avatar doshitan
Browse files

Updates for make and shell pages

parent 3097228c
......@@ -12,7 +12,8 @@ Note I almost always use and build for [[https://www.gnu.org/software/make/][GNU
than other makes]].
* Basics
See the [[https://www.gnu.org/software/make/manual/make.html#Introduction][intro section]] of the GNU Make manual.
See the [[https://www.gnu.org/software/make/manual/make.html#Introduction][intro section]] of the GNU Make manual. And [[https://www.jfranken.de/homepages/johannes/vortraege/make.en.html][this page]] is a good intro and
reference.
Basic format:
......@@ -25,11 +26,16 @@ target: prerequisites ...
...
#+END_SRC
Each line execute in a separate subshell, so you can not set variables inside a
target.
Each line execute in a separate shell, so you can not set variables inside a
target. Chain commands together with ~&&~ or ~;~, e.g.,
#+BEGIN_SRC makefile
thing:
a_command && b_command # execute in same shell
a_command; b_command # also executes in same shell
#+END_SRC
If you have a long line, use backslashes to break it up across lines.
#+BEGIN_SRC make
#+BEGIN_SRC makefile
do-thing:
command thing $(SOME_VAR) \
--a-flag \
......@@ -67,17 +73,18 @@ target build.
#+BEGIN_SRC makefile
clean:
-rm -f *.o
-rm *.o
#+END_SRC
Finishes successfully even if the ~rm~ errors (like if there are no ~.o~ files
to clean).
to clean, for this specific case you'd probably just use the ~-f~ on ~rm~
itself, but this is just an example).
You can combine them:
#+BEGIN_SRC makefile
clean:
@-rm -f *.o
@-rm *.o
#+END_SRC
Doesn't say what it's doing or care if it errors.
......@@ -95,22 +102,38 @@ I prefer lowercase ~makefile~ as it's ever so slightly easier to type and it's
usually surrounded by other lowercased names so it looks better to me to match.
* Make as a task runner
Often in projects you have a number of commands that you may wish to run on
occasion. There are many existing task runner solutions out there, the
JavaScript world in particular seems to love to create them. I shy away from
occasion. There are many existing task runner solutions out there[fn::The
JavaScript world in particular seems to love to create them.]. I shy away from
these as I think often they are overcomplicated for the task at hand.
If a project has a JavaScript dependency, often it will use the ~scripts~
section in it's ~package.json~ file, which can easily be run with ~npm run
<script name>~. I dislike this for a few reasons:
1. I avoid JavaScript dependencies as much as possible
2. I generally want a package manager to be responsible for one thing, getting packages
2. I generally want a package manager to be responsible for one thing, getting
packages
3. I prefer to have npm scripts disabled globally as they are a [[https://blog.npmjs.org/post/141702881055/package-install-scripts-vulnerability][security hazard]]
4. Can't have comments (because JSON)
Similar reasons apply to pipenv scripts or whatever, especially point two.
If you already have a build system, e.g., rake, Ant/Maven/Gradle, SCons, etc.,
maybe it's easier to stuff tasks in there, but even then I think a simple
makefile might have room for high level project management stuff.
Advantages of make:
- Been around forever
- Available everywhere and probably already installed
- Just let's you write shell scripts
- Easy/automatic [[#shell-tab-complete][tab-completion support]] for targets in most shells
- Independent of your project specific language environment, a couple advantages to this:
- A more stable tool, language specific tooling usually grows and changes a
lot more than make will, so it can paper over those changes. It's also
divorced from you package management, so less churn, easier to track
history, etc.
- A common interface between different projects (which are all potentially
written in a variety of different languages). You can build up common
targets and practices around make that can be shared between all your
projects, providing a consistent interface when bouncing between projects.
Disadvantages of make:
- Old and thus not purposely designed for today's environment
......@@ -292,28 +315,233 @@ targets for the common cases. But you do you.
[[https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html][This post]] describes the approach in detail and is quite handy. In sort, annotate
your targets like:
#+BEGIN_SRC make
#+BEGIN_SRC makefile
deps: ## Install project dependencies
#+END_SRC
Add the magic incantation
Add the magic incantation:
#+BEGIN_SRC makefile
help: ## Displays this help screen
@grep -Eh '^[[:print:]]+:.*?##' $(MAKEFILE_LIST) | \
sort -d | \
awk -F':.*?## ' '{printf "\033[36m%s\033[0m\t%s\n", $$1, $$2}' | \
column -ts $$'\t'
#+END_SRC
And then when you run ~make help~ you get a nicely formatted help page.
Quick line-by-line breakdown, first grep the makefile(s) for our special comments:
#+BEGIN_SRC
┌ tell make not to print the line, we are only interested in the output of the command
| ┌ the regex to match your comment pattern
| | ┌ automatic variable, holds all filenames that have been parsed for make rules on this invocation
| ┌--------+----------┐ ┌-------+------┐
@grep -Eh '^[[:print:]]+:.*?##' $(MAKEFILE_LIST) | \
||
|└ don't print filenames, for when multiple makefiles are parsed (say by including another)
└ extended regex support
#+END_SRC
Then we sort the lines grep returns:
#+BEGIN_SRC
sort -d | \
#+END_SRC
Then add some color to the target name, which means: split the line into it's
parts, color the name, then recombine:
#+BEGIN_SRC
┌ split the input based on our comment pattern
| set color ┐ ┌ reset color
┌----+----┐ ┌--+---┐ ┌--+--┐
awk -F':.*?## ' '{printf "\033[36m%s\033[0m\t%s\n", $$1, $$2}' | \
└┘ └┘└┘
target name ┘ | └ comment/help text
└ separator character (TAB in this case)
#+END_SRC
Then nicely align everything:
#+BEGIN_SRC
┌ table mode, so columns and rows fill correctly
|┌ set separator character between columns
||
column -ts $$'\t'
|└-+-┘
| └ a literal tab character in shell
└ escape the $ for the tab character since we are in make
#+END_SRC
#+BEGIN_SRC make
I use a tab character to separate the target and help text as it's unlikely to
be used in the help text, feel free to chose a different one, just keep it in
sync between the `awk` and `column`.
You can of course go simpler, something like:
#+BEGIN_SRC makefile
help:
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-10s\033[0m %s\n", $$1, $$2}'
grep -E "^[[:print:]]+: ##" [Mm]akefile
#+END_SRC
And then when you run ~make help~ you get a nicely formatted help page, with the
simple example above:
Which will show the targets in the order they are in the makefile, even
highlighting the target name (and other comment pattern junk).
#+BEGIN_SRC bash
deps Install project dependencies
Or use a different pattern:
#+BEGIN_SRC makefile
# target: help - Display callable targets.
help:
@grep -E "^# target:" [Mm]akefile
# target: list - List source files
list:
# Won't work. Each command is in separate shell
cd src
ls
# Correct, continuation of the same shell
cd src; \
ls
# target: dist - Make a release.
dist:
tar -cf $(RELEASE_DIR)/$(RELEASE_FILE) && \
gzip -9 $(RELEASE_DIR)/$(RELEASE_FILE).tar
#+END_SRC
Ultimately, you're just grepping some files, do whatever works for you.
* When order matters
The prerequisites of a target are not guaranteed to be run in any particular
order. For instance:
#+BEGIN_SRC makefile
thing: do-one do-two do-three
./scripts/thing.sh
#+END_SRC
So the ~do-{one,two,three}~ targets will get run before ~thing~ does, but the
order they get run in not deterministic, ~do-one~ is not guaranteed to run
before ~do-two~ and so on[fn::This is ignoring what prerequisites the ~do-~
targets might have for the moment].
Now they often *do* get run in the order you list them and for simple, small
things, you can usually get by, you just have to be careful the targets are not
a part of any other targets you might want to run with ~-j~.
But sometimes this matters, sometimes you really want to run a series of other
make targets in a guaranteed order. For this you have to reach for [[https://www.gnu.org/software/make/manual/html_node/Recursion.html][recursive
make]].
WARNING: Recursive make can be a bit of a gotcha, and should be avoided if
possible in simple cases. For instance, variables that are not marked for export
are not automatically inherited to the sub-make. Read up before you start down
the recursive path.
#+BEGIN_SRC makefile
thing:
$(MAKE) do-one
$(MAKE) do-two
$(MAKE) do-three
./scripts/thing.sh
#+END_SRC
~$(MAKE)~ is just a variable that is set to the same make command that is
currently running.
Calling ~make~ inside of a make target is traditionally used to call other
makefiles in sub-directories to build smaller components of a system and so make
will print some extra information about what directory it's running in when
called recursively.
For task-runner purposes this can be a little annoying. GNU make has a easy flag
to turn this off though, so I'd recommend setting a special variable with the
flag and use it, like:
#+BEGIN_SRC makefile
# (q)uiet (make), use whatever you want
QMAKE := $(MAKE) --no-print-directory
thing:
$(QMAKE) do-one
$(QMAKE) do-two
$(QMAKE) do-three
./scripts/thing.sh
#+END_SRC
If you want to quiet if for every recursive make call, then you could just add
it to ~MAKEFLAGS~ like:
#+BEGIN_SRC makefile
MAKEFLAGS += --no-print-directory
thing:
$(MAKE) do-one
$(MAKE) do-two
$(MAKE) do-three
./scripts/thing.sh
#+END_SRC
And achieve the same thing.
For non-GNU make you can set the ~-s~ flag instead, but that silences *all*
output, which is usually less desirable.
*If* you can manage to structure your targets just so, then you can enforce some
ordering with just prerequisites, like:
#+BEGIN_SRC makefile
do-one:
do-two: do-one
do-three: do-two
# technically would only need to say do-three
thing: do-one do-two do-three
./scripts/thing.sh
#+END_SRC
But often you can't structure your targets like this.
Also, there are things called [[https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html][order-only prerequisites]], but don't let that name
fool you, they do not get run in a specified order, they are just prerequisites
that are always considered out of date and so are always run (which sometimes
matters).
* Running other targets
Our old friend recursive make.
#+BEGIN_SRC makefile
build:
gcc -Wall $(args)
build-dev:
$(MAKE) build args='-O0'
build-prod:
$(MAKE) build args='-O9'
#+END_SRC
Though sometimes it might be better to pull out the common bits into a variable,
and not call make, like:
#+BEGIN_SRC makefile
BUILD_CMD := gcc -Wall
build:
$(BUILD_CMD) $(args)
build-dev:
$(BUILD_CMD) -Og
build-prod:
$(BUILD_CMD) -O2
#+END_SRC
But sometimes your other targets you want to call, have certain prerequisites
setup that you don't want to have to duplicate and other stuff. So sometimes it
is easier to use recursive make.
* Set SHELL
By default, regardless of what your personal shell is, make targets run under
~/bin/sh~. You can [[https://www.gnu.org/software/make/manual/html_node/Choosing-the-Shell.html][change this]] by explicitly setting the ~SHELL~ variable either
globally in the makefile or for a specific target like:
#+BEGIN_SRC make
#+BEGIN_SRC makefile
clean: SHELL:=bash
rm $(SOME_DIR)/{one,two,three}/*.junk
#+END_SRC
......@@ -365,6 +593,8 @@ test.html: test.md
~$<~ is the expanded prerequisite name (~test.md~ in this case).
~$@~ is the expanded target name (~test.html~ in this case).
Find more [[https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html][automatic variables]] in the docs.
Better. But we can generalize a little more, say if we have a bunch of markdown
files, maybe documentation stuff, we can use a [[https://www.gnu.org/software/make/manual/html_node/Pattern-Rules.html][pattern rule]]:
......
......@@ -8,10 +8,29 @@ cross-system compatibility. I'll try to note when something is Zsh-specific.
My [[https://gitlab.com/doshitan/dots-zsh][config files]].
* explainshell.com
If you come across a command you don't understand[fn::One you wrote a while ago
more often than not.], plop it in https://explainshell.com/, hopefully it can
help break it down.
* TAB complete
A wonderful feature in almost every shell. Start typing a part of a command
name, file name, whatever, then press the @@html:<kbd>TAB</kbd>@@ key, the shell
will then auto-complete the rest for you or cycle through possibilities if there
isn't a unique option.
Will save your fingers.
Different shells come with different completion providers built in. Often there
is a bundle of extra ones available you can install, like
https://github.com/scop/bash-completion for bash.
And you can write your own too if needed or desired.
* Moving around
~cd~ (with no argument) moves to your home directory.
~cd -~ moves to your last location.
~cd -~ moves to your last location[fn::This is a pattern supported by some other
tools as well, like git, ~git checkout -~ will checkout the last branch you were
on, makes it easy to swap between branches.].
~pushd~ and ~popd~ create a directory stack which can be helpful when juggling
deeply nested paths or simply to store the current location, move around other
......@@ -57,7 +76,6 @@ slightly shorter ~^foo^bar~ syntax, but ~!!:s//~ often comes to my mind first.
too thin on examples. If that's the case, checking [[https://tldr.sh/][tldr.sh]] can maybe yield
clearer docs. There are a variety of CLI clients for it as well as the [[https://tldr.ostera.io/][web
client]].
* Keyboard navigation
Most shells use [[https://en.wikipedia.org/wiki/GNU_Readline][Readline]] to handle user movements which operates in ~emacs~ mode
by default. Often you can switch to a ~vi~ mode, but it's good to know some
......@@ -67,6 +85,10 @@ basic movements.
- @@html:<kbd>Ctrl-e</kbd>@@ moves to end of line
- @@html:<kbd>Alt-b</kbd>@@ moves back one word
- @@html:<kbd>Alt-f</kbd>@@ moves forward one word
- @@html:<kbd>Ctrl-w</kbd>@@ delete back one word
- @@html:<kbd>Alt-d</kbd>@@ delete forward one word
- @@html:<kbd>Ctrl-u</kbd>@@ delete line backward
- @@html:<kbd>Ctrl-k</kbd>@@ delete line forward
- @@html:<kbd>Alt-.</kbd>@@ inserts the last argument to previous command
- @@html:<kbd>Ctrl-r</kbd>@@ search command history backwords
- @@html:<kbd>Ctrl-x</kbd> <kbd>Ctrl-e</kbd>@@ to edit current line in
......@@ -195,8 +217,10 @@ A full loaded line like:
set -Eeuxo pipefail
#+END_SRC
[[https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/][Arguably]] makes shell scripts much safer, but they usually need to be written
from the start with those flags in mind to work at all.
This can be thought of like a [[http://redsymbol.net/articles/unofficial-bash-strict-mode/][strict mode]] for your script. Your scripts usually
need to be written from the start with those flags in mind to work at all.
An alternate and longer discussion of the flags can be found [[https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/][here]].
** Parameters/Arguments
You can reference script arguments positionally, ~$1~ for the first argument,
~$2~ for the second and so on[fn:: ~$0~ gives you the name of the script.].
......@@ -305,3 +329,83 @@ options, such as ~nix-shell~:
This is a feature of the particular program, not a general feature, so you'll
know if you can do something like that.
** Functions
Functions are a thing in most shell languages and are great for the same reason
they are great in other languages.
Basic example:
#+BEGIN_SRC bash
do_thing() {
local param=$1
echo $param
}
do_thing "hello, world"
#+END_SRC
Notes:
- [[#parametersarguments][Parameters]] are handled just like a script, ~$1~, ~$2~, ~$@~, etc.
- Functions need to be defined in the file before they are executed.
- ~local~ scopes the variable to the function instead of the global space (as
shell variables usually are) and should generally be preferred.
- Functions don't return values, the ~return~ statement exists and it sets the
return status of the command (retrievable with ~$?~ after running the
command), which is sometimes what you want. If you want to pass values out of
a function either a) set a global variable or b) ~echo~ the value to stdout
and capture the output when you call it (i.e., ~result=$(do_thing "hello")~).
If you have a few scripts that could share some functionality, you can define
functions in a separate file, say ~lib.sh~ and source it in your other scripts
~source lib.sh~[fn::Or ~. lib.sh~] making the functions available there.
* Scripting Scripts
Sometimes there are interactive programs (i.e., they prompt the user for input)
that you want to automate. The most basic situation being a command that prompts
for confirmation.
If there's only one prompt:
#+BEGIN_SRC bash
$ echo "yes" | command_that_prompts
#+END_SRC
works fine. If there are multiple prompts:
#+BEGIN_SRC bash
$ yes | command_that_prompts_multiple_times
#+END_SRC
~yes~ just repeatedly outputs the string ~y~, actually it just repeatedly
repeats whatever you string you pass in, defaulting to ~y~. So say you wanted to
say no to a bunch of prompts:
#+BEGIN_SRC bash
$ yes 'n' | command_you_say_no_to
#+END_SRC
When you have more complicated interactions, might want to reach for [[https://en.wikipedia.org/wiki/Expect][~expect~]].
* Record shell session
Sometimes you want to log some work you are doing in the terminal. The ~script~
command can help with that.
#+BEGIN_SRC bash
$ script my_log
#+END_SRC
Will start a subshell, recording all commands and output to the filename
specified (~my_log~ in this case). When you want to stop recording, just ~exit~
the shell (or @@html:<kbd>Ctrl-d</kbd>@@).
* ~true~ alternative
~:~ (a single colon) is equivalent to ~true~ in most shells. It can be useful as
a no-op on occasion or a quick way to ignore the failure of a command
(~command || :~), but often using ~true~ is more readable.
* More resources
- The [[https://wiki.bash-hackers.org/][Bash Hackers Wiki]] is a wonderful reference. Skim it, pick something you
don't recognize and learn about it.
- The [[https://www.tldp.org/LDP/abs/html/][Advanced Bash-Scripting Guide]] is a solid introductory text if you are just
getting started shell scripting and want something structured to read. But
also a ton of advanced topics, *with examples*, and reference material.
- It depends on your system, but the man page for your shell will either be
helpful or only something for masochists, try ~man bash~.
- https://github.com/jlevy/the-art-of-command-line
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment