---
title: Shell
toc: true
toc-margin: true
published: 2019-04-16T11:23:06-05:00
modified: 2019-08-21T08:27:38-04:00
---

I primarily use [[https://www.zsh.org/][Zsh]], though often must write code suitable for Bash for
cross-system compatibility. I'll try to note when something is Zsh-specific.

My [[https://gitlab.com/doshitan/dots-zsh][config files]].

* explainshell.com
If you come across a command you don't understand[fn::One you wrote a while ago
more often than not.], plop it in https://explainshell.com/, hopefully it can
help break it down.
* TAB complete
A wonderful feature in almost every shell. Start typing a part of a command
name, file name, whatever, then press the @@html:<kbd>TAB</kbd>@@ key, the shell
will then auto-complete the rest for you or cycle through possibilities if there
isn't a unique option.

Will save your fingers.

Different shells come with different completion providers built in. Often there
is a bundle of extra ones available you can install, like
https://github.com/scop/bash-completion for bash.

And you can write your own too if needed or desired.
* Moving around
~cd~ (with no argument) moves to your home directory.

~cd -~ moves to your last location[fn::This is a pattern supported by some other
tools as well, like git, ~git checkout -~ will checkout the last branch you were
on, makes it easy to swap between branches.].

~pushd~ and ~popd~ create a directory stack which can be helpful when juggling
deeply nested paths or simply to store the current location, move around other
places and come back without having to explicitly store the original location
somewhere.

~z~ learns your most used locations and makes it quick to jump to them. It's
[[https://github.com/robbyrussell/oh-my-zsh/tree/master/plugins/z][bundled with oh-my-zsh]], but is [[https://github.com/rupa/z][independent]] and works in bash too. Say you have a
place ~~/some/cool/project/named/awesome~, after moving to it a few times, a
simple ~z awe~ would jump you directly there.
* Have command skip history
A space before a command excludes it from the shell history/getting logged.
Useful when you need to set some sensitive info for a command and don't want it
sticking around on your computer.

#+BEGIN_SRC bash
$  export MY_SECRET="a very secret string I don't want logged"
$ command -i $MY_SECRET
#+END_SRC

You have to check your shell is setup correctly or set the options in your shell
config, for [[https://unix.stackexchange.com/a/6104][zsh]] and for [[https://unix.stackexchange.com/a/32461][bash]].
* Last command
~!!~ expands to the last command, which is very useful in things like ~sudo !!~,
i.e., rerun the last command, but with sudo. You can also move farther back,
with things like ~!-2~ (~!!~ is equivalent to ~!-1~).

Another very useful application of ~!!~ is with search-and-replace ~:s~ / ~:gs~.
For example,

#+BEGIN_SRC bash
$ cp some_dir/with/some/file/foo other_dir/to/store/copy/foo-copy
$ !!:gs/foo/bar
# expands to
$ cp some_dir/with/some/file/bar other_dir/to/store/copy/bar-copy
#+END_SRC

For the common case of search-and-replace the last command you can use the
slightly shorter ~^foo^bar~ syntax, but ~!!:s//~ often comes to my mind first.
* man page supplement

~man~ pages are your friend, but sometimes they can either be too detailed or
too thin on examples. If that's the case, checking [[https://tldr.sh/][tldr.sh]] can maybe yield
clearer docs. There are a variety of CLI clients for it as well as the [[https://tldr.ostera.io/][web
client]].
* Keyboard navigation
Most shells use [[https://en.wikipedia.org/wiki/GNU_Readline][Readline]] to handle user movements which operates in ~emacs~ mode
by default. Often you can switch to a ~vi~ mode, but it's good to know some
basic movements.

- @@html:<kbd>Ctrl-a</kbd>@@ moves to beginning of line
- @@html:<kbd>Ctrl-e</kbd>@@ moves to end of line
- @@html:<kbd>Alt-b</kbd>@@ moves back one word
- @@html:<kbd>Alt-f</kbd>@@ moves forward one word
- @@html:<kbd>Ctrl-w</kbd>@@ delete back one word
- @@html:<kbd>Alt-d</kbd>@@ delete forward one word
- @@html:<kbd>Ctrl-u</kbd>@@ delete line backward
- @@html:<kbd>Ctrl-k</kbd>@@ delete line forward
- @@html:<kbd>Alt-.</kbd>@@ inserts the last argument to previous command
- @@html:<kbd>Ctrl-r</kbd>@@ search command history backwords
- @@html:<kbd>Ctrl-x</kbd> <kbd>Ctrl-e</kbd>@@ to edit current line in
  ~$EDITOR~, when you have a really gnarly command

* Piping tips

~|&~ for easy ~2>&1 |~, e.g.

#+BEGIN_SRC bash
a_command 2>&1 | other_command
#+END_SRC

Could be written as

#+BEGIN_SRC bash
a_command |& other_command
#+END_SRC

(it pipes ~stdout~ and ~stderr~)

** ~tee~
~tee~ is a very useful tool.

Use it for capturing/logging a command's output but also printing it to ~stdout~
so you can follow along:

#+BEGIN_SRC bash
long_running_command | tee output.txt
#+END_SRC

Especially if you want to capture timing info as well like:

#+BEGIN_SRC bash
(time long_running_command) |& tee output.txt
#+END_SRC

Using ~|&~ as ~time~ prints to ~stderr~.

You can stick it in multiple places in a pipeline to record the data flowing
through it as it's transformed (for debugging or just better insight):

#+BEGIN_SRC bash
cat file | tee raw.txt | sort | tee sorted.txt | uniq | tee uniqed.txt
#+END_SRC

And for escalation permissions to ~sudo~ write a file:

#+BEGIN_SRC bash
cat file | sudo tee -a /privileged/file
#+END_SRC

As something like ~sudo cat file > /privileged/file~ doesn't work since the
~sudo~ applies to the ~cat~ (reading ~file~) not the redirect ~>~ (which writes
to ~/privileged/file~).
* Brace Expansion
This can be a real finger saver.

#+BEGIN_SRC bash
$ touch file-{1,2,3}.txt
# touch file-1.txt file-2.txt file-3.txt
#+END_SRC

#+BEGIN_SRC bash
$ touch file-{1..3}.txt
# touch file-1.txt file-2.txt file-3.txt
#+END_SRC

#+BEGIN_SRC bash
$ mv /some/very/long/path/file.txt{,.bak}
# mv /some/very/long/path/file.txt /some/very/long/path/file.txt.bak
#+END_SRC

#+BEGIN_SRC bash
$ mv ./env/{dev,prod}/config
# mv ./env/dev/config ./env/prod/config
#+END_SRC

#+BEGIN_SRC bash
$ mv ./env/{dev,prod}/config
# mv ./env/dev/config ./env/prod/config
#+END_SRC

#+BEGIN_SRC bash
$ echo {a,b,c}-{foo,bar,baz}
a-foo a-bar a-baz b-foo b-bar b-baz c-foo c-bar c-baz
#+END_SRC

#+BEGIN_SRC bash
$ echo {a{,1,2},b,c}-{foo,bar,baz}
a-foo a-bar a-baz a1-foo a1-bar a1-baz a2-foo a2-bar a2-baz b-foo b-bar b-baz c-foo c-bar c-baz
#+END_SRC

You can think of it like a small template, the shell will make a copy of the
string for each value in braces.

See the section on [[https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html][brace expansion]] in the bash manual for more.
* Scripts
These are things most applicable to writing shell scripts.

** ShellCheck
[[https://www.shellcheck.net/][ShellCheck]] is a linter for shell scripts. Very useful. I'd suggest having it
installed globally. [[https://github.com/koalaman/shellcheck#user-content-in-your-editor][Most editors]] support it directly.
** ~set~ flags

By default, shell scripts don't exit if a command errors. This is often
undesirable.

#+BEGIN_SRC bash
set -e
#+END_SRC

At the top of the script can help with that. There are [[https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html][many flags]] to explore.
Some common ones:
- ~-e~: causes the shell to exit immediately if a command returns a non-zero
  status, though it's [[http://mywiki.wooledge.org/BashFAQ/105][not perfect]]
- ~-x~: echo each command before it runs it, like ~make~, often useful in CI
  scripts when you want to be able to inspect what it's running
- ~-u~: treats unset variables and parameters as errors, so if you expect
  something to be set or require a positional argument by referencing say ~$2~,
  this helps avoid continuing running without it, [[http://mywiki.wooledge.org/BashFAQ/112][some discussion on it]]

A full loaded line like:

#+BEGIN_SRC bash
set -Eeuxo pipefail
#+END_SRC

This can be thought of like a [[http://redsymbol.net/articles/unofficial-bash-strict-mode/][strict mode]] for your script. Your scripts usually
need to be written from the start with those flags in mind to work at all.

An alternate and longer discussion of the flags can be found [[https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/][here]].
** Parameters/Arguments
You can reference script arguments positionally, ~$1~ for the first argument,
~$2~ for the second and so on[fn:: ~$0~ gives you the name of the script.].

~$@~ return an array of all arguments, very useful if you just need to pass all
the arguments through to another command, say if your script just does a little
setup and pre-flight checks. Use ~${@:2}~ to get all the arguments passed to the
script starting from the second one (so skipping the first one), ~${@:3}~ all
arguments starting from the third one, and so on. The general format is
~${parameter:offset:length}~ and the offset can be negative to grab from the end
of the array.

Use ~${parameter:-default}~ to set a default value, e.g., ~FOO=${FOO:-"foo"}~,
if ~$FOO~ exists, it will be used, otherwise ~$FOO~ will be set to
~"foo"~[fn::You can do this more compactly with ~${FOO:="foo"}~, the ~${x:=y}~
form sets ~x~ directly.]. Helpful if you have an optional argument to your
script:
#+BEGIN_SRC bash
set -u

FOO="$1"
BAR="${2:-bar}"

cat "$FOO" "$BAR"
#+END_SRC

Use ~${parameter#word}~ to remove ~word~ from the beginning of the value of
~parameter~:

#+BEGIN_SRC bash
$ FOO=foobar && echo ${FOO#foo}
bar
#+END_SRC

~${parameter%word}~ does the same for the end of a value:
#+BEGIN_SRC bash
$ FOO=foobar && echo ${FOO%bar}
foo
#+END_SRC

Use ~${parameter/pattern/string}~ to search-and-replace the value of ~parameter~:
#+BEGIN_SRC bash
$ FOO=foobar && echo ${FOO/bar/foo}
foofoo
#+END_SRC

These can be useful for quickly trimming or swapping extensions on file paths
and such.

See the section on [[https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion][parameter expansion]] in the bash manual for more.
** Shebang
The ~#!/bin/sh~[fn::Whitespace after the ~#!~ is optional. I tend to prefer a
space there.] is known as a [[https://en.wikipedia.org/wiki/Shebang_(Unix)][shebang]] line. It tells which executable should run
the script.

For greatest cross-system compatibility, you should almost always use the

#+BEGIN_SRC bash
#! /usr/bin/env <executable>
#+END_SRC

form. Not all systems place all executables in the same spot, but almost all
systems ensure ~/usr/bin/env~ exists to find the proper program.

For example

#+BEGIN_SRC bash
#! /usr/bin/env bash
#+END_SRC

instead of

#+BEGIN_SRC bash
#! /bin/bash
#+END_SRC

One limitation of this is that on Linux systems ~env~ takes everything after it
as a single argument to look up in the environment, meaning flags on the
executable don't work.

#+BEGIN_SRC bash
#! /usr/bin/env bash -e
#+END_SRC

Does not set the ~-e~ flag on the ~bash~ executable, ~env~ looks for an
executable with the literal name ~bash -x~ (which doesn't exist of course). Most
programs that supporting running as an interpreter support a way set these
things in the script itself. For the above example, you could have your script
start like:

#+BEGIN_SRC bash
#! /usr/bin/env bash

set -e
#+END_SRC

Some programs support an additional shebang or comment line with the config
options, such as ~nix-shell~:

#+BEGIN_SRC bash
#! /usr/bin/env nix-shell
#! nix-shell -i bash -p parallel -p flac

# do script things with parallel and flac tools present
#+END_SRC

This is a feature of the particular program, not a general feature, so you'll
know if you can do something like that.
** Functions
Functions are a thing in most shell languages and are great for the same reason
they are great in other languages.

Basic example:

#+BEGIN_SRC bash
do_thing() {
    local param=$1
    echo $param
}

do_thing "hello, world"
#+END_SRC


Notes:
- [[#parametersarguments][Parameters]] are handled just like a script, ~$1~, ~$2~, ~$@~, etc.
- Functions need to be defined in the file before they are executed.
- ~local~ scopes the variable to the function instead of the global space (as
  shell variables usually are) and should generally be preferred.
- Functions don't return values, the ~return~ statement exists and it sets the
  return status of the command (retrievable with ~$?~ after running the
  command), which is sometimes what you want. If you want to pass values out of
  a function either a) set a global variable or b) ~echo~ the value to stdout
  and capture the output when you call it (i.e., ~result=$(do_thing "hello")~).

If you have a few scripts that could share some functionality, you can define
functions in a separate file, say ~lib.sh~ and source it in your other scripts
~source lib.sh~[fn::Or ~. lib.sh~] making the functions available there.
* Scripting Scripts
Sometimes there are interactive programs (i.e., they prompt the user for input)
that you want to automate. The most basic situation being a command that prompts
for confirmation.

If there's only one prompt:

#+BEGIN_SRC bash
$ echo "yes" | command_that_prompts
#+END_SRC

works fine. If there are multiple prompts:

#+BEGIN_SRC bash
$ yes | command_that_prompts_multiple_times
#+END_SRC

~yes~ just repeatedly outputs the string ~y~, actually it just repeatedly
repeats whatever you string you pass in, defaulting to ~y~. So say you wanted to
say no to a bunch of prompts:

#+BEGIN_SRC bash
$ yes 'n' | command_you_say_no_to
#+END_SRC

When you have more complicated interactions, might want to reach for [[https://en.wikipedia.org/wiki/Expect][~expect~]].
* Record shell session
Sometimes you want to log some work you are doing in the terminal. The ~script~
command can help with that.

#+BEGIN_SRC bash
$ script my_log
#+END_SRC

Will start a subshell, recording all commands and output to the filename
specified (~my_log~ in this case). When you want to stop recording, just ~exit~
the shell (or @@html:<kbd>Ctrl-d</kbd>@@).
* ~true~ alternative
~:~ (a single colon) is equivalent to ~true~ in most shells. It can be useful as
a no-op on occasion or a quick way to ignore the failure of a command
(~command || :~), but often using ~true~ is more readable.
* More resources
- [[http://mywiki.wooledge.org/][Greg's Wiki]] is a great resource, includes a [[http://mywiki.wooledge.org/BashFAQ][Bash FAQ]], [[http://mywiki.wooledge.org/BashPitfalls][common
  pitfalls]], [[http://mywiki.wooledge.org/BashGuide][learning guide]], [[http://mywiki.wooledge.org/BashSheet][reference sheet]], and more.
- The [[https://wiki.bash-hackers.org/][Bash Hackers Wiki]] is a wonderful reference. Skim it, pick something you
  don't recognize and learn about it.
- The [[https://www.tldp.org/LDP/abs/html/][Advanced Bash-Scripting Guide]] can be a solid reference, I often find
  myself there, but it hasn't been updated in a while, and [[http://wooledge.org/~greybot/meta/abs][some discourage using
  it]] as it can show some outdated/unsafe/buggy approaches, so best to refer to
  once you know how to filter out the junk. But it covers a ton of advanced
  topics, *with examples*, and makes reasonable reference material.
- It depends on your system, but the man page for your shell will either be
  helpful or only something for masochists, try ~man bash~.
- https://github.com/jlevy/the-art-of-command-line
- https://awesome-shell.readthedocs.io/en/latest/README/