shell.org 15 KB
Newer Older
doshitan's avatar
doshitan committed
1
2
3
---
title: Shell
toc: true
doshitan's avatar
doshitan committed
4
toc-margin: true
5
6
published: 2019-04-16T11:23:06-05:00
modified: 2019-08-21T08:27:38-04:00
doshitan's avatar
doshitan committed
7
8
9
10
11
12
13
---

I primarily use [[https://www.zsh.org/][Zsh]], though often must write code suitable for Bash for
cross-system compatibility. I'll try to note when something is Zsh-specific.

My [[https://gitlab.com/doshitan/dots-zsh][config files]].

doshitan's avatar
doshitan committed
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
* explainshell.com
If you come across a command you don't understand[fn::One you wrote a while ago
more often than not.], plop it in https://explainshell.com/, hopefully it can
help break it down.
* TAB complete
A wonderful feature in almost every shell. Start typing a part of a command
name, file name, whatever, then press the @@html:<kbd>TAB</kbd>@@ key, the shell
will then auto-complete the rest for you or cycle through possibilities if there
isn't a unique option.

Will save your fingers.

Different shells come with different completion providers built in. Often there
is a bundle of extra ones available you can install, like
https://github.com/scop/bash-completion for bash.

And you can write your own too if needed or desired.
doshitan's avatar
doshitan committed
31
32
33
* Moving around
~cd~ (with no argument) moves to your home directory.

doshitan's avatar
doshitan committed
34
35
36
~cd -~ moves to your last location[fn::This is a pattern supported by some other
tools as well, like git, ~git checkout -~ will checkout the last branch you were
on, makes it easy to swap between branches.].
doshitan's avatar
doshitan committed
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

~pushd~ and ~popd~ create a directory stack which can be helpful when juggling
deeply nested paths or simply to store the current location, move around other
places and come back without having to explicitly store the original location
somewhere.

~z~ learns your most used locations and makes it quick to jump to them. It's
[[https://github.com/robbyrussell/oh-my-zsh/tree/master/plugins/z][bundled with oh-my-zsh]], but is [[https://github.com/rupa/z][independent]] and works in bash too. Say you have a
place ~~/some/cool/project/named/awesome~, after moving to it a few times, a
simple ~z awe~ would jump you directly there.
* Have command skip history
A space before a command excludes it from the shell history/getting logged.
Useful when you need to set some sensitive info for a command and don't want it
sticking around on your computer.

#+BEGIN_SRC bash
$  export MY_SECRET="a very secret string I don't want logged"
$ command -i $MY_SECRET
#+END_SRC

You have to check your shell is setup correctly or set the options in your shell
config, for [[https://unix.stackexchange.com/a/6104][zsh]] and for [[https://unix.stackexchange.com/a/32461][bash]].
* Last command
~!!~ expands to the last command, which is very useful in things like ~sudo !!~,
i.e., rerun the last command, but with sudo. You can also move farther back,
with things like ~!-2~ (~!!~ is equivalent to ~!-1~).

Another very useful application of ~!!~ is with search-and-replace ~:s~ / ~:gs~.
For example,

#+BEGIN_SRC bash
$ cp some_dir/with/some/file/foo other_dir/to/store/copy/foo-copy
$ !!:gs/foo/bar
# expands to
$ cp some_dir/with/some/file/bar other_dir/to/store/copy/bar-copy
#+END_SRC

For the common case of search-and-replace the last command you can use the
slightly shorter ~^foo^bar~ syntax, but ~!!:s//~ often comes to my mind first.
* man page supplement

~man~ pages are your friend, but sometimes they can either be too detailed or
too thin on examples. If that's the case, checking [[https://tldr.sh/][tldr.sh]] can maybe yield
clearer docs. There are a variety of CLI clients for it as well as the [[https://tldr.ostera.io/][web
client]].
* Keyboard navigation
Most shells use [[https://en.wikipedia.org/wiki/GNU_Readline][Readline]] to handle user movements which operates in ~emacs~ mode
by default. Often you can switch to a ~vi~ mode, but it's good to know some
basic movements.

- @@html:<kbd>Ctrl-a</kbd>@@ moves to beginning of line
- @@html:<kbd>Ctrl-e</kbd>@@ moves to end of line
- @@html:<kbd>Alt-b</kbd>@@ moves back one word
- @@html:<kbd>Alt-f</kbd>@@ moves forward one word
doshitan's avatar
doshitan committed
91
92
93
94
- @@html:<kbd>Ctrl-w</kbd>@@ delete back one word
- @@html:<kbd>Alt-d</kbd>@@ delete forward one word
- @@html:<kbd>Ctrl-u</kbd>@@ delete line backward
- @@html:<kbd>Ctrl-k</kbd>@@ delete line forward
doshitan's avatar
doshitan committed
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
- @@html:<kbd>Alt-.</kbd>@@ inserts the last argument to previous command
- @@html:<kbd>Ctrl-r</kbd>@@ search command history backwords
- @@html:<kbd>Ctrl-x</kbd> <kbd>Ctrl-e</kbd>@@ to edit current line in
  ~$EDITOR~, when you have a really gnarly command

* Piping tips

~|&~ for easy ~2>&1 |~, e.g.

#+BEGIN_SRC bash
a_command 2>&1 | other_command
#+END_SRC

Could be written as

#+BEGIN_SRC bash
a_command |& other_command
#+END_SRC

(it pipes ~stdout~ and ~stderr~)

** ~tee~
~tee~ is a very useful tool.

Use it for capturing/logging a command's output but also printing it to ~stdout~
so you can follow along:

#+BEGIN_SRC bash
long_running_command | tee output.txt
#+END_SRC

Especially if you want to capture timing info as well like:

#+BEGIN_SRC bash
(time long_running_command) |& tee output.txt
#+END_SRC

Using ~|&~ as ~time~ prints to ~stderr~.

You can stick it in multiple places in a pipeline to record the data flowing
through it as it's transformed (for debugging or just better insight):

#+BEGIN_SRC bash
cat file | tee raw.txt | sort | tee sorted.txt | uniq | tee uniqed.txt
#+END_SRC

And for escalation permissions to ~sudo~ write a file:

#+BEGIN_SRC bash
cat file | sudo tee -a /privileged/file
#+END_SRC

As something like ~sudo cat file > /privileged/file~ doesn't work since the
~sudo~ applies to the ~cat~ (reading ~file~) not the redirect ~>~ (which writes
to ~/privileged/file~).
* Brace Expansion
This can be a real finger saver.

#+BEGIN_SRC bash
$ touch file-{1,2,3}.txt
# touch file-1.txt file-2.txt file-3.txt
#+END_SRC

#+BEGIN_SRC bash
$ touch file-{1..3}.txt
# touch file-1.txt file-2.txt file-3.txt
#+END_SRC

#+BEGIN_SRC bash
$ mv /some/very/long/path/file.txt{,.bak}
# mv /some/very/long/path/file.txt /some/very/long/path/file.txt.bak
#+END_SRC

#+BEGIN_SRC bash
$ mv ./env/{dev,prod}/config
# mv ./env/dev/config ./env/prod/config
#+END_SRC

#+BEGIN_SRC bash
$ mv ./env/{dev,prod}/config
# mv ./env/dev/config ./env/prod/config
#+END_SRC

#+BEGIN_SRC bash
$ echo {a,b,c}-{foo,bar,baz}
a-foo a-bar a-baz b-foo b-bar b-baz c-foo c-bar c-baz
#+END_SRC

#+BEGIN_SRC bash
$ echo {a{,1,2},b,c}-{foo,bar,baz}
a-foo a-bar a-baz a1-foo a1-bar a1-baz a2-foo a2-bar a2-baz b-foo b-bar b-baz c-foo c-bar c-baz
#+END_SRC

You can think of it like a small template, the shell will make a copy of the
string for each value in braces.

See the section on [[https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html][brace expansion]] in the bash manual for more.
* Scripts
These are things most applicable to writing shell scripts.

** ShellCheck
[[https://www.shellcheck.net/][ShellCheck]] is a linter for shell scripts. Very useful. I'd suggest having it
installed globally. [[https://github.com/koalaman/shellcheck#user-content-in-your-editor][Most editors]] support it directly.
** ~set~ flags

By default, shell scripts don't exit if a command errors. This is often
undesirable.

#+BEGIN_SRC bash
set -e
#+END_SRC

At the top of the script can help with that. There are [[https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html][many flags]] to explore.
Some common ones:
- ~-e~: causes the shell to exit immediately if a command returns a non-zero
doshitan's avatar
doshitan committed
210
  status, though it's [[http://mywiki.wooledge.org/BashFAQ/105][not perfect]]
doshitan's avatar
doshitan committed
211
212
213
214
- ~-x~: echo each command before it runs it, like ~make~, often useful in CI
  scripts when you want to be able to inspect what it's running
- ~-u~: treats unset variables and parameters as errors, so if you expect
  something to be set or require a positional argument by referencing say ~$2~,
doshitan's avatar
doshitan committed
215
  this helps avoid continuing running without it, [[http://mywiki.wooledge.org/BashFAQ/112][some discussion on it]]
doshitan's avatar
doshitan committed
216
217
218
219
220
221
222

A full loaded line like:

#+BEGIN_SRC bash
set -Eeuxo pipefail
#+END_SRC

doshitan's avatar
doshitan committed
223
224
225
226
This can be thought of like a [[http://redsymbol.net/articles/unofficial-bash-strict-mode/][strict mode]] for your script. Your scripts usually
need to be written from the start with those flags in mind to work at all.

An alternate and longer discussion of the flags can be found [[https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/][here]].
doshitan's avatar
doshitan committed
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
** Parameters/Arguments
You can reference script arguments positionally, ~$1~ for the first argument,
~$2~ for the second and so on[fn:: ~$0~ gives you the name of the script.].

~$@~ return an array of all arguments, very useful if you just need to pass all
the arguments through to another command, say if your script just does a little
setup and pre-flight checks. Use ~${@:2}~ to get all the arguments passed to the
script starting from the second one (so skipping the first one), ~${@:3}~ all
arguments starting from the third one, and so on. The general format is
~${parameter:offset:length}~ and the offset can be negative to grab from the end
of the array.

Use ~${parameter:-default}~ to set a default value, e.g., ~FOO=${FOO:-"foo"}~,
if ~$FOO~ exists, it will be used, otherwise ~$FOO~ will be set to
~"foo"~[fn::You can do this more compactly with ~${FOO:="foo"}~, the ~${x:=y}~
form sets ~x~ directly.]. Helpful if you have an optional argument to your
script:
#+BEGIN_SRC bash
set -u

FOO="$1"
BAR="${2:-bar}"

cat "$FOO" "$BAR"
#+END_SRC

Use ~${parameter#word}~ to remove ~word~ from the beginning of the value of
~parameter~:

#+BEGIN_SRC bash
$ FOO=foobar && echo ${FOO#foo}
bar
#+END_SRC

~${parameter%word}~ does the same for the end of a value:
#+BEGIN_SRC bash
$ FOO=foobar && echo ${FOO%bar}
foo
#+END_SRC

Use ~${parameter/pattern/string}~ to search-and-replace the value of ~parameter~:
#+BEGIN_SRC bash
$ FOO=foobar && echo ${FOO/bar/foo}
foofoo
#+END_SRC

These can be useful for quickly trimming or swapping extensions on file paths
and such.

See the section on [[https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion][parameter expansion]] in the bash manual for more.
** Shebang
The ~#!/bin/sh~[fn::Whitespace after the ~#!~ is optional. I tend to prefer a
space there.] is known as a [[https://en.wikipedia.org/wiki/Shebang_(Unix)][shebang]] line. It tells which executable should run
the script.

For greatest cross-system compatibility, you should almost always use the

#+BEGIN_SRC bash
#! /usr/bin/env <executable>
#+END_SRC

form. Not all systems place all executables in the same spot, but almost all
systems ensure ~/usr/bin/env~ exists to find the proper program.

For example

#+BEGIN_SRC bash
#! /usr/bin/env bash
#+END_SRC

instead of

#+BEGIN_SRC bash
#! /bin/bash
#+END_SRC

One limitation of this is that on Linux systems ~env~ takes everything after it
as a single argument to look up in the environment, meaning flags on the
executable don't work.

#+BEGIN_SRC bash
#! /usr/bin/env bash -e
#+END_SRC

Does not set the ~-e~ flag on the ~bash~ executable, ~env~ looks for an
executable with the literal name ~bash -x~ (which doesn't exist of course). Most
programs that supporting running as an interpreter support a way set these
things in the script itself. For the above example, you could have your script
start like:

#+BEGIN_SRC bash
#! /usr/bin/env bash

set -e
#+END_SRC

Some programs support an additional shebang or comment line with the config
options, such as ~nix-shell~:

#+BEGIN_SRC bash
#! /usr/bin/env nix-shell
#! nix-shell -i bash -p parallel -p flac

# do script things with parallel and flac tools present
#+END_SRC

This is a feature of the particular program, not a general feature, so you'll
know if you can do something like that.
doshitan's avatar
doshitan committed
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
** Functions
Functions are a thing in most shell languages and are great for the same reason
they are great in other languages.

Basic example:

#+BEGIN_SRC bash
do_thing() {
    local param=$1
    echo $param
}

do_thing "hello, world"
#+END_SRC


Notes:
- [[#parametersarguments][Parameters]] are handled just like a script, ~$1~, ~$2~, ~$@~, etc.
- Functions need to be defined in the file before they are executed.
- ~local~ scopes the variable to the function instead of the global space (as
  shell variables usually are) and should generally be preferred.
- Functions don't return values, the ~return~ statement exists and it sets the
  return status of the command (retrievable with ~$?~ after running the
  command), which is sometimes what you want. If you want to pass values out of
  a function either a) set a global variable or b) ~echo~ the value to stdout
  and capture the output when you call it (i.e., ~result=$(do_thing "hello")~).

If you have a few scripts that could share some functionality, you can define
functions in a separate file, say ~lib.sh~ and source it in your other scripts
~source lib.sh~[fn::Or ~. lib.sh~] making the functions available there.
* Scripting Scripts
Sometimes there are interactive programs (i.e., they prompt the user for input)
that you want to automate. The most basic situation being a command that prompts
for confirmation.

If there's only one prompt:

#+BEGIN_SRC bash
$ echo "yes" | command_that_prompts
#+END_SRC

works fine. If there are multiple prompts:

#+BEGIN_SRC bash
$ yes | command_that_prompts_multiple_times
#+END_SRC

~yes~ just repeatedly outputs the string ~y~, actually it just repeatedly
repeats whatever you string you pass in, defaulting to ~y~. So say you wanted to
say no to a bunch of prompts:

#+BEGIN_SRC bash
$ yes 'n' | command_you_say_no_to
#+END_SRC

When you have more complicated interactions, might want to reach for [[https://en.wikipedia.org/wiki/Expect][~expect~]].
* Record shell session
Sometimes you want to log some work you are doing in the terminal. The ~script~
command can help with that.

#+BEGIN_SRC bash
$ script my_log
#+END_SRC

Will start a subshell, recording all commands and output to the filename
specified (~my_log~ in this case). When you want to stop recording, just ~exit~
the shell (or @@html:<kbd>Ctrl-d</kbd>@@).
* ~true~ alternative
~:~ (a single colon) is equivalent to ~true~ in most shells. It can be useful as
a no-op on occasion or a quick way to ignore the failure of a command
(~command || :~), but often using ~true~ is more readable.
* More resources
doshitan's avatar
doshitan committed
407
408
- [[http://mywiki.wooledge.org/][Greg's Wiki]] is a great resource, includes a [[http://mywiki.wooledge.org/BashFAQ][Bash FAQ]], [[http://mywiki.wooledge.org/BashPitfalls][common
  pitfalls]], [[http://mywiki.wooledge.org/BashGuide][learning guide]], [[http://mywiki.wooledge.org/BashSheet][reference sheet]], and more.
doshitan's avatar
doshitan committed
409
410
- The [[https://wiki.bash-hackers.org/][Bash Hackers Wiki]] is a wonderful reference. Skim it, pick something you
  don't recognize and learn about it.
doshitan's avatar
doshitan committed
411
412
413
414
415
- The [[https://www.tldp.org/LDP/abs/html/][Advanced Bash-Scripting Guide]] can be a solid reference, I often find
  myself there, but it hasn't been updated in a while, and [[http://wooledge.org/~greybot/meta/abs][some discourage using
  it]] as it can show some outdated/unsafe/buggy approaches, so best to refer to
  once you know how to filter out the junk. But it covers a ton of advanced
  topics, *with examples*, and makes reasonable reference material.
doshitan's avatar
doshitan committed
416
417
418
- It depends on your system, but the man page for your shell will either be
  helpful or only something for masochists, try ~man bash~.
- https://github.com/jlevy/the-art-of-command-line
doshitan's avatar
doshitan committed
419
- https://awesome-shell.readthedocs.io/en/latest/README/