shell.org 19.3 KB
Newer Older
doshitan's avatar
doshitan committed
1
2
3
---
title: Shell
toc: true
doshitan's avatar
doshitan committed
4
toc-margin: true
5
published: 2019-04-16T11:23:06-05:00
6
modified: 2021-03-11T09:37:41-0500
doshitan's avatar
doshitan committed
7
8
9
10
11
12
13
---

I primarily use [[https://www.zsh.org/][Zsh]], though often must write code suitable for Bash for
cross-system compatibility. I'll try to note when something is Zsh-specific.

My [[https://gitlab.com/doshitan/dots-zsh][config files]].

doshitan's avatar
doshitan committed
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
* explainshell.com
If you come across a command you don't understand[fn::One you wrote a while ago
more often than not.], plop it in https://explainshell.com/, hopefully it can
help break it down.
* TAB complete
A wonderful feature in almost every shell. Start typing a part of a command
name, file name, whatever, then press the @@html:<kbd>TAB</kbd>@@ key, the shell
will then auto-complete the rest for you or cycle through possibilities if there
isn't a unique option.

Will save your fingers.

Different shells come with different completion providers built in. Often there
is a bundle of extra ones available you can install, like
https://github.com/scop/bash-completion for bash.

And you can write your own too if needed or desired.
doshitan's avatar
doshitan committed
31
32
33
* Moving around
~cd~ (with no argument) moves to your home directory.

doshitan's avatar
doshitan committed
34
35
36
~cd -~ moves to your last location[fn::This is a pattern supported by some other
tools as well, like git, ~git checkout -~ will checkout the last branch you were
on, makes it easy to swap between branches.].
doshitan's avatar
doshitan committed
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

~pushd~ and ~popd~ create a directory stack which can be helpful when juggling
deeply nested paths or simply to store the current location, move around other
places and come back without having to explicitly store the original location
somewhere.

~z~ learns your most used locations and makes it quick to jump to them. It's
[[https://github.com/robbyrussell/oh-my-zsh/tree/master/plugins/z][bundled with oh-my-zsh]], but is [[https://github.com/rupa/z][independent]] and works in bash too. Say you have a
place ~~/some/cool/project/named/awesome~, after moving to it a few times, a
simple ~z awe~ would jump you directly there.
* Have command skip history
A space before a command excludes it from the shell history/getting logged.
Useful when you need to set some sensitive info for a command and don't want it
sticking around on your computer.

#+BEGIN_SRC bash
$  export MY_SECRET="a very secret string I don't want logged"
$ command -i $MY_SECRET
#+END_SRC

You have to check your shell is setup correctly or set the options in your shell
config, for [[https://unix.stackexchange.com/a/6104][zsh]] and for [[https://unix.stackexchange.com/a/32461][bash]].
* Last command
~!!~ expands to the last command, which is very useful in things like ~sudo !!~,
i.e., rerun the last command, but with sudo. You can also move farther back,
with things like ~!-2~ (~!!~ is equivalent to ~!-1~).

Another very useful application of ~!!~ is with search-and-replace ~:s~ / ~:gs~.
For example,

#+BEGIN_SRC bash
$ cp some_dir/with/some/file/foo other_dir/to/store/copy/foo-copy
$ !!:gs/foo/bar
# expands to
$ cp some_dir/with/some/file/bar other_dir/to/store/copy/bar-copy
#+END_SRC

For the common case of search-and-replace the last command you can use the
slightly shorter ~^foo^bar~ syntax, but ~!!:s//~ often comes to my mind first.
* man page supplement

~man~ pages are your friend, but sometimes they can either be too detailed or
too thin on examples. If that's the case, checking [[https://tldr.sh/][tldr.sh]] can maybe yield
clearer docs. There are a variety of CLI clients for it as well as the [[https://tldr.ostera.io/][web
client]].
* Keyboard navigation
Most shells use [[https://en.wikipedia.org/wiki/GNU_Readline][Readline]] to handle user movements which operates in ~emacs~ mode
by default. Often you can switch to a ~vi~ mode, but it's good to know some
basic movements.

- @@html:<kbd>Ctrl-a</kbd>@@ moves to beginning of line
- @@html:<kbd>Ctrl-e</kbd>@@ moves to end of line
- @@html:<kbd>Alt-b</kbd>@@ moves back one word
- @@html:<kbd>Alt-f</kbd>@@ moves forward one word
doshitan's avatar
doshitan committed
91
92
93
94
- @@html:<kbd>Ctrl-w</kbd>@@ delete back one word
- @@html:<kbd>Alt-d</kbd>@@ delete forward one word
- @@html:<kbd>Ctrl-u</kbd>@@ delete line backward
- @@html:<kbd>Ctrl-k</kbd>@@ delete line forward
doshitan's avatar
doshitan committed
95
96
97
98
99
100
- @@html:<kbd>Alt-.</kbd>@@ inserts the last argument to previous command
- @@html:<kbd>Ctrl-r</kbd>@@ search command history backwords
- @@html:<kbd>Ctrl-x</kbd> <kbd>Ctrl-e</kbd>@@ to edit current line in
  ~$EDITOR~, when you have a really gnarly command

* Piping tips
doshitan's avatar
doshitan committed
101
** Handling both stdout and stderr
doshitan's avatar
doshitan committed
102
103
104
105
106
107
108
109
110
111
112
113
114
115
~|&~ for easy ~2>&1 |~, e.g.

#+BEGIN_SRC bash
a_command 2>&1 | other_command
#+END_SRC

Could be written as

#+BEGIN_SRC bash
a_command |& other_command
#+END_SRC

(it pipes ~stdout~ and ~stderr~)

doshitan's avatar
doshitan committed
116
Similarly ~&>~ for redirection, say ~&> /dev/null~ instead of ~>/dev/null 2>&1~.
doshitan's avatar
doshitan committed
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
** ~tee~
~tee~ is a very useful tool.

Use it for capturing/logging a command's output but also printing it to ~stdout~
so you can follow along:

#+BEGIN_SRC bash
long_running_command | tee output.txt
#+END_SRC

Especially if you want to capture timing info as well like:

#+BEGIN_SRC bash
(time long_running_command) |& tee output.txt
#+END_SRC

Using ~|&~ as ~time~ prints to ~stderr~.

You can stick it in multiple places in a pipeline to record the data flowing
through it as it's transformed (for debugging or just better insight):

#+BEGIN_SRC bash
cat file | tee raw.txt | sort | tee sorted.txt | uniq | tee uniqed.txt
#+END_SRC

And for escalation permissions to ~sudo~ write a file:

#+BEGIN_SRC bash
cat file | sudo tee -a /privileged/file
#+END_SRC

As something like ~sudo cat file > /privileged/file~ doesn't work since the
~sudo~ applies to the ~cat~ (reading ~file~) not the redirect ~>~ (which writes
to ~/privileged/file~).
doshitan's avatar
doshitan committed
151
152
153
154
155
156
157
158
159
160
161
162
163
** Grouping commands
~()~ or ~{}~ can be used to [[https://www.gnu.org/software/bash/manual/bash.html#Command-Grouping][group commands to run as one unit]], either in
subshell or not (respectively).

Variety of uses for this, but say in the middle of some pipeline, you want to
prepend or append some content:

#+begin_src bash
$ echo "bar" | rev | (echo "foo"; cat; echo "baz") | rev
oof
bar
zab
#+end_src
doshitan's avatar
doshitan committed
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
* Brace Expansion
This can be a real finger saver.

#+BEGIN_SRC bash
$ touch file-{1,2,3}.txt
# touch file-1.txt file-2.txt file-3.txt
#+END_SRC

#+BEGIN_SRC bash
$ touch file-{1..3}.txt
# touch file-1.txt file-2.txt file-3.txt
#+END_SRC

#+BEGIN_SRC bash
$ mv /some/very/long/path/file.txt{,.bak}
# mv /some/very/long/path/file.txt /some/very/long/path/file.txt.bak
#+END_SRC

#+BEGIN_SRC bash
$ mv ./env/{dev,prod}/config
# mv ./env/dev/config ./env/prod/config
#+END_SRC

#+BEGIN_SRC bash
$ echo {a,b,c}-{foo,bar,baz}
a-foo a-bar a-baz b-foo b-bar b-baz c-foo c-bar c-baz
#+END_SRC

#+BEGIN_SRC bash
$ echo {a{,1,2},b,c}-{foo,bar,baz}
a-foo a-bar a-baz a1-foo a1-bar a1-baz a2-foo a2-bar a2-baz b-foo b-bar b-baz c-foo c-bar c-baz
#+END_SRC

You can think of it like a small template, the shell will make a copy of the
string for each value in braces.

See the section on [[https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html][brace expansion]] in the bash manual for more.
* Scripts
These are things most applicable to writing shell scripts.

** ShellCheck
[[https://www.shellcheck.net/][ShellCheck]] is a linter for shell scripts. Very useful. I'd suggest having it
installed globally. [[https://github.com/koalaman/shellcheck#user-content-in-your-editor][Most editors]] support it directly.
** ~set~ flags

By default, shell scripts don't exit if a command errors. This is often
undesirable.

#+BEGIN_SRC bash
set -e
#+END_SRC

At the top of the script can help with that. There are [[https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html][many flags]] to explore.
Some common ones:
- ~-e~: causes the shell to exit immediately if a command returns a non-zero
doshitan's avatar
doshitan committed
219
  status, though it's [[http://mywiki.wooledge.org/BashFAQ/105][not perfect]]
doshitan's avatar
doshitan committed
220
221
222
223
- ~-x~: echo each command before it runs it, like ~make~, often useful in CI
  scripts when you want to be able to inspect what it's running
- ~-u~: treats unset variables and parameters as errors, so if you expect
  something to be set or require a positional argument by referencing say ~$2~,
doshitan's avatar
doshitan committed
224
  this helps avoid continuing running without it, [[http://mywiki.wooledge.org/BashFAQ/112][some discussion on it]]
doshitan's avatar
doshitan committed
225
226
227
228
229
230
231

A full loaded line like:

#+BEGIN_SRC bash
set -Eeuxo pipefail
#+END_SRC

doshitan's avatar
doshitan committed
232
233
234
235
This can be thought of like a [[http://redsymbol.net/articles/unofficial-bash-strict-mode/][strict mode]] for your script. Your scripts usually
need to be written from the start with those flags in mind to work at all.

An alternate and longer discussion of the flags can be found [[https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/][here]].
doshitan's avatar
doshitan committed
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
** Parameters/Arguments
You can reference script arguments positionally, ~$1~ for the first argument,
~$2~ for the second and so on[fn:: ~$0~ gives you the name of the script.].

~$@~ return an array of all arguments, very useful if you just need to pass all
the arguments through to another command, say if your script just does a little
setup and pre-flight checks. Use ~${@:2}~ to get all the arguments passed to the
script starting from the second one (so skipping the first one), ~${@:3}~ all
arguments starting from the third one, and so on. The general format is
~${parameter:offset:length}~ and the offset can be negative to grab from the end
of the array.

Use ~${parameter:-default}~ to set a default value, e.g., ~FOO=${FOO:-"foo"}~,
if ~$FOO~ exists, it will be used, otherwise ~$FOO~ will be set to
~"foo"~[fn::You can do this more compactly with ~${FOO:="foo"}~, the ~${x:=y}~
form sets ~x~ directly.]. Helpful if you have an optional argument to your
script:
#+BEGIN_SRC bash
set -u

FOO="$1"
BAR="${2:-bar}"

cat "$FOO" "$BAR"
#+END_SRC

Use ~${parameter#word}~ to remove ~word~ from the beginning of the value of
~parameter~:

#+BEGIN_SRC bash
$ FOO=foobar && echo ${FOO#foo}
bar
#+END_SRC

~${parameter%word}~ does the same for the end of a value:
#+BEGIN_SRC bash
$ FOO=foobar && echo ${FOO%bar}
foo
#+END_SRC

Use ~${parameter/pattern/string}~ to search-and-replace the value of ~parameter~:
#+BEGIN_SRC bash
$ FOO=foobar && echo ${FOO/bar/foo}
foofoo
#+END_SRC

These can be useful for quickly trimming or swapping extensions on file paths
and such.

See the section on [[https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion][parameter expansion]] in the bash manual for more.
** Shebang
The ~#!/bin/sh~[fn::Whitespace after the ~#!~ is optional. I tend to prefer a
space there.] is known as a [[https://en.wikipedia.org/wiki/Shebang_(Unix)][shebang]] line. It tells which executable should run
the script.

For greatest cross-system compatibility, you should almost always use the

#+BEGIN_SRC bash
#! /usr/bin/env <executable>
#+END_SRC

form. Not all systems place all executables in the same spot, but almost all
systems ensure ~/usr/bin/env~ exists to find the proper program.

For example

#+BEGIN_SRC bash
#! /usr/bin/env bash
#+END_SRC

instead of

#+BEGIN_SRC bash
#! /bin/bash
#+END_SRC

One limitation of this is that on Linux systems ~env~ takes everything after it
as a single argument to look up in the environment, meaning flags on the
executable don't work.

#+BEGIN_SRC bash
#! /usr/bin/env bash -e
#+END_SRC

Does not set the ~-e~ flag on the ~bash~ executable, ~env~ looks for an
doshitan's avatar
doshitan committed
321
executable with the literal name ~bash -e~ (which doesn't exist of course). Most
doshitan's avatar
doshitan committed
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
programs that supporting running as an interpreter support a way set these
things in the script itself. For the above example, you could have your script
start like:

#+BEGIN_SRC bash
#! /usr/bin/env bash

set -e
#+END_SRC

Some programs support an additional shebang or comment line with the config
options, such as ~nix-shell~:

#+BEGIN_SRC bash
#! /usr/bin/env nix-shell
#! nix-shell -i bash -p parallel -p flac

# do script things with parallel and flac tools present
#+END_SRC

This is a feature of the particular program, not a general feature, so you'll
know if you can do something like that.
doshitan's avatar
doshitan committed
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
** Functions
Functions are a thing in most shell languages and are great for the same reason
they are great in other languages.

Basic example:

#+BEGIN_SRC bash
do_thing() {
    local param=$1
    echo $param
}

do_thing "hello, world"
#+END_SRC


Notes:
- [[#parametersarguments][Parameters]] are handled just like a script, ~$1~, ~$2~, ~$@~, etc.
- Functions need to be defined in the file before they are executed.
- ~local~ scopes the variable to the function instead of the global space (as
  shell variables usually are) and should generally be preferred.
- Functions don't return values, the ~return~ statement exists and it sets the
  return status of the command (retrievable with ~$?~ after running the
  command), which is sometimes what you want. If you want to pass values out of
  a function either a) set a global variable or b) ~echo~ the value to stdout
  and capture the output when you call it (i.e., ~result=$(do_thing "hello")~).

If you have a few scripts that could share some functionality, you can define
functions in a separate file, say ~lib.sh~ and source it in your other scripts
doshitan's avatar
doshitan committed
373
374
375
~source lib.sh~[fn:dot-source] making the functions available there.

[fn:dot-source] Can use ~.~ in as a shorthand for ~source~, like: ~. lib.sh~
doshitan's avatar
doshitan committed
376
* Scripting Scripts
doshitan's avatar
doshitan committed
377
378
379
Sometimes there are interactive programs (i.e., they prompt for input) that you
want to automate. The most basic situation being a command that prompts for
confirmation.
doshitan's avatar
doshitan committed
380
381
382
383
384
385
386
387
388
389
390
391
392

If there's only one prompt:

#+BEGIN_SRC bash
$ echo "yes" | command_that_prompts
#+END_SRC

works fine. If there are multiple prompts:

#+BEGIN_SRC bash
$ yes | command_that_prompts_multiple_times
#+END_SRC

doshitan's avatar
doshitan committed
393
394
Functionally ~yes~ just repeatedly outputs a provided string, defaulting to ~y~.
So say you wanted to say no to a bunch of prompts:
doshitan's avatar
doshitan committed
395
396

#+BEGIN_SRC bash
doshitan's avatar
doshitan committed
397
$ yes 'no' | command_you_say_no_to
doshitan's avatar
doshitan committed
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
#+END_SRC

When you have more complicated interactions, might want to reach for [[https://en.wikipedia.org/wiki/Expect][~expect~]].
* Record shell session
Sometimes you want to log some work you are doing in the terminal. The ~script~
command can help with that.

#+BEGIN_SRC bash
$ script my_log
#+END_SRC

Will start a subshell, recording all commands and output to the filename
specified (~my_log~ in this case). When you want to stop recording, just ~exit~
the shell (or @@html:<kbd>Ctrl-d</kbd>@@).
* ~true~ alternative
~:~ (a single colon) is equivalent to ~true~ in most shells. It can be useful as
a no-op on occasion or a quick way to ignore the failure of a command
(~command || :~), but often using ~true~ is more readable.
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
* Repeatedly run command
For the times you need to run a command repeatedly to monitor some thing,
there's ~watch~.

#+begin_src bash
watch -n1 df -h
#+end_src

Runs ~df -h~ every second, showing the latest output.

Just being able to poll something is useful enough, but some additional flags
can be very convenient:
- ~-b/--beep~ beep when command has a non-zero exit
- ~-d/--differences~ highlight changes between updates
* Notify when command finishes
Can beep with ~printf \\a~[fn::Or equivalently ~echo -en "\007"~ or ~echo -en
\\a~ ].

#+begin_src bash
sleep 5; printf \\a
#+end_src

When you want more than a beep, on Linux, ~espeak~ (likely provided by an
~espeak-ng~ package) or ~spd-say~ (default in recent Ubuntus) will play sound
via text-to-speech[fn::There are a good number of TTS engines for Linux, this
[[https://askubuntu.com/a/501917][StackOverflow question discusses some]].]:

#+begin_src bash
sleep 5; espeak "Fly, you fools!"
#+end_src

When you don't want sound, on Linux, display a notification with ~notify-send~
(provided by ~libnotify~):
#+begin_src bash
sleep 5; notify-send "Tea is ready."
#+end_src
doshitan's avatar
doshitan committed
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
* Paste-able markup for WYSIWYGs

Quite often there may be some plain-text markup or data that you want to get
into a WYSIWYG environment, for various reasons.

[[https://pandoc.org/][pandoc]] is a super useful tool for this.

Different systems have different clipboard tools with different
behavior[fn::Under Linux there are generally two "selections" available. The
"primary" selection is automatically filled with the last text selected, and is
paste-able usually with the middle-click of a mouse or Shift+Ins. The
"clipboard" selection is what Ctrl-c/Ctrl-v use.], but will try to present
examples that are all functionally equivalent.

For an existing file you want to copy:

Linux (X11):
#+begin_src bash
pandoc a_file.md | xclip -sel c -t text/html
#+end_src

Linux (Wayland):
#+begin_src bash
pandoc a_file.md | wl-copy -t text/html
#+end_src

macOS:
#+begin_src bash
pandoc -t rtf a_file.md | pbcopy
#+end_src

Or say you have some content you want to transform already selected/in the clipboard:

Linux (X11):
#+begin_src bash
xclip -sel c -o | pandoc -f markdown -t html | xclip -sel c -t text/html
#+end_src

Linux (Wayland):
#+begin_src bash
wl-paste | pandoc -f markdown -t html | wl-copy -t text/html
#+end_src

macOS:
#+begin_src
pbpaste | pandoc -f markdown -t rtf | pbcopy
#+end_src

This is also handy for tables. Say you have some JSON data you want to dump into
a document/wiki formatted as a table or into a spreadsheet:

#+begin_src bash
echo '[{"foo": 1, "bar": "yes"},{"foo": 2, "bar": "no"}]' | jq -r '.[] | [.foo, .bar] | @tsv' | (echo "Number\tYes/No" && echo "---\t---" && cat) | column -t -s "$(printf '\t')" | pandoc -t html | xclip -sel c -t text/html
#+end_src

These examples all use markdown as the input format, but pandoc supports a large
number of formats depending on your needs (~pandoc --list-input-formats~).
* Alternate sed delimiters

The standard/typical delimiter is ~/~, which can complicate things when wanting
to rename file paths in particular.

Say you wanted to replace ~foo/bar~ instances with ~bar/baz~, using ~/~ as a
delimiter is a bit messy, having to escape the slashes in the path:

#+begin_src bash
sed 's/foo\/bar/bar\/baz/'
#+end_src

But conveniently any single-byte character can be the delimiter, say ~|~:

#+begin_src bash
sed 's|foo/bar|bar/baz|'
#+end_src

Or ~;~:
#+begin_src bash
sed 's;foo/bar;bar/baz;'
#+end_src

And so on. So simplify your life, choose a delimiter that limits the amount of
escaping you need to do.

This also applies to "patterns" for sed in general (though if there is no
leading command character, do need to escape the initial non-slash delimiter,
e.g., ~sed '\;delete/me;d~ vs ~sed '/delete\/me/d'~).
538

doshitan's avatar
doshitan committed
539
(a variety of other tools that use a similar syntax also support this, but YMMV)
doshitan's avatar
doshitan committed
540
* More resources
doshitan's avatar
doshitan committed
541
542
- [[http://mywiki.wooledge.org/][Greg's Wiki]] is a great resource, includes a [[http://mywiki.wooledge.org/BashFAQ][Bash FAQ]], [[http://mywiki.wooledge.org/BashPitfalls][common
  pitfalls]], [[http://mywiki.wooledge.org/BashGuide][learning guide]], [[http://mywiki.wooledge.org/BashSheet][reference sheet]], and more.
doshitan's avatar
doshitan committed
543
544
- The [[https://wiki.bash-hackers.org/][Bash Hackers Wiki]] is a wonderful reference. Skim it, pick something you
  don't recognize and learn about it.
doshitan's avatar
doshitan committed
545
546
547
548
- The [[https://www.tldp.org/LDP/abs/html/][Advanced Bash-Scripting Guide]] can be a solid reference, I often find
  myself there, but it hasn't been updated in a while, and [[http://wooledge.org/~greybot/meta/abs][some discourage using
  it]] as it can show some outdated/unsafe/buggy approaches, so best to refer to
  once you know how to filter out the junk. But it covers a ton of advanced
549
  topics, /with examples/, and makes reasonable reference material.
doshitan's avatar
doshitan committed
550
551
552
- It depends on your system, but the man page for your shell will either be
  helpful or only something for masochists, try ~man bash~.
- https://github.com/jlevy/the-art-of-command-line
doshitan's avatar
doshitan committed
553
- https://awesome-shell.readthedocs.io/en/latest/README/
doshitan's avatar
doshitan committed
554
- https://shellhaters.org/