logo

drewdevault.com

[mirror] blog and personal website of Drew DeVault git clone https://hacktivis.me/git/mirror/drewdevault.com.git

Shell-literacy.md (6749B)


  1. ---
  2. date: 2020-12-12
  3. title: Become shell literate
  4. ---
  5. Shell literacy is one of the most important skills you ought to possess as a
  6. programmer. The Unix shell is one of the most powerful ideas ever put to code,
  7. and should be second nature to you as a programmer. No other tool is nearly as
  8. effective at commanding your computer to perform complex tasks quickly —
  9. or at storing them as scripts you can use later.
  10. In my workflow, I use Vim as my editor, and Unix as my "IDE". I don't trick out
  11. [my vimrc](https://git.sr.ht/~sircmpwn/dotfiles/tree/master/.vimrc) to add a
  12. bunch of IDE-like features — the most substantial plugin I use on a daily
  13. basis is [Ctrl+P](https://github.com/ctrlpvim/ctrlp.vim), and that just makes it
  14. easier to open files. Being Vim literate is a valuable skill, but an important
  15. detail is knowing when to drop it. My daily workflow involves several open
  16. terminals, generally one with Vim, another to run builds or daemons, and a third
  17. which just keeps a shell handy for anything I might ask of it.
  18. [![Screenshot of my workspace](https://redacted.moe/f/1bbaf26c.png)](https://redacted.moe/f/1bbaf26c.png)
  19. The shell I keep open allows me to perform complex tasks and answer complex
  20. questions as I work. I find interesting things with [git grep][git grep],
  21. perform bulk find-and-replace with [sed][sed], answer questions with
  22. [awk][awk], and perform more intricate tasks on-demand with ad-hoc shell
  23. commands and pipelines. I have the freedom to creatively solve problems without
  24. being constrained to the rails laid by IDE designers.
  25. [git grep]: https://git-scm.com/docs/git-grep
  26. [sed]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html#top
  27. [awk]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html#top
  28. Here's an example of a problem I encountered recently: I had a bunch of changes
  29. in a git repository. I wanted to restore deleted files without dropping the rest
  30. of my changes, but there were hundreds of these. How can I efficiently address
  31. this problem?
  32. Well, I start by getting a grasp of the scale of the issue with git status,
  33. which shows hundreds of deleted files that need to be restored. This scale is
  34. beyond the practical limit of manual intervention, so I switch to git status
  35. -s to get a more pipeline-friendly output.
  36. ```
  37. $ git status -s
  38. D main/a52dec/APKBUILD
  39. D main/a52dec/a52dec-0.7.4-build.patch
  40. D main/a52dec/automake.patch
  41. D main/a52dec/fix-globals-test-x86-pie.patch
  42. D main/aaudit/APKBUILD
  43. D main/aaudit/aaudit
  44. D main/aaudit/aaudit-common.lua
  45. D main/aaudit/aaudit-repo
  46. D main/aaudit/aaudit-server.json
  47. D main/aaudit/aaudit-server.lua
  48. ...
  49. ```
  50. I can work with this. I add grep \'^ D\' to filter out any entries which were
  51. not deleted, and pipe it through awk \'{ print $2 }\' to extract just the
  52. filenames. I'll often run the incomplete pipeline just to check my work and
  53. catch my bearings:
  54. ```
  55. $ git status -s | grep '^ D' | awk '{ print $2 }'
  56. main/a52dec/APKBUILD
  57. main/a52dec/a52dec-0.7.4-build.patch
  58. main/a52dec/automake.patch
  59. main/a52dec/fix-globals-test-x86-pie.patch
  60. main/aaudit/APKBUILD
  61. main/aaudit/aaudit
  62. main/aaudit/aaudit-common.lua
  63. main/aaudit/aaudit-repo
  64. main/aaudit/aaudit-server.json
  65. main/aaudit/aaudit-server.lua
  66. ...
  67. ```
  68. Very good — we have produced a list of files which we need to address.
  69. Note that, in retrospect, I could have dropped the grep and just used awk to the
  70. same effect:
  71. ```
  72. $ git status -s | awk '/^ D/ { print $2 }'
  73. main/a52dec/APKBUILD
  74. main/a52dec/a52dec-0.7.4-build.patch
  75. main/a52dec/automake.patch
  76. main/a52dec/fix-globals-test-x86-pie.patch
  77. main/aaudit/APKBUILD
  78. main/aaudit/aaudit
  79. main/aaudit/aaudit-common.lua
  80. main/aaudit/aaudit-repo
  81. main/aaudit/aaudit-server.json
  82. main/aaudit/aaudit-server.lua
  83. ...
  84. ```
  85. However, we're just writing an ad-hoc command here to solve a specific,
  86. temporary problem — finesse is not important. This command isn't going to
  87. be subjected to a code review. Often my thinking in these situations is to solve
  88. one problem at a time: "filter the list" and "reword the list". Anyway, the
  89. last step is to actually use this list of files to address the issue, with the
  90. help of [xargs][xargs].
  91. [xargs]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html#top
  92. ```
  93. $ git status -s | awk '/^ D/ { print $2 }' | xargs git checkout --
  94. ```
  95. Let's look at some more examples of interesting ad-hoc shell pipelines.
  96. Naturally, I wrote a shell pipeline to find some:
  97. ```
  98. $ history | cut -d' ' -f2- | awk -F'|' '{ print NF-1 " " $0 }' | sort -n | tail
  99. ```
  100. Here's the breakdown:
  101. - `history` prints a list of my historical shell commands.
  102. - `cut -d' ' -f2-` removes the first field from each line, using space as a
  103. delimiter. `history` numbers every command, and this removes the number.
  104. - `awk -F'|' '{ print NF-1 " " $0 }` tells awk to use | as the field delimiter
  105. for each line, and print each line prefixed with the number of fields. This
  106. prints every line of my history, prefixed with the number of times the pipe
  107. operator appears in that line.
  108. - `sort -n` numerically sorts this list.
  109. - `tail` prints the last 10 items.
  110. This command, written in the moment, finds, characterizes, filters, and sorts my
  111. shell history by command complexity. Here are a couple of the cool shell
  112. commands I found:
  113. Play the 50 newest videos in a directory with
  114. [mpv](https://github.com/mpv-player/mpv):
  115. ```
  116. ls -tc | head -n50 | tr '\n' '\0' | xargs -0 mpv
  117. ```
  118. I use this command all the time. If I want to watch a video later, I will
  119. [touch][touch] the file so it appears at the top of this list. Another command
  120. transmits a tarball of a patched version of [Celeste][celeste] to a friend using
  121. netcat, minus the (large) game assets, with a progress display via [pv][pv]:
  122. [touch]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/touch.html#top
  123. [Celeste]: http://www.celestegame.com/
  124. [pv]: http://www.ivarch.com/programs/pv.shtml
  125. ```
  126. find . ! -path './Content/*' | xargs tar -cv | pv | zstd | nc 204:fbf5:... 12345
  127. ```
  128. And on my friend's end:
  129. ```
  130. nc -vll :: 12345 | zstdcat | pv | tar -xv
  131. ```
  132. tar, by the way, is an under-rated tool for moving multiple files through a
  133. pipeline. It can read and write tarballs to stdin and stdout!
  134. I hope that this has given you a tantalizing taste of the power of the Unix
  135. shell. If you want to learn more about the shell, I can recommend
  136. [shellhaters.org](http://shellhaters.org/) as a great jumping-off point into
  137. various shell-related parts of the POSIX specification. Don't be afraid of the
  138. spec — it's concise, comprehensive, comprehensible, and full of examples.
  139. I would also *definitely* recommend taking some time to learn awk in particular:
  140. [here's a brief tutorial](https://ferd.ca/awk-in-20-minutes.html).