Shell script best practices, from a decade of scripting things (opens in new tab)

My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different programming paradigm than languages like Perl, Python, whatever.

Obviously, if your mental model is BASIC and you try to write Python, then you encounter lots of friction and it's easy for the latter to feel hacky, bad and ugly. To enjoy and program Python well, it's probably best to shift your mental model. The same goes for Shell.

What is the Shell paradigm? I would argue that it's line-oriented pipelines. There is a ton to unpack in that, but a huge example where I see friction is overuse of variables in scripts. Trying to stuff data inside variables, with shell's paucity of data types is a recipe for irritation. However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against.

/2cents

I think fish is quite a bit different in terms of syntax and semantics (I'm not very familiar with it), but zsh is essentially the same as bash except without most of the needless footguns and awkwardness. zsh also has many more advanced features, which you don't need to use (and many people are unaware of them anyway), but will very quickly become useful; in bash all sorts of things require obscure incantations and/or shell pipelines that almost make APL seem obvious in comparison.

In my experience few people understand bash (or POSIX sh) in the first place, partly because everything is so difficult and full of caveats. Half my professional shell scripting experience on the job is fixing other people's scripts. So might as well use something that doesn't accidentally introduce bugs every other line.

Most – though obviously far from all – scripts tend to be run in environments you control; portability is often overrated and not all that important (except when it is of course). Once upon a time I insisted on POSIX sh, and then I realised that actually, >90% of the scripts I wrote were run just by me or run only in an environment otherwise under my control, and that it made no sense. I still use POSIX sh for some public things I write, when it makes sense, but that's fairly rare.

I think bash is really standing in the way of progress, whether that progress is in the form of fish, zsh, oil shell, or something else, because so many people conflate "shell" with "bash", similar to how people conflate "Google" with "search" or "git" with "GitHub" (to some degree).

I don't agree with this one. When I name my script without extension (btw, .sh is fine, .bash is ugly) I want my script to look just like any other command: as a user I do not care what the language program is written in, I care about its output and what it does.

When I develop a script, I get the correct syntax highlight becuase of the shebang so the extension doesn't matter.

The rest of the post is great.

Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior.

"Prefer to use set -o nounset."

ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". There is no real downside.

"Use set -o pipefail."

Waste of time. You will spend so much time debugging your app from random pipe failures that actually didn't matter. Dont use this option; just check the output of the pipe for sane values.

"Use [[ ]] for conditions"

No!!! Only use that for bashisms where there's no POSIX alternative and try to avoid them wherever possible. YAGNI!

"Use cd "$(dirname "$0")""

Use either "$(dirname "${BASH_SOURCE[0]}")" or grab a POSIX readfile-f implementation.

"Use shellcheck."

This should have been Best Practice #1. You will learn more about scripting from shellcheck than 10 years worth of blog posts. Always use shellcheck. Always.

Also, don't use set -o nounset when set -u will do. Always avoid doing something "fancy" with a Bashism if there's a simpler POSIX way. The whole point of scripts is for them to be dead simple.

Pass all scripts through https://www.shellcheck.net/ or use `shellcheck` on the commandline.

Learn the things it tells you and implement them in future scripts.

He suggests to `set -eu`, which is a good idea, but then immediately does this:

    if [[ "$1" =~ ^-*h(elp)?$ ]]; ...

    if [[ "${1-}" =~ ^-*h(elp)?$ ]]; then

Unfortunately, `errexit` is fairly subtle. For example

    [ "${some_var-}" ] && do_something

However! If that's the last line of a function, then the function's return code will inherit the exit code of that line, meaning that

    f(){ [ "${some_var-}" ] && do_something;}; f

Anyway, there are a few subtleties like this that are worth being aware of. This is a good, but dated, reference: https://mywiki.wooledge.org/BashFAQ/105

I'm a fan of using BASH3 boilerplate: https://bash3boilerplate.sh/

It's standalone, so you just start a script using it as a template and delete bits that you don't want. To my mind, the best feature is having consistent logging functions, so you're encouraged to put in lots of debug commands to output variable contents and when you change LOG_LEVEL, all the extraneous info doesn't get shown so there's no need to remove debug statements at all.

The other advantage is the option parsing, although I don't like the way that options have to have a short option (e.g. -a) - I'd prefer to just use long options.

> And it’s usually always appropriate.

I wouldn't think so. You don't know where your script will be called from, and many times the parameters to the script are file paths, which are relative to the caller's path. So you usually don't want to do it.

I collected many tips&tricks from my experience with shell scripts that you may also find useful: https://raimonster.com/scripting-field-guide/

If a shell script needs any kind of functionality beyond POSIX, then that's a good time to upgrade to a higher-structure programming language.

Here's my related list of shell script tactics:

http://github.com/sixarm/unix-shell-script-tactics

Agreed on the powerful bit, however [[ ]] is not a "builtin" (whereas [ and test are builtins in bash), it's a reserved word which is more similar to if and while.

That why [[ ]] can break some rules that builtins cannot, such as `[[ 1 = 1 && 2 = 2 ]]` (vs `[ 1 = 1 ] && [ 2 = 2]` or `[ 1 = 1 -a 2 = 2 ]`, -a being deprecated).

Builtins should be considered as common commands (like ls or xargs) since they cannot bypass some fundamental shell parsing rules (assignment builtins being an exception), the main advantages of being a builtin being speed (no fork needed) and access to the current shell process env (e.g. read being able to assign a variable in the current process).

For example, using cd "$(dirname "$0")" to get the scripts location is not reliable, you could use a more sophisticated option such as: $(dirname $BASH_SOURCE)

> People can now enable debug mode, by running your script as TRACE=1 ./script.sh instead of ./script.sh.

The above "if" condition will set xtrace even when user explicitly disables trace by setting TRACE=0.

A correct way of doing this would be:

    if [[ "${TRACE-0}" == "1" ]]; then set -o xtrace; fi

  while getopts :hvr:e: opt
  do
      case $opt in
          v)
              verbose=true
              ;;
          e)
              option_e="$OPTARG"
              ;;
          r)
              option_r="$option_r $OPTARG"
              ;;
          h)
              usage
              exit 1
              ;;
          \*)
              echo "Invalid option: -$OPTARG" >&2
              usage # call some echos to display docs or something...
              exit 2
              ;;
      esac
  done

For someone who knows errexit can't be trusted, and codes defensively anyway, it's fine.

* safe ways to do things in bash: https://github.com/anordal/shellharden/blob/master/how_to_do...

* better scripting: https://robertmuth.blogspot.in/2012/08/better-bash-scripting...

* robust scripting: https://www.davidpashley.com/articles/writing-robust-shell-s...

The order of commandline args shouldn't matter.

Env vars are better at passing key/value inputs than commandline arguments are.

Process-substitution can often be used to avoid intermediate files, e.g. `diff some-file <(some command)` rather than `some command > temp; diff some-file temp`

If you're making intermediate files, make a temp dir and `cd` into that

- Delete temp dirs using an exit trap (more reliable than e.g. putting it at the end of the script)

- It may be useful to copy `$PWD` into a variable before changing directory

Be aware of subshells and scope. For example, if we pipe into a loop, the loop is running in a sub-shell, and hence its state will be discarded afterwards:

  LINE_COUNT=0
  some command | while read -r X
  do
    # This runs in a sub-shell; it inherits the initial LINE_COUNT from the parent,
    # but any mutations are limited to the sub-shell, will be discarded
    (( LINE_COUNT++ ))
  done
  echo "$LINE_COUNT"  # This will echo '0', since the incremented version was discarded

  LINE_COUNT=0
  while read -r X
  do
    # This runs in the main shell; its increments will remain afterwards
    (( LINE_COUNT++ ))
  done < <(some command)
  echo "$LINE_COUNT"  # This will echo the number of lines outputted by 'some command'

The programmer, who was very proud of his mastery of C, said: “How can this be? C is the language in which the very kernel of Unix is implemented!”

Master Foo replied: “That is so. Nevertheless, there is more Unix-nature in one line of shell script than there is in ten thousand lines of C.”

The programmer grew distressed. “But through the C language we experience the enlightenment of the Patriarch Ritchie! We become as one with the operating system and the machine, reaping matchless performance!”

Master Foo replied: “All that you say is true. But there is still more Unix-nature in one line of shell script than there is in ten thousand lines of C.”

The programmer scoffed at Master Foo and rose to depart. But Master Foo nodded to his student Nubi, who wrote a line of shell script on a nearby whiteboard, and said: “Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?”

The programmer muttered through his beard, contemplating what Nubi had written. Finally he agreed that it was so.

“And how many hours would you require to implement and debug that C program?” asked Nubi.

“Many,” admitted the visiting programmer. “But only a fool would spend the time to do that when so many more worthy tasks await him.”

“And who better understands the Unix-nature?” Master Foo asked. “Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?”

Upon hearing this, the programmer was enlightened.

(https://catb.org/~esr/writings/unix-koans/)

`-h` and `--help` are fine. But `help` and `h` should only display the help if the script has subcommands (like `git`, which has `git commit` as a subcommand). Scripts that don't have subcommands should treat `h` and `help` as regular arguments -- imagine if `cp h h.bak` displayed a help message instead of copying the file named "h"!

I wouldn't encourage `-help` for displaying the help because it conflicts with the syntax for a group of single-letter options (though if `-h` displays the help, there is no legitimate reason for grouping `-h` with other options).

And ideally scripts that support both option and non-option arguments should allow `--` to separate them (e.g. `rm -- --help` removes the file called "--help"). But parsing options is complicated and probably out of scope for this article.

> If appropriate, change to the script’s directory close to the start of the script. And it’s usually always appropriate.

This is very problematic if the script accepts paths as arguments, because the user would (rightly) expect paths to be interpreted relative to the original working directory rather than the script's location. A more robust approach is to compute the script's location and store it in a variable, then explicitly prepend this variable when you want paths to be relative to the script's location.

As soon as output needs to be parsed — especially when it’s being fed back into other parts of the script — it gets harder. Handling errors and exceptions is even more difficult.

Things really fall down on modularity. There are tricks and conventions: for example you can put all functions to do with x in a file called lib/x.sh, prefix them all with x_, and require that all positional parameters must be declared at the top of each function with local names.

At that point though, I would rather move to a language with named parameters, namespaced modules, and exception handling. In Python, it’s really easy to do the shell bits with:

  def sh(script):
    subprocess.run(
      [‘sh’, ‘-c’, script, ‘--‘, *args],
      check=True,
    )

(Disclaimer: I'm one of the authors)

After falling in love with ShellCheck several years ago, with the help of another person, I made the ShellCheck REPL tool for Bash:

  https://github.com/HenrikBengtsson/shellcheck-repl  

I found it to be an excellent way of learning about pit falls and best practices in Bash as you type, because it gives you instant feedback on possible mistakes. It won't execute the command until the ShellCheck issues are fixed, e.g. missing quotes, use of undefined variables, or incorrect array syntax.

It's designed to be flexible, e.g. you can configure ShellCheck rules to be ignored, and you can force executtion by adding two spaces at the end.

License: ISC (similar to MIT). Please help improve it by giving feedback, bug reports, feature requests, PRs, etc.

Traditional shell might be:

  grep -q thing < file
  if [ $? -eq 0 ] ; then echo "thing is there ; fi
  

  if
    grep -q thing < file
  then
    echo "thing is there"
  fi

I'm not saying *never* write shell scripts, but always consider doing something else, or at least add a TODO, or issue, to write in a more robust language.

always quote filenames, because you never know if there's a space in them.

filenames with dashes or periods will kill you

prepend current directory file manipulation filenames with "./", because the file might start with a period or dash

Dashes in filenames still might kill you, especially if you pass those to another command

I'd also say that in most cases Python is also a better choice, especially when you use the ! syntax.

exec &> >( ts '[%Y-%m-%d.%H:%M:%S] ' | tee ${LOGFILENAME} )

Case in point: Declaring an array. IMHO, it’s just not ergonomic at all. Especially not in sh/dash.

* use bats for testing * use shfmt for code formatting * use shellcheck for linting

    export PS4='+ ${BASH_SOURCE:-}:${FUNCNAME[0]:-}:L${LINENO:-}:   '

[0]: https://marketplace.visualstudio.com/items?itemName=timonwon...

I think this site is amazing, and it must be older than at least two decades.

    echo_command()
    {
        echo
        echo '$' "$@"
    }

    echo_and_run_command()
    {
        echo_cmd "$@"
        "$@"
    }

    main()
    {
        # For simple commands that do not use | < > etc.
        echo_and_run_command cp --verbose ...

        # More complex commands
        echo_command grep ... '|' find ...
        grep ... | find ...
    }

    main "$@"

Don’t.

Use a proper programming language instead. bash (and similar scripting languages) are non-portable (Linux/Windows) and the opposite of what I want in a good programming language.

I agree we need a shell scripting language, I disagree that bash zsh or anything that frequently uses double square brackets and awful program names is the epitome of shell scripting language design.

giving up on the notion "others will use or collaborate with my scripts" was the single most productive thing i've done for my scripting.

A couple of days ago this link was posted to hn http://mywiki.wooledge.org/BashFAQ/105

It showed me once again how little bash I know even after all those years. I checked the examples to see if only set -e is dangerous or also set -o like the author suggested and sure enough it's just as bad es set -e. You just got to thoroughly check your bash scripts and do proper error handling.

Adding an extension to make it easier to tell what's inside without opening it is being lazy rather than following best practices. Best practice is half century of leaving them off.

Unlike Windows, which ignores extensions and lets you run a command omitting them, Unix has a better (I'm not saying perfect) approach which allow the metadata to pulled from the first line of the file, tuned exactly to what the script needs. No sane extension is going to capture this info well.

Extensions expose (usually incompletely) the implementation details of what's inside, to the detriment of the humans using them (the OS doesn't care), who will then guess at what the extension means.

However, many extensions are WRONG, or too vague to actually tell what interpreter to call on them - which this subgroup of devs does all the time, mostly commonly using the wrong version of python (wrong major, wrong minor, not from a specific python env) and breaking things. .sh is manifestly wrong as an extension for Bash scripts, which have different syntax.

The exception is scripts that should be "."-ed in (sourced), where having a meaningful .sh or .bash (which are NOT interchangeable) is ACTUALLY good, because it highlights that they are NOT COMMANDS. (and execute isn't enabled)

If you want a script to make it easier to list commands that are shell scripts or whatever, there's a simple one at the end of:

https://www.talisman.org/~erlkonig/documents/commandname-ext...

I've seen several cases of .sh scripts which contained perl code, python, or were actually binary, because the final lynchpin in this (abridged) case against extensions is that in complex systems the extensions often have to be kept even after the implementation is upgraded to avoid breaking callers. It's very normal for a program to start as shell, get upgraded to python, and sometimes again to something compiled. Setting up a situation which would force the extension to be changed in all clients in a large network to keep it accurate is beyond stupid.

Don't use extensions on commands, and stop trying to rationalize it because you (for those to whom this applies) just like to do "ls *.sh" (on your bash scripts). These are a violation of Unix best practices, and cause harm when humans try to interpret them.

shellcheck

It's like pylint for your shellscripts.

- `set -o errtrace`: trap errors inside functions

- `shopt -s inherit_errexit`: subprocesses inherit exit error

Unfortunately the list of Bash pitfalls is neverending, but that's a good start.

And the right-hand-side of a variable assignment.

And the WORD in a case statement. (Not in the patterns, though).

Plus a bunch of other single-token(?) contexts.

I don't recommend relying on the context though, it's clever and makes it hard to verify that the script does not have expansion bugs.

1. end all your lines C-style; this may save your life many times;

2. declare -is variables and -r CONSTANTS at the beginning, again, C-style;

3. print TIMESTAMP="$(date +%Y-%m-%d\ %H:%M:%S)"; where appropriate if your script logs its job;

4. Contrary to OPs reommendation I strongly try to stick to pure SH compatibility in smaller acripts so they can run on routers, TVs, androids and other busybox-like devices; BASH isn't everywhere.

http://robertmuth.blogspot.com/2012/08/better-bash-scripting...

https://google.github.io/styleguide/shellguide.html

  cd "$(dirname "$0")"

EDIT: I should've just tested this to see :) I did and it does exactly that. Very helpful. I didn't realize $0 is always the first argument. Kind of like how `self` is the first implicit argument in OOP methods?

Kudos for nicely put tips that are easy to follow and understand.

I'm newcomer in bash

For example, instead of

    USER=mail
    UID=8

    USER=mail
    UID=$(id -u $USER)

Also note that this is something that should be done in any programming language, not just shell scripts.

The explanation for that wasnt really an explanation either ...

I won't check with for version those tips applies, and continue writing POSIX shell as much as can. I might check which or those suggestions are POSIX, though.

    1. use shellcheck
    2. use shfmt (to format your shell script)
    3. set -euo pipefail  (much shorter)

My order preference would be:

    1. use shellcheck.
    … rest …

Perhaps not surprisingly, it's bourne shell, not bash. But still, it's an actual published standard all can refer to when the language in question is "shell scripts", i.e. .sh files, or "shell commands" in some context where a shell command is called for (e.g. portable makefiles).

  https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html

* Write help text to stdout, not stderr, so I can grep it

* Set exit status to 0 on success and 1 or some other small positive integer on failure so I can use || and &&

> 9. Always quote variable accesses with double-quotes.

Does the author refer to "$MYVAR"? Why would you want to use that over ${MYVAR}?

But I use, set -euo pipefail. I think -u is -o unset ,etc? Just easier to type.

2. Shell scripts are wonderful, but once they exceed a few lines (give or take 50), they've entered the fast track on becoming a maintenance headache and a liability.

.sh is appropriate for a shell library module which you source from another shell script. It is not really appropriate for something which is more abstract (such as a "program" inside your PATH).

set -e / set -o errexit will only be helpful if you fundamentally understand exactly what it does, if you don't, you are bound to end up with broken code. Once you fundamentally understand set -e you will be better placed to decide whether it is appropriate to use it or more appropriate to simply do proper error handling. The oft repeated mantra of using set -e is really misleading a lot of people into thinking that bash has some sane mode of operation which will reduce their chance of making mistakes, people should never be mislead to think that bash will ever do anything sane.

set -u / set -o nounset breaks a lot of perfectly sensible bash idioms and is generally bad at what proponents of it claim it will help solve (using unset variables by accident or by misspelling). There are better linters which solve this problem much better without having to sacrifice some of what makes bash scripts easier to write/read.

set -o pipefail is not an improvement/detriment, it is simply changing the way that one of bash's features functions. pipefail should only be set around specific uses of pipelines when it is known that it will produce the intended result. For example, take this common idiom:

    if foo | grep -q bar; then ...

Guarding set -x / set -o xtrace seems unnecessary, -x is already automatically inherited. Just set it before running the program.

Good advice on using [[ but it is important to fundamentally understand the nuances of this, quoting rules change within the context of [[.

Accepting h and help seems incredibly unnecessary. If someone who has never used a unix-like operating system happens upon your script then they may find it useful. But I don't think catering to such a low common denominator makes sense. Your script should just handle invalid arguments by printing a usage statement with maybe a hint of how to get a full help message.

I'd say changing to your script's directory is almost never appropriate.

Shellcheck, while useful, is useful only if you understand bash well.

The lesson here is that if you think that you have spent enough time writing bash to suggest best practices, you've not spent enough time writing bash. Only when you realise that the best practice is to not use bash have you used bash long enough (or short enough).

If you want to write a script which you're going to rely on or distribute, learn bash inside out and then carefully consider if it's still the right option.

If you are unwilling or unable to learn bash inside out then please use something else.

Do not be fooled into thinking that some "best practices" you read online will save you from bash.

Are you listening Apple?

1. Bash shouldn't be used, not because of portability, but because its features aren't worth their weight and can be picked up by another command, I recommend (dash) any POSIX complaint shell (bash --posix included) so you aren't tempted to use features of bash and zsh that are pointless, tricky or are there for interactivity. Current POSIX does quite well for what you would use shell for.

2. Never use #!/usr/bin/env bash. Even if you are using full bash, bash should be installed in /usr/bin/bash. If you don't even know something this basic about the environment, then you shouldn't be programming it, the script is likely to create a mess somewhere in the already strange environment of the system.

3. Don't use extensions unless you're writing for Windows machines. Do you add extensions to any other executable? head, sed can help you retrieve the first line of a file and neither of them have extensions.

4, 5, 6. You may do this is obscure scenarios where you absolutely cannot have a script run if there is any unforeseen error, but it's definitely not something that should be put on without careful consideration, http://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail explains this better. And it goes without saying that this is not a substitute for proper error handling.

7. I agree that people should trace their shell scripts, but this has nothing to with shell.

8. [[]] is powerful, so I very often see it used when the [] builtin would suffice. Also, [[ is a command like done, not a bash builtin.

9. Quote only what needs quoting. If you don't know what needs quoting, then you don't understand your script. I know it seems like a waste of time, but it will make you a much better shell programmer then these always do/don't do X unless Y then do Z, rules that we are spouting.

10. Use either local or global variables in functions, depending on which you want. I see no reason to jump through this weird hoop because it might become an easily fixable problem later.

11. This is a feature, make usage appear when you blink, I don't care, if anything variations of -h too limited,

12. Finally, one "opinion" we agree on, not sure how else to redirect to stderr, but I'm sure that other way isn't as good as this one.

13. No, read the usage. If you want inferior long options, then you can add them to your scripts, but they are not self documenting, they only serve to make commands less readable and clutter completion.

14. No, it's not usually appropriate, do you want all installed scripts writing to /bin? The directory the script is running in should be clearly communicated to the user, with cd "$(dirname "$0")", "It runs in the directory the script is in." Needs to be communicated somewhere, or you have failed.

15. Yes, use ShellCheck.

16. Please call your list Bash Script Practices if it's unrelated to shell.

Credibility gone.