Though, understanding it this way makes the direction of the angled bracket a little odd; at least for me it's more natural to understand dup2(2, 1) as 2<1, as in make fd 2 a duplicate of fd 1, but in terms of abstract I/O semantics that would be misleading.
$ cat foo.sh
#!/usr/bin/env bash
>&1 echo "will print on stdout"
>&2 echo "will print on stderr"
>&3 echo "will print on fd 3"
$ ./foo.sh 3>&1 1>/dev/null 2>/dev/null
will print on fd 3
The danger is that if you don't open it before running the script, you'll get an error:
$ ./foo.sh
will print on stdout
will print on stderr
./foo.sh: line 5: 3: Bad file descriptor if [[ ! -e /proc/$$/fd/3 ]]; then
# check if fd 3 already open and if not open, open it to /dev/null
exec 3>/dev/null
fi
>&3 echo "will print on fd 3"
Now with that exec trick the fun only gets started. Because you can redirect to subshells and subshells inherit their redirection of the parent:
set -x # when debugging, print all commands ran prefixed with CMD:
PID=$$
BASH_XTRACEFD=7
LOG_FILE=/some/place/to/your/log/or/just/stdout
exec 3> >(gawk '!/^RUN \+ echo/{ print strftime("[%Y-%m-%d %H:%M:%S] <PID:'$PID'> "), $0; fflush() }' >> $LOG_FILE)
exec > >(sed -u 's/^/INFO: /' >&3)
exec 2> >(sed -u 's/^/ERROR: /' >&3)
exec 7> >(sed -u 's/^/CMD: /' >&3)
exec 8>&1 #normal stdout with >&8
exec 9>&2 #normal stderr with >&9
Now the disclaimer is that you will not have gaurantees that the order of stdout and stderr will be correct unfortunately, even though we run it unbuffered (-u and fflush).
The shell syntactical sugars also have some weird gotchas. The &2>&1 question and its answer are a good example of that. You're just trading one complexity (low level knowledge) for another (the long list of syntax rules). Shell languages break the rule of not letting abstractions get in the way of insight and intuitiveness.
I know that people will argue that shell languages are not programming languages, and that terseness is important for the former. And yet, we still have people complaining about it. This is the programmer ego and the sysadmin ego of people clashing with each other. After all, nobody is purely just one of those two.
People who build a system or at least know how it works internally want to simplify their life by building abstractions.
As people come later to use the system with the embedded abstractions, they only know the abstractions but have no idea of the underlying implementations. Those abstractions used to make perfect sense for those with prior knowledge but can also carry subtle bias which makes their use error prone for non initiated users.
This redirection relies on foundational concepts (file descriptors, stdin 0, stdout 1, stderr 2) that need to be well understood when using unix. IMO, this helps to build insight and intuitiveness. A pipe is not magic, it is just a simple operation on file descriptors. Complexity exists (buffering, zombies), but not there.
? (defun even(num) (= (mod num 2) 0))
? (filter '(6 4 3 5 2) #'even)
Which is lost when using more modern or languages foreign to Unix.
Any time the shell executes a program it forks, not just for redirections. Redirections will use dup before exec on the child process. Piping will be two forks and obviously the `pipe` syscall, with one process having its stdout dup'd to the input end of the pipe, and the other process having its stdin dup'd to the output end.
Honestly, I find the BASH manual to be excellently written, and it's probably available on your system even without an internet connection. I'd always go there than rely on stack overflow or an LLM.
https://www.gnu.org/software/bash/manual/bash.html#Redirecti...
open a terminal (OSX/Linux) and type:
man dup
man dup
To get recursive, you can try:
man man unix
Since they're both just `dup2(1, 2)`, `2>&1` and `2<&1` are the same. However, yes, `2<&1` would be misleading because it looks like you're treating stderr like an input.
https://man7.org/linux/man-pages/man2/dup.2.html
and
https://man.archlinux.org/man/dup2.2.en
A lot of bots are reading this. Amazing.
And I also disagree, your suggestion is not easier. The & operator is quite intuitive as it is, and conveys the intention.
> Respectfully, what was the purpose of this comment, really?
Judging by its replies alone, not everyone considers it purposeless. And even though I know enough to use shell redirections correctly, I still found that comment insightful. This is why I still prefer human explanations over AI. It often contains information you didn't think you needed. HN is one of the sources of the gradually dwindling supply of such information. That comment is still on-topic. Please don't discourage such habits.
> but then shouldn't it rather be &2>&1?
> & is only interpreted to mean "file descriptor" in the context of redirections. Writing command &2>& is parsed as command & and 2>&1
That's where all the confusion comes from. I believe most people can intuitively understand > is redirection, but the asymmetrical use of & throws them off.
Interestingly, Powershell also uses 2>&1. Given an once-a-lifetime chance to redesign shell, out of all the Unix relics, they chose to keep (borrow) this.
dir C:\, fakepath 2>&1 > .\dir.log
[1] https://learn.microsoft.com/en-us/powershell/module/microsof...
It's also not a file descriptor. It's a PowerShell Stream, of which there are five? you can redirect to that are similar to log levels.
You redirect stdout with ">" and stderr with "2>" (a two-letter operator).
If you want to redirect to stdout / stderr, you use "&1" or "&2" instead of putting a file name.
So, >foo is the same as 1>foo
If you want to get really into the weeds, I think 2>>&1 will create a file called 1, append to a file descriptor makes no sense (or maybe, truncate to a file descriptor makes no sense is maybe what I mean), but why this is the case is probably an oversight 50 years ago in sh, although i'd be surprised if this was codified anywhere, or relied upon in scripts.
I still acutely remember the gatekeeping and hostility of peak stack overflow, and the inanity of churning out jira tickets as fast as possible for misguided product initiatives. It's just wild yo
I also had a better experience with Stack Overflow over AI. It's been unable to tell me that I couldn't assign a new value to my std::optional in my specific case, and kept hallucinating copy constructor rules. A Stack Overflow question matching my problem cleared that up for me.
Sometimes you need someone to tell you no.
I could not disagree more! With pesky humans, you have all sorts of things to worry about:
- is my question stupid? will they think badly of me if i ask it?
- what if they dont know the answer? did i just inadvertantly make them look stupid?
- the question i have is related to their current work... i hope they dont see me as a threat!
and on and on. asking questions in such a manner as to elicit the answer, without negative externalities, is quite the art form as i'm sure many stack overflow users will tell you. many word orderings trigger a 'latent space' which activates the "umm, why are you even doing this?" with the implication begin "you really are stupid!", totally useless to the question-asker and a much more frustrating time-waster than even the most moralizing LLM.
with LLMs, you don't have to play these 'token games'. you throw your query at it, and irrespective of the word order, word choice, or the nture of the question - it gives you a perfectly neutral response, or at worst politely refuses to answer.
> many word orderings trigger a 'latent space' which activates the "umm, why are you even doing this?" with the implication begin "you really are stupid!"
You may have heard of the XY situation when people asks a Y question only because they have an incorrect answer to X. A question has a goal (unless rethorical) and to the person being asked, it may be confusing. You may have a valid reason to go against common sense, but if the other person is not your tutor or a fellow researcher, he may not be willing to accommodate you and spend his time for a goal he have no context about.
Remember the car wash question for LLMs? Some phrasing have the pattern of a trick question and that’s another thing people watch out for.
File descriptors are like handing pointers to the users of your software. At least allow us to use names instead of numbers.
And sh/bash's syntax is so weird because the programmer at the time thought it was convenient to do it like that. Nobody ever asked a user.
You can for the destination. That's the whole reason you need the "&": to tell the shell the destination is not a named file (which itself could be a pipe or socket). And by default you don't need to specify the source fd at all. The intent is that stdout is piped along but stderr goes directly to your tty. That's one reason they are separate.
And for those saying "<" would have been better: that is used to read from the RHS and feed it as input to the LHS so it was taken.
2>/dev/stdout
And of course I wish you could use a friendlier name for STDERR instead of `2>`
The situation where this is going to cause confusion is when you do this for multiple commands. It looks like they're all writing to a single file. Of course, that file is not an ordinary file - it's a device file. But even that isn't enough. You have to know that each command sees its own incarnation of /dev/stdout, which refers to its own fd1.
Shell is from a time when you had a huge selection of languages, each for different purposes, and you picked the right one for the job. For complex applications, you would have multiple languages working together.
People look at Bash and think, "I would never dare do $Task with that language!". And you'd be right, because you're thinking you only have one tool in the toolbox.
Which means that reading someone else's shell script (or awk, or perl, or regex) is INCREDIBLY inconvenient.
But my main reason is that most scripts break when you call them with filenames that contain spaces. And they break spectacularly.
In the C API of course there's symbolic names for these. STDIN_FILENO, STDOUT_FILENO, etc for the defaults and variables for the dynamically assigned ones.
You can use /dev/stdin, /dev/stdout, /dev/stderr in most cases, but it's not perfect.
Never ever write code that assumes this. These dev shorthands are Linux specific, and you'll even need a certain minimum Linux version.
I cringe at the amount of shell scripts that assume bash is the system interpreter, and not sh or ksh.
Always assume sh, it's the most portable.
Linux != Unix.
Many people probably think in terms of "fd 0" and "fd 1" instead of "standard in" and "standard out", but should you wish to use names at least on modern Linux/BSD systems do:
echo message >/dev/stdout
echo error_message >/dev/stderr echo >&2 error_message
install /dev/stdin file <<EOF
something
EOFI want to be able to route x independent input and y independent output trivially from the terminal
Proper i/o routing
It shouldn't be hard, it shouldn't be unsolved, and it shouldn't be esoteric
Sure. Here's what that looked like:
Even if you're a programmer, that doesn't mean you magically know what other programmers find easy or logical.
What should be the syntax according to contemporary IT people? JSON? YAML? Or just LLM prompt?
Why is there a 2 on the left, when the numbers are usually on the right. What's the relationship between 2 and 1? Is the 2 for std err? Is that `&` to mean "reference"? The fact you only grok it if you know POSIX sys calls means it's far from self explanatory. And given the proportion of people that know POSIX sys calls among those that use Bash, I think it's a bit of an elitist syntax.
https://www.gnu.org/software/bash/manual/html_node/Redirecti...
The usual thing (before LLMs) is to Google the question, but for the question to appear in Google, someone has to ask it first, and here we are.
Also the Stackoverflow answers give different perspectives, context, etc... rather than just telling you what it does, which is useful to someone unfamiliar with how redirections work. As I said, someone who doesn't know about "2>&1" is unlikely to be an expert given how common the pattern is, so a little hand holding doesn't hurt.
Where else would you look but in the manual of your shell? And you don’t have to know in which section to look, you can just search for “2>&1” in the bash man page.
Google search literally is useless for these days, for Average Joe.
In Emacs, when I hit C-h i I get a menu of all my info manuals and I first read the bash one there.
diff <(seq 1 20) <(seq 1 10)
Also the reason why Zsh has an additional =(command) construct which uses temporary files instead.
It would be great to be able to open a socket in bash[^1] and pass it to another program to read/write from without having an extra socat process and pipes running (and the buffering, odd flush behaviour, etc.). It would be great if programs expected to receive input file arguments as open fds, rather than providing filenames and having the process open them itself. Sandboxing would be trivial, as would understanding the inputs and outputs of any program.
It's frustrating to me because the underlying unix system supports this so well, it's just the conventions of userspace that get in the way.
[^1]: I know about /dev/tcp, but it's very limited.
[1]: https://www.oreilly.com/library/view/essential-system-admini...
Normally when you do something like command > file.txt, you’re only capturing the normal output — errors still go to your screen.
2>&1 is how you say: “send the error pipe into the same place as the normal output pipe.” Breaking it down without jargon: • 2 means “the error output” • > means “send it to” • &1 means “wherever the normal output is currently going” (the & just means “I’m referring to a pipe, not a file named 1”)
This response is essentially just the second answer to the linked question (the response by dbr) with a bunch of the important words taken out.
And all it cost you to get it was more water and electricity than simply clicking the link and scrolling down — to say nothing of the other costs.
"I didn't have time to write you a short letter, so I wrote you a long one." is real.
If you want it with the correct terminology:
2 means "file descriptor 2", > means "assign the previous mentioned to the following", &2 means "file descriptor 1" (and not file named "1")
It's very, very easy to get shell scripts wrong; for instance the location of the file redirect operator in a pipeline is easy to get wrong.
One customer complained about our software corrupting files on their hard disk. Turns out they had modified their systems so that a newly-spawned program was not given a stderr. That is, it was not handed 0, 1, and 2 (file descriptors), but only 0 and 1. So whenever our program wrote something to stderr, it wrote to whatever file had been the first one opened by the program.
We talked about fixing this, briefly. Instead we decided to tell the customer to fix their broken environment.
It redirects STDERR (2) to where STDOUT is piped already (&1). Good for dealing with random CLI tools if you're not a human.
It also teaches how && and || work, their relation to [output redirection][3] and [command piping][2], [(...) versus {...}][4], and tricky parts like [word expansion][5], even a full grammar. It's not exciting reading, but it's mostly all there, and works on all POSIXy shells, e.g. sh, bash, ksh, dash, ash, zsh.
[1]: https://pubs.opengroup.org/onlinepubs/7908799/xcu/chap2.html
[2]: https://pubs.opengroup.org/onlinepubs/7908799/xcu/chap2.html...
[3]: https://pubs.opengroup.org/onlinepubs/7908799/xcu/chap2.html...
[4]: https://pubs.opengroup.org/onlinepubs/7908799/xcu/chap2.html...
[5]: https://pubs.opengroup.org/onlinepubs/7908799/xcu/chap2.html...
foo &> file
foo |& programThe question was how to remember it's "2>&1" and not "2&>1". If you think of "&1" as the address/destination of, the syntax is quite natural.
https://man.cat-v.org/unix_7th/1/sh#:~:text=%3C%26digit%0A%2...
Seriously when it comes to unix RTFM RTFM RTFM and you'll get the top comment on SO and HN rolled into one.
Which actually means that an undelrying dup2 operation happens in this direction:
2 <- 1 // dup2(2, 1)
The notation follows I/O redirections: cmd > file actually means that a descriptor [n] is first created for the open file, and then that descriptor's decription is duplicated into [1]:
n <- open("file", O_RDONLY)
1 <- nI've only ever been tricked into working on C++...
The comment about "why not &2>&1" is probably the best one on the page, with the answer essentially being that it would complicate the parser too much / add an unnecessary byte to scripts.
$ ./outerr >blah 2>&1
$ ./outerr | 2>&1 cat >blah
err
(where outerr is the following...)
echo out
echo err >&2But also | isnt a redirection, it takes stdout and pipes it to another program.
So, if you want stderr to go to stdout, so you can pipe it, you need to do it in order.
bob 2>&1 | prog
You usually dont want to do this though.
First the | pipe is established as fd [1]. And then 2>&1 duplicates that pipe into [2]. I.e. right to left: opposite to left-to-right processing of redirections.
When you need to capture both standard error and standard output to a file, you must have them in this order:
bob > file 2>&1
bob 2>&1 > file
But if you change > file to | process, then it's fine! process gets the combined error and regular output.
# echo 1 >&2 2>| echo
I had never made the connection of the & symbol in this context. I think I never really understood the operation before, treating it just as a magic incantation but reading this just made it click for me.
To be consistent, it would be &2>&1, but that makes it more verbose than necessary and actually means something else -- the first & means that the command before it runs asynchronously.
Thus you cannot write:
2 > &1
2 >& 1
2>& 1
[0] https://stackoverflow.com/questions/3618078/pipe-only-stderr...
Treating ">&" as a distinct operator actually makes an elegant solution here. I like the idea.
Would probably be hard to guess since the process may not have opened any file once it started.
It is not. You can use any arbitrary numbers provided they're initialized properly. These values are just file descriptors.
For Example -> https://gist.github.com/valarauca/71b99af82ccbb156e0601c5df8...
I've used (see: example) to handle applications that just dump pointless noise into stdout/stderr, which is only useful when the binary crashes/fails. Provided the error is marked by a non-zero return code, this will then correctly display the stdout/stderr (provided there is <64KiB of it).
> Would probably be hard to guess since the process may not have opened any file once it started.
You need to not only inspect the current state, but also race the process before the assignments change.
command &2>&1
Since the use of & signifies a file descriptor. I get what this ACTUALLY does is run command in the background and then run 2 sending its stout to stdout. That’s completely not obvious by the way.
On the other hand, pipe “|” is brilliant!
[0] <https://www.gnu.org/software/bash/manual/bash.html#Redirecti...>
[1] <https://www.gnu.org/software/bash/manual/bash.html#Pipelines...>
cmd >&out-and-err.txtLook man, I didn’t invent this stupid shit, and I’m not telling you it’s brilliant, so don’t kill the messenger.
I thought I’d seen somewhere that zsh had a better way to do this but I must have imagined it. Or maybe I’m confusing it with fish.