What Python is to Java, Shell is to Python. It speeds you up several times. I started using inline 'python -c' more often than the python repl now as it stores the command in shell history and it is then one fzf search away.
While neither Shell or SQL are perfect, there have been many ideas to improve them and for sure people can't wait for something new like oil shell to get production ready, getting the shell quoting hell right, or somebody fixing up SQL, bringing old ideas from Datalog and QUEL into it, fixing the goddamn NULL joins, etc.
But honestly, nothing else even comes close to this 10x productivity increase over the next best alternative. No, Thank you, I will not rewrite my 10 lines of sh into python to explode it into 50 lines of shuffling clunky objects around. I'll instead go and reread that man page how to write an if expression in bash again.
Shameless plug coming, it this has been a pain point for me too. I found the issue with quotes (in most languages, but particularly in Bash et al) is that the same character is used to close the quote as is used to open it.m. So in my own shell I added support to use parentheses as quotes in addition to the single and double quotation ASCII symbols. This then allows you to nest quotation marks.
https://murex.rocks/docs/parser/brace-quote.html
You also don’t need to worry about quoting variables as variables are expanded to an argv[] item rather than expanded out to a command line and then any spaces converted into new argv[]s (or in layman’s terms, variables behave like you’d expect variables to behave).
Though Ruby makes it confusing AF because there are two quoting types for both strings and symbols, and they're different. (%Q %q %W %w %i) I can never remember which does which.... the letter choice feels really arbitrary.
Elixir has sigils, which are useful for defining all kinds of literals easier, not just strings:
https://elixir-lang.org/getting-started/sigils.html#strings-...
You can also define your own. It's pretty great.
$X=q( foo() )
I agree that this qualified quotation is really underutilized.
I also write shell scripts, but I'm just curious what you would think about a comparison.
For those that haven't taken the time yet, I think this is a good place to start:
https://learnxinyminutes.com/docs/awk/
Of course, some people do very advanced things in awk and I absolutely agree that 1 hour of study isn't going to make you a ninja, but it's absolutely enough to learn the awk programming paradigm so that when the need arises you can quickly mobilize the solution you need.
For example: If you're quick to the draw, it can take less time to write an awk one liner to calculate the average of a column in a csv than it does to copy the csv into excel and highlight the column. It's a massive productivity booster.
[1] https://www.gnu.org/software/gawk/manual/gawk.html
For the sarcasm impared among us, everything above this, but possibly including this sentence is sarcasm.
Submitted yesterday:
Learn to use Awk with hundreds of examples
https://github.com/learnbyexample/Command-line-text-processi...
Here's a nice example of something similar: https://drewdevault.com/dynlib
Scripted cholmskey grammar ( https://en.wikipedia.org/wiki/Universal_grammar ) to unleash the power of regular expressions.
Language aside, the ecosystem and culture do not afford enough in way of testing, dependency management, feature flags, static analysis, legibility, and so on. The reason people say to keep shell programs short is because of these problems, it needs to be possible to rewrite shell programs on a whim. At least then, you can A/B test and deploy at that scope.
* Shell scripts force you to think in a more scalable way (data streams)
* Shell scripts compose rich programs rather than simplistic functions
* Shells encourage you to program with a rich, extensible feature set (ad-hoc I/O redirection, files)
The only times I don’t like shell scripts are when dealing with regex and dealing with parallelism
What is used both in case/esac and globbing are "shell patterns." They are also found in variable pattern removal with ${X% and ${X#.
In "The Unix Programming Environment," Kernighan and Pike apologized for these close concepts that are easily mistaken for one another.
"Regular expressions are specified by giving special meaning to certain characters, just like the asterix, etc., used by the shell. There are a few more metacharacters, and, regrettably, differences in meanings." (page 102)
Bash does implement both patterns and regex, which means discerning their difference becomes even more critical. The POSIX shell is easier in memory for this reason, and others.
Wow, for me parallelism is one of the best features of a unix shell and I find it vastly superior to most other programming languages.
Do you not have a ~/.python_history? The exact same search functions are available on the REPL. Ctrl-R, type your bit, bam.
It's a shell that is actually built for structured data, taking lessons learned from PowerShell and others.
Running `parallel --shellquote --shellquote --shellquote` and pasting in the line you want to quote thrice may alleviate some of the pain.
By no means ideal, though.
The comparison should be to perl or Ruby, both of which will fare better than Python for typical shell-type tasks.
But different strokes and all that.
This also works up to a point where those GBs turn into hundreds of GBs, or even PBs, and a proper distributed setup can return results in seconds.
That's not to say I'd do everything in shell. Most stuff fits well into SQl, but when it comes to optimizing processing over TB or PB scale, you won't beat shell+massive hw orchestration.
Do you have an example of this? I didn’t even know you could make sql calls in scripts.
https://adamdrake.com/command-line-tools-can-be-235x-faster-...
PSQL="psql postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@$DATABASE_HOST:$DATABASE_PORT/$POSTGRES_DB -t -P pager=off -c "
OLD="CURRENT_DATE - INTERVAL '5 years'"
$PSQL "SELECT id from apt WHERE apt.created_on > $OLD order by apt.created_on asc;" | while
read -r id; do
if [[ $id != "" ]]; then
printf "\n\*\* Do something in the loop with id where newer than \"$OLD\" \*\*\*\n"
# ...
fi
doneI don't do much sql in bash scripts but I do keep some wrapper scripts that let me run queries from stdin to databases in my environment