Jo – a shell command to create JSON (2016) (opens in new tab)

I remember that when I worked at Google about a decade ago, there was this common saying:

"If the first version of your shell script is more than five lines long, you should have written it in Python."

I think there's a lot of truth in that. None of the examples presented in the article look better than had they been written in some existing scripting/programming language. In fact, had they been written in Python or Javascript, it would have been far more obvious what the resulting output would have been, considering that those languages already use {} for objects and [] for lists.

For example, take this example:

    jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false

In Python you would write it like this:

    json.dumps({"name": "JP", "object": {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday": False})

Only a bit more code, but at least it won't suffer from the Norway problem. Even though Python isn't the fastest language out there, it's likely still faster than the shell command above. There is no need to construct a fork bomb just to generate some JSON.

Some time ago I wrote zsh helper [1] that uses jo to quickly construct complex JSON requests, mainly for testing and quering services from console.

Paired with httpie [2] aliases [3] it produces concise APL-like syntax:

  POST https://httpbin.org/post test:=j`a=b c=`e=3` l=`*1 2 3``

Which translates to:

  http POST https://httpbin.org/post test:="$(jo a=b c="$(jo e=3)" l="$(jo -a 1 2 3)")"

Or, in other words, sending POST with the following body:

  {"a":"b","c":{"e":3},"l":[1,2,3]}

[1]: https://github.com/seletskiy/dotfiles/blob/78ac45c01bdf019ae... [2]: https://httpie.io/ [3]: https://github.com/seletskiy/dotfiles/blob/78ac45c01bdf019ae...

I like it, very concise but retains all the features you need.

`jq` can construct JSON "safely" from shell constructs, but is rather more verbose - e.g. with the same examples:

        $ jq -n --arg name Jane '{"name":$name}'
        
        $ jq -n \
        --argjson time $(date +%s) \
        --arg dir $HOME \
        '{"time":$time,"dir":$dir}'
        
        $ jq -n '$ARGS.positional' --args spring summer winter
        
        $ jq -n \
        --arg name JP \
        --argjson object "$(
          jq -n \
          --arg fruit Orange \
          --argjson point "$(
            jq -n \
            --argjson x 10 \
            --argjson y 20 \
            '{"x":$x,"y":$y}' \
          )" \
          --argjson number 17 \
          '{"fruit":$fruit,"point":$point,"number":$number}' \
        )" \
        --argjson sunday false \
        '{"name":$name,"object":$object,"sunday":$sunday}'

>Bam! Jo tries to be clever about types and knows null, booleans, strings and numbers.

I'm very skeptical of this. If I put x=001979 in as a value I dont think I want you trying to guess if that's supposed to be an integer or a string.

This sounds like the Norway Problem waiting to happen.

    > jo normally treats value as a literal string value, unless it begins with one of the following characters:  
    > value  action
    > @file  substitute the contents of file as-is
    > %file  substitute the contents of file in base64-encoded form
    > :file  interpret the contents of file as JSON, and substitute the result

This is convenient but also very dangerous. This feature will cause the content of an arbitrary file to be embed in the JSON if any value starts with a `@`, `%`, or `:`.

This will be a source of bugs or security issues for any script generating json with dynamic values.

Five minute job:

  $ ./jo foo=1 bar=2 obj=$(./jo -a 1 2 3 "</script" '"')
  {"foo":1,"obj":[1,2,3,"<\/script","\""],"bar":2}

  $ ./jo foo='abc
  > def
  > ghi'
  {"foo":"abc\ndef\nghi"}

  $ cat jo
  #!/usr/local/bin/txr --lisp
  
  (define-option-struct jo-opts nil
    (a   array   :bool
         "Produce array instead of object")
    (nil help    :bool
         "Print this help"))
  
  (defvarl jo-name *load-path*)
  
  (defun json-val (str)
    (match-case str
      ("true" t)
      ("false" nil)
      ("null" 'null)
      (`{@nil` (get-json str))
      (`[@nil` (get-json str))
      (@else (iflet ((num (tofloat else)))
               num
               else))))
  
  (let ((o (new jo-opts)))
    o.(getopts *args*)
    (when o.help
      (put-line "Usage:\n")
      (put-line `  @{jo-name} [options] arg*`)
      o.(opthelp)
      (exit 0))
  
    (if o.array
      (let ((items [mapcar json-val o.out-args]))
        (put-jsonl (vec-list items)))
      (let ((pairs [mapcar (lambda (:match)
                             ((`@this=@that`) (list (json-val this) (json-val that)))
                             ((@else) (error "~a: arguments must be name=obj pairs" jo-name)))
                           o.out-args]))
        (put-jsonl ^#H(() ,*pairs)))))

This is cool.

In the spirit of "do one thing well", I'd so rather use this to construct JSON payloads to curl requests than the curl project's own "json part" proposal[1] under consideration.

[1]: https://github.com/curl/curl/wiki/JSON#--jp-part

The more I work with JSON, the more I crave some kind of dedicated json editor to easily visualize and manipulate json objects, and to serialize / deserialize strings. This is especially the case with truly massive JSON objects with multiple layers of nesting. Anyway, cool tool that makes one part of the process of whipping up JSON a little less painful

Looks neat, but the documentation was hard to follow. A lot of typos, and some things didn't make sense. For example, I think 2 paragraphs were swapped because the example code below the paragraph taking about using square brackets has no square brackets (but nested objects) while the paragraph right after talks about nested objects but it's example doesn't show it (it does however seemingly demonstrate the square bracket feature mentioned previously)

And I just use Nushell. You have built-ins to create (and parse) not only json but also url, xml and more... https://github.com/skelly37/Reject-POSUCKS-embrace-Nushell#b...

This looks like an informally specified shell-friendly alternative json syntax.

I wonder if a formal syntax would help? Perhaps including relevant shell syntax (interpolation, subshell). It could clarify issues, and this different perspective might suggest improvements or different approaches.

There's also jshon which is a simple stack-based DSL for constructing JSON from shell scripts.

http://kmkeen.com/jshon/

It's written in C and is not actively developed. The latest commit, it seems, was a pull request from me back in 2018 that fixed a null-termination issue that led to memory corruption.

Because I couldn't rely on jshon being correct, I rewrote it in Haskell here:

https://github.com/dapphub/dapptools/tree/master/src/jays

This is also not developed actively but it's a single simple ~200 line Haskell program.

In the common case where you trust your input entirely you can just interpret your string as JavaScript. Then you don't even need to use quotes for the key names.

    $ alias fooson="node --eval \"console.log(JSON.stringify(eval('(' + process.argv[1] + ')')))\""
    $ fooson "{time: $(date +%s), dir: '$HOME'}"
    {"time":1457195712,"dir":"/Users/jpm"}

It may be a bit nicer to place that JavaScript in your path as a node script instead of using an alias.

    #!/usr/bin/env node
    console.log(JSON.stringify(eval('(' + process.argv[2] + ')')))

Since fooson's argument is being interpreted as JavaScript, you can access your environment through process.env. But you could make a slightly easier syntax in various ways. Like with this script:

    #!/usr/bin/env node
    for(const [k, v] of Object.entries(process.env)) {
        if (!global.hasOwnProperty(k)) {
            global[k] = v;
        }
    }
    console.log(JSON.stringify(eval('(' + process.argv[2] + ')')))

Now environmental variables can be access as if they were JS variables. This can let you handle strings with annoying quoting.

    $ export BAR="\"'''\"\""
    $ fooson '{bar: BAR}'
    {"bar": "\"'''\"\""}

If you wanted to do this without trusting your input so much, a JSON dialect where you can use single-quoted strings would get you pretty far.

    $ fooson "{'time': $(date +%s), 'dir': '$HOME'}"
    {"time":1457195712,"dir":"/Users/jpm"}

If you taught the utility to expand env variables itself you'd be able to handle strings with mixed quoting as well.

    $ export BAR="\"'''\"\""
    $ fooson '{"bar": "$BAR"}'
    {"bar": "\"'''\"\""}

You'd only need small modifications to a JSON parser to make this work.

Does it suffer from the Norway problem?

Depending on your exact needs, you might also want to try Next Generation Shell. It's a fully featured programming language for the DevOps-y stuff with convenient "print JSON" switch so "ngs -pj YOUR_EXPR" evaluates the expression, serializes the result as JSON and prints it.

Disclosure: I'm the author

Project link: https://github.com/ngs-lang/ngs

Enjoy!

Dang yo, why it's not " Jo – a shell command to create JSON written in C (2016) (jpmens.net)"

You guys do that for "written in Rust"

Direct link to repo:

https://github.com/jpmens/jo

You can implement this and many other single purpose CLI tools with inline Python or Perl or other language. Easier to remember because it's your favorite language.

    python -c "import json; print(json.dumps(dict(a=10, b=False)))"

I've been using it for years and it's a great companion of jq and jp (JMESPath).

Awesome, thank you. Lovely counterpart to jq :)

next to "copy as cURL" in dev tools, this might make the regular rotation

Could be simply called `json`.