3

When I do this:

arr=($(echo '{"crew":[{"name":"kirk"},{"name":"bones"},{"name":"mr spock"}]}' | jq -r '.crew[].name | @sh'))

I get:

echo "${arr[2]}"
mr
echo "${arr[3]}"
spock

However when I do this:

arr=("kirk" "bones" "mr spock")

I get this:

echo "${arr[2]}"
mr spock

Why, in the first example, is bash ignoring the quotes that each jq value is wrapped in when it creates the array?

peak
  • 105,803
  • 17
  • 152
  • 177
Jon Hudson
  • 1,134
  • 1
  • 10
  • 17
  • 2
    You stripped out the quotes by using jq -r – Raman Sailopal Feb 25 '21 at 14:51
  • @RamanSailopal The jq operator `@sh` escapes the output, as per the jq manual: "The input is escaped suitable for use in a command-line for a POSIX shell. If the input is an array, the output will be a series of space-separated strings." – Jon Hudson Feb 25 '21 at 14:59
  • 2
    Can you not do ... readarray -t arr <<< "$(echo '{"crew":[{"name":"kirk"},{"name":"bones"},{"name":"mr spock"}]}' | jq '.crew[].name')" – Raman Sailopal Feb 25 '21 at 15:05
  • @RamanSailopal yes, `readarray` seems to work. Thank you. However, I am still interested to know why bash seems to remove/ignore the quotes that jq is wrapping the values in. – Jon Hudson Feb 25 '21 at 15:40
  • @JonHudson See ["Why does shell ignore quoting characters in arguments passed to it through variables?"](https://stackoverflow.com/questions/12136948/why-does-shell-ignore-quotes-in-arguments-passed-to-it-through-variables) (it's about a different situation, but the word-splitting process is exactly the same). – Gordon Davisson Feb 25 '21 at 17:22

2 Answers2

2
$ ary=($(echo '{"crew":[{"name":"kirk"},{"name":"bones"},{"name":"mr spock"}]}' | jq -r '.crew[].name | @sh'))
$ declare -p ary
declare -a ary=([0]="'kirk'" [1]="'bones'" [2]="'mr" [3]="spock'")

This does not work as expected because the command substitution is unquoted, bash will perform word splitting on the output. It doesn't matter that the actual output contains quote characters. Refer to 3.5 Shell Expansions in the manual

You need to delay the array assignment until the contents of the output can be examined by the shell. This could be done with eval, but for variable assignment it's better done with declare:

$ declare -a "ary=($(echo '{"crew":[{"name":"kirk"},{"name":"bones"},{"name":"mr spock"}]}' | jq -r '.crew[].name | @sh'))"
$ declare -p ary
declare -a ary=([0]="kirk" [1]="bones" [2]="mr spock")

For readability, split that into steps:

$ jq_out=$(echo '{"crew":[{"name":"kirk"},{"name":"bones"},{"name":"mr spock"}]}' | jq -r '.crew[].name | @sh')
$ declare -a "ary=( $jq_out )"
$ declare -p ary
declare -a ary=([0]="kirk" [1]="bones" [2]="mr spock")
glenn jackman
  • 238,783
  • 38
  • 220
  • 352
  • Thanks, this helps me understand why it was happening. I didn't know quote removal was a thing https://www.gnu.org/software/bash/manual/bash.html#Quote-Removal I think however that for simplicity and because I understand it more, I will use the `readarray` solution above. – Jon Hudson Feb 25 '21 at 17:24
1

If your bash does not support mapfile/readarray, you could use the idiom illustrated by the following:

echo $BASH_VERSION

while read -r name ; do
    ary+=("$name")
done < <(echo '{"crew":[{"name":"kirk"},{"name":"bones"},{"name":"mr spock"}]}' |
   jq -r '.crew[].name')

declare -p ary
Output 
3.2.57(1)-release
declare -a ary='([0]="kirk" [1]="bones" [2]="mr spock")'

Notice in particular that

  • $name is quoted in the array-update line
  • there is no need to use @sh, which indeed in this case is probably not what you want.

Of course, if the JSON strings might contain embedded new lines, then the above would have to be tweaked accordingly, e.g. to use NUL ("\u0000") as the delimiter.

peak
  • 105,803
  • 17
  • 152
  • 177
  • You'll want to use `IFS= read -r name` -- that will read the line exactly verbatim. Without `IFS=` then leading/trailing whitespace will be removed. – glenn jackman Feb 25 '21 at 21:00