How can I tell if a jq filter successfully pulls data from a JSON data structure?

Question

I want to know if a given filter succeeds in pulling data from a JSON data structure. For example:

###### For the user steve...

% Name=steve
% jq -j --arg Name "$Name" '.[]|select(.user == $Name)|.value' <<<'
[
   {"user":"steve", "value":false},
   {"user":"tom", "value":true},
   {"user":"pat", "value":null},
   {"user":"jane", "value":""}
]'
false
% echo $?
0

Note: successful results can include boolean values, null, and even the empty string.

###### Now for user not in the JSON data...

% Name=mary
% jq -j --arg Name "$Name" '.[]|select(.user == $Name)|.value' <<<'
[
   {"user":"steve", "value":false},
   {"user":"tom", "value":true},
   {"user":"pat", "value":null},
   {"user":"jane", "value":""}
]'
% echo $?
0

If the filter does not pull data from the JSON data structure, I need to know this. I would prefer the filter to return a non-zero return code.

How would I go about determining if a selector successfully pulls data from a JSON data structure vs. fails to pull data?

Important: The above filter is just an example, the solution needs to work for any jq filter.

Note: the evaluation environment is Bash 4.2+.

Answer 1

I've found a solution that meets all of my requirements! Please let me know what you think!

The idea is use jq -e "$Filter" as a first-pass check. Then for the return code of 1, do a jq "path($Filter)" check. The latter will only succeed if, in fact, there is a path into the JSON data.

Select.sh

#!/bin/bash

Select()
{
   local Name="$1"
   local Filter="$2"
   local Input="$3"
   local Result Status

   Result="$(jq -e --arg Name "$Name" "$Filter" <<<"$Input")"
   Status=$?

   case $Status in
   1) jq --arg Name "$Name" "path($Filter)" <<<"$Input" >/dev/null 2>&1
      Status=$?
      ;;
   *) ;;
   esac

   [[ $Status -eq 0 ]] || Result="***ERROR***"
   echo "$Status $Result"
}

Filter='.[]|select(.user == $Name)|.value'
Input='[
   {"user":"steve", "value":false},
   {"user":"tom", "value":true},
   {"user":"pat", "value":null},
   {"user":"jane", "value":""}
]'

Select steve "$Filter" "$Input"
Select tom   "$Filter" "$Input"
Select pat   "$Filter" "$Input"
Select jane  "$Filter" "$Input"
Select mary  "$Filter" "$Input"

And the execution of the above:

% ./Select.sh
0 false
0 true
0 null
0 ""
4 ***ERROR***

Answer 2

You can use the -e / --exit-status flag from the jq Manual, which says

Sets the exit status of jq to 0 if the last output values was neither false nor null, 1 if the last output value was either false or null, or 4 if no valid result was ever produced. Normally jq exits with 2 if there was any usage problem or system error, 3 if there was a jq program compile error, or 0 if the jq program ran.

I can demonstrate the usage with a basic filter as below, as your given example is not working for me.

For a successful query,

dudeOnMac:~$ jq -e '.foo?' <<< '{"foo": 42, "bar": "less interesting data"}'
42
dudeOnMac:~$ echo $?
0

For an invalid query, done with a non-existent entity zoo,

dudeOnMac:~$ jq -e '.zoo?' <<< '{"foo": 42, "bar": "less interesting data"}'
null
dudeOnMac:~$ echo $?
1

For an error scenario, returning code 2 which I created by double-quoting the jq input stream.

dudeOnMac:~$ jq -e '.zoo?' <<< "{"foo": 42, "bar": "less interesting data"}"
jq: error: Could not open file interesting: No such file or directory
jq: error: Could not open file data}: No such file or directory
dudeOnMac:~$ echo $?
2

Answer 3

Given that jq is the way it is, and in particular that it is stream-oriented, I'm inclined to think that a better approach would be to define and use one or more filters that make the distinctions you want. Thus rather than writing .a to access the value of a field, you'd write get("a") assuming that get/1 is defined as follows:

def get(f): if has(f) then .[f] else error("\(type) is not defined at \(f)") end;

Now you can easily tell whether or not an object has a key, and you're all set to go. This definition of get can also be used with arrays.

Answer 4

I've added an updated solution below

The fundamental problem here is that when try to retrieve a value from an object using the .key or .[key] syntax, jq — by definition — can't distinguish a missing key from a key with a value of null.

You can instead define your own lookup function:

def lookup(k):if has(k) then .[k] else error("invalid key") end;

Then use it like so:

$ jq 'lookup("a")' <<<'{}' ; echo $?
jq: error (at <stdin>:1): invalid key
5

$ jq 'lookup("a")' <<<'{"a":null}' ; echo $?
null
0

If you then use lookup consistently instead of the builtin method, I think that will give you the behaviour you want.

---

Here's another way to go about it, with less bash and more jq.

#!/bin/bash

lib='def value(f):((f|tojson)//error("no such value"))|fromjson;'

users=( steve tom pat jane mary )

Select () {
  local name=$1 filter=$2 input=$3
  local -i status=0
  result=$( jq --arg name "$name" "${lib}value(${filter})" <<<$input  2>/dev/null )
  status=$? 
  (( status )) && result="***ERROR***"
  printf '%s\t%d %s\n' "$name" $status "$result"
}

filter='.[]|select(.user == $name)|.value'

input='[{"user":"steve","value":false},
        {"user":"tom","value":true},
        {"user":"pat","value":null},
        {"user":"jane","value":""}]'

for name in "${users[@]}"
do
  Select "$name" "$filter" "$input"
done

This produces the output:

steve   0 false
tom     0 true
pat     0 null
jane    0 ""
mary    5 ***ERROR***

This takes advantage of the fact the absence of input to a filter acts like empty, and empty will trigger the alternative of //, but a string — like "null" or "false" — will not.

It should be noted that value/1 will not work for filters that are simple key/index lookups on objects/arrays, but neither will your solution. I'm reasonably sure that to cover all the cases, you'd need something like this (or yours) and something like get or lookup.