Filtering Nested Arrays with jq: A Practical Guide

Introduction

When working with JSON data, you often encounter nested arrays that require filtering based on specific conditions. jq is a powerful command-line tool for processing and transforming JSON data. In this tutorial, we will explore how to filter an array of objects based on values within inner arrays using jq. This skill is particularly useful when dealing with complex data structures in scripting or data analysis tasks.

Understanding the Problem

Consider a scenario where you have a JSON array containing multiple objects, each with an Id and a nested Names array. Your goal is to filter these objects based on whether any of the names within the Names array contain a specific substring, such as "data". We aim to retrieve only those Ids whose corresponding Names do not include this substring.

Here’s an example JSON input:

[
  {
    "Id": "cb94e7a42732b598ad18a8f27454a886c1aa8bbba6167646d8f064cd86191e2b",
    "Names": [
      "condescending_jones",
      "loving_hoover"
    ]
  },
  {
    "Id": "186db739b7509eb0114a09e14bcd16bf637019860d23c4fc20e98cbe068b55aa",
    "Names": [
      "foo_data"
    ]
  },
  {
    "Id": "a4b7e6f5752d8dcb906a5901f7ab82e403b9dff4eaaeebea767a04bac4aada19",
    "Names": [
      "jovial_wozniak"
    ]
  },
  {
    "Id": "76b71c496556912012c20dc3cbd37a54a1f05bffad3d5e92466900a003fbb623",
    "Names": [
      "bar_data"
    ]
  }
]

Our objective is to extract Ids for objects where none of the names in the Names array contain "data".

Filtering with jq

Using select and map

The first approach involves using jq‘s select and map functions. The select function filters elements based on a condition, while map applies a transformation to each element.

Here’s how you can achieve the desired filtering:

jq 'map(select(.Names[] | contains("data")) | .Id) | map(select(length == 0)) | add' input.json

Explanation:

1. Map and Select:

   • map(select(.Names[] | contains("data"))) filters objects where any name in the Names array contains "data".

2. Extract Ids:

   • .Id extracts the Id of these filtered objects.

3. Invert Selection:

   • map(select(length == 0)) selects objects that were not included in the previous filter, effectively inverting the selection.

4. Combine Results:

   • add combines the results into a single array.

Using any/2

An alternative method uses the any/2 function to determine if any element in an array satisfies a condition:

jq 'map(select(any(.Names[]; contains("data")) | not) | .Id)' input.json

Explanation:

1. Iterate and Check:

   • .Names[] | contains("data") checks each name for the substring "data".

2. Use any/2:

   • any(.Names[]; contains("data")) returns true if any name matches, otherwise false.

3. Invert Condition:

   • | not inverts the condition to select objects where no names match.

4. Extract Ids:

   • .Id extracts the Id of these selected objects.

Using select with any

Another approach involves using a combination of select and any:

jq '.[] | select([ .Names[] | contains("data") ] | any) | .Id' input.json | jq -s 'map(select(. != null))'

Explanation:

1. Iterate Over Objects:

   • .[] iterates over each object in the array.

2. Check for Substring:

   • [ .Names[] | contains("data") ] | any checks if any name contains "data".

3. Select and Extract Ids:

   • select(...) filters objects where the condition is true, and .Id extracts their IDs.

4. Filter Non-null Results:

   • jq -s 'map(select(. != null))' removes any null results from the output.

Conclusion

Filtering nested arrays in JSON using jq can be achieved through various methods, each leveraging different functions like select, map, and any. Understanding these techniques allows you to efficiently process complex data structures. By practicing with examples, you’ll become proficient in extracting specific information tailored to your needs.

Best Practices

• Test Incrementally: Break down your jq expressions into smaller parts and test them incrementally to ensure correctness.
• Use jqplay.org: This online tool is excellent for experimenting with jq queries interactively.
• Refer to the jq Manual: The official manual provides comprehensive details on all available functions and operators.

With these techniques, you can handle intricate JSON data transformations effectively using jq.