jq
Transform JSON data using jq - the powerful JSON processor.
Overview
Transform JSON data using jq - the powerful JSON processor. This step applies jq filters to event data, enabling complex JSON transformations, queries, and restructuring. jq is a lightweight and flexible command-line JSON processor with its own query language. Use it to extract fields, transform arrays, filter data, compute aggregations, or reshape JSON structures. The input can be read from a specific field path (or defaults to the entire event), processed through your jq filter, and the result is written to an output field (defaulting to replacing the event). If the input field doesn't exist, the event passes through unchanged. When writing back to the entire event, the jq result must be a mapping.
Quick Start
steps:
- type: jq
filter: <string>Configuration
| Parameter | Type | Required | Description |
|---|---|---|---|
filter | string | Yes | jq program applied to the resolved input value (e.g. '.items[0].id'). |
input_from | string | No | Dot path to the JSON value used as jq input (e.g. 'data.payload'). When omitted, jq receives the entire event. |
output_to | string | No | Dot path where the jq result will be written (created if missing). When omitted, the jq result must be a mapping and replaces the entire event. |
input_field | string | No | Deprecated. Use `input_from` instead. |
output_field | string | No | Deprecated. Use `output_to` instead. |
Examples
Reshape entire event
Use jq defaults to read and write the full event payload
type: jq
filter: "{user: {id: .user_id, email: .details.email}}"
Extract specific fields
Create a summary object with only selected fields
type: jq
input_from: user
filter: "{name: .name, email: .email, id: .id}"
output_to: user_summary
Transform array of objects
Map over an array to extract and rename fields
type: jq
input_from: customers
filter: "[.[] | {customer_id: .id, full_name: .name, contact: .email}]"
output_to: customer_list
Filter and sort
Filter array items by condition and sort results
type: jq
input_from: orders
filter: "[.[] | select(.status == \"completed\")] | sort_by(.amount) | reverse"
output_to: top_orders
Calculate aggregations
Compute sum, count, and average from array data
type: jq
input_from: transactions
filter: "{total: map(.amount) | add, count: length, average: (map(.amount) | add / length)}"
output_to: transaction_stats
Nested data extraction
Flatten nested structures and extract deeply nested values
type: jq
input_from: api_response
filter: ".data.results[] | {id: .id, title: .attributes.title, author: .relationships.author.data.name}"
output_to: articles
Advanced Options
These options are available on all steps for error handling and retry logic:
| Parameter | Type | Default | Description |
|---|---|---|---|
retries | integer | 0 | Number of retry attempts (0-10) |
backoff_seconds | number | 0 | Backoff (seconds) applied between retry attempts |
retry_propagate | boolean | false | If True, raise last exception after exhausting retries; otherwise swallow. |