JQ Parsing a json stream input - Amazon Firehose

The error message suggests there is a syntax issue with the jq expression used in AWS Firehose for dynamic partitioning. The problem might be related to how the jq expression is interpreted when embedded in a larger JSON structure. Let's break down the solution to ensure it aligns with the jq syntax and Firehose's requirements.

Here's how you can structure your jq expression correctly to derive the first level partition key (metric or report) and the k1 value.

Determine the parent key (metric or report):

jq

if has("metric") then "metric" else "report" end

Extract the value of k1 depending on the parent key:

jq

if has("metric") then .metric.k1 else .report.k1 end

To create a dynamic partition key, you need to ensure the entire expression is valid when used in Firehose. One way to do this is to output a JSON object with the desired fields. Here is how you can combine these steps in a jq expression:

jq

{
  "dataType": if has("metric") then "metric" else "report" end,
  "k1": if has("metric") then .metric.k1 else .report.k1 end
}

Ensure that this jq expression is correctly quoted and escaped when defining it in Firehose's configuration. Here is how you can define it in the Firehose dynamic partitioning key configuration:

json

{
  "MetadataExtractionQuery": "{\"dataType\":if has(\"metric\") then \"metric\" else \"report\" end,\"k1\":if has(\"metric\") then .metric.k1 else .report.k1 end}"
}

This JSON ensures the jq expression is correctly embedded within the Firehose configuration. Note the escape characters for quotes inside the jq expression.

Summary Verify your jq expression syntax is correct. Ensure the expression is properly escaped and formatted within the Firehose configuration. Try updating your Firehose configuration with the above MetadataExtractionQuery and it should resolve the issue you are encountering.