Kinesis Firehose Dynamic Partitoning with inline jq expression
Following the aws blog post : Dynamic Partitioning.
So I have a firehose delivery stream configured to convert api calls into parquet output to s3 - and now partitioned according to the value of a field from within my data - 'playtype'.
However, in some rows the value in this partioning field is null/empty and this causes those rows to error. Therefore I want to adjust the jq expression for the dynamic partioning key to substitute a default value for those null cases. Having trouble with the syntax though.
JQ expression specified for the field is .playtype
So I have tried things like:
.playtype // "some-hardcoded-default" .playtype | if . == null or . == "" then "some-hardcoded-default" else . end
These evalute as valid jq in jqplay - but in practice i.e. in firehose are returning a 'jq syntax error'. How can i specify this logic correctly? i.e. partition on this field but if it the value is null or empty then use 'some-hardcoded-default'. Thankyou!
(.playtype | if . == null or . == "" then "some-hardcoded-default" else . end)
put it inside the brackets , it will work .
Hi there, please try as per above user suggestion in the comments and let us know the results. jqplay is currently blocked on our end so couldn't check the validity. I believe having brackets for each expression should fix the issue. However if that doesn't work, I would recommend you to resort to using a transformation Lambda function like mentioned here: https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html in the case where field is not present in the incoming data. That way you would have a lot more control on the fields.
AWS-User-4356880 is correct! Many thanks. I had used a transformation lambda function as you also suggest and this works too but the former was a much simpler, quicker way to go.
Cost optimisation for Kinesis FirehoseAccepted Answerasked 3 years ago
Kinesis Firehose Dynamic Partitoning with inline jq expressionAccepted Answerasked 5 months ago
Using JQ expression in S3 backup bucket prefix in Backup settings and S3 bucket error output prefix in Amazon S3 destination while configuring Kinesis Firehose delivery streamasked 5 months ago
How to create a kinesis firehose delivery stream with dynamic partitions enabled using python cdk?
Multiple Kinesis Firehose DestinationsAccepted Answerasked 2 years ago
[Kinesis] Dynamic shardingAccepted Answerasked 2 years ago
Do I need to create two kinesis delivery streams to send messages under two different paths in the same s3 bucket?
Kinesis Dynamic Partitioning "Non JSON record provided" errorasked 7 months ago
Multiple Kinesis Data Analytics apps to use the same Kinesis firehose delivery stream as sourceAccepted Answerasked a year ago
Monitoring AWS SES Sent Emails using Kinesis Firehoseasked 15 days ago