By using AWS re:Post, you agree to the Terms of Use
/Kinesis Firehose Dynamic Partitoning with inline jq expression/

Kinesis Firehose Dynamic Partitoning with inline jq expression

0

Following the aws blog post : Dynamic Partitioning.

So I have a firehose delivery stream configured to convert api calls into parquet output to s3 - and now partitioned according to the value of a field from within my data - 'playtype'.

However, in some rows the value in this partioning field is null/empty and this causes those rows to error. Therefore I want to adjust the jq expression for the dynamic partioning key to substitute a default value for those null cases. Having trouble with the syntax though.

JQ expression specified for the field is .playtype

So I have tried things like:

.playtype // "some-hardcoded-default"
.playtype | if . == null or . == "" then "some-hardcoded-default" else . end

These evalute as valid jq in jqplay - but in practice i.e. in firehose are returning a 'jq syntax error'. How can i specify this logic correctly? i.e. partition on this field but if it the value is null or empty then use 'some-hardcoded-default'. Thankyou!

  • (.playtype | if . == null or . == "" then "some-hardcoded-default" else . end)

    put it inside the brackets , it will work .

1 Answers
1
Accepted Answer

Hi there, please try as per above user suggestion in the comments and let us know the results. jqplay is currently blocked on our end so couldn't check the validity. I believe having brackets for each expression should fix the issue. However if that doesn't work, I would recommend you to resort to using a transformation Lambda function like mentioned here: https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html in the case where field is not present in the incoming data. That way you would have a lot more control on the fields.

SUPPORT ENGINEER
answered 5 months ago
  • AWS-User-4356880 is correct! Many thanks. I had used a transformation lambda function as you also suggest and this works too but the former was a much simpler, quicker way to go.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions