By using AWS re:Post, you agree to the Terms of Use

How to create a kinesis firehose delivery stream with dynamic partitions enabled using python cdk?

0

I am trying to create a firehose delivery stream with dynamic partitions enabled. Below is what I have got so far.

analytics_delivery_stream = kinesisfirehose.CfnDeliveryStream(
    self, "AnalyticsDeliveryStream",
    delivery_stream_name='analytics',
    extended_s3_destination_configuration=kinesisfirehose.CfnDeliveryStream.ExtendedS3DestinationConfigurationProperty(
        bucket_arn=f'arn:aws:s3:::{analytic_bucket_name}',
        buffering_hints=kinesisfirehose.CfnDeliveryStream.BufferingHintsProperty(
            interval_in_seconds=60
        ),
        dynamic_partitioning_configuration = kinesisfirehose.CfnDeliveryStream.DynamicPartitioningConfigurationProperty(
        enabled=True,
        retry_options=kinesisfirehose.CfnDeliveryStream.RetryOptionsProperty(
            duration_in_seconds=123
        )),
        compression_format="UNCOMPRESSED",
        role_arn=firehose_role.role_arn,
        prefix="!{partitionKeyFromQuery:log_type}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/",
        error_output_prefix="errors/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}/",
    )
)

When I run this, I get below error .

Processing Configuration is not enabled when DataPartitioning is enabled.

I found below references to Processing Configuration in the docs

processing_configuration=kinesisfirehose.CfnDeliveryStream.ProcessingConfigurationProperty(
    enabled=False,
    processors=[kinesisfirehose.CfnDeliveryStream.ProcessorProperty(
        type="type",

        # the properties below are optional
        parameters=[kinesisfirehose.CfnDeliveryStream.ProcessorParameterProperty(
            parameter_name="parameterName",
            parameter_value="parameterValue"
        )]
    )]
),

I am not sure what values to put for parameters or type inside processing_configuration.

I have logs being put into firehose with below structure.

type A - {'log_type':'type_A_log',....other props....}

type B - {'log_type':'type_B_log',....other props....}

Using dynamic partitioning, I want to achieve the scenario where all logs of type A go into type_A_log directory inside s3 and type B log into type_B_log directory.

Can someone please help here ? I am going down a rabbithole.

No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions