How to create a kinesis firehose delivery stream with dynamic partitions enabled using python cdk?

0

I am trying to create a firehose delivery stream with dynamic partitions enabled. Below is what I have got so far.

analytics_delivery_stream = kinesisfirehose.CfnDeliveryStream(
    self, "AnalyticsDeliveryStream",
    delivery_stream_name='analytics',
    extended_s3_destination_configuration=kinesisfirehose.CfnDeliveryStream.ExtendedS3DestinationConfigurationProperty(
        bucket_arn=f'arn:aws:s3:::{analytic_bucket_name}',
        buffering_hints=kinesisfirehose.CfnDeliveryStream.BufferingHintsProperty(
            interval_in_seconds=60
        ),
        dynamic_partitioning_configuration = kinesisfirehose.CfnDeliveryStream.DynamicPartitioningConfigurationProperty(
        enabled=True,
        retry_options=kinesisfirehose.CfnDeliveryStream.RetryOptionsProperty(
            duration_in_seconds=123
        )),
        compression_format="UNCOMPRESSED",
        role_arn=firehose_role.role_arn,
        prefix="!{partitionKeyFromQuery:log_type}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/",
        error_output_prefix="errors/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}/",
    )
)

When I run this, I get below error .

Processing Configuration is not enabled when DataPartitioning is enabled.

I found below references to Processing Configuration in the docs

processing_configuration=kinesisfirehose.CfnDeliveryStream.ProcessingConfigurationProperty(
    enabled=False,
    processors=[kinesisfirehose.CfnDeliveryStream.ProcessorProperty(
        type="type",

        # the properties below are optional
        parameters=[kinesisfirehose.CfnDeliveryStream.ProcessorParameterProperty(
            parameter_name="parameterName",
            parameter_value="parameterValue"
        )]
    )]
),

I am not sure what values to put for parameters or type inside processing_configuration.

I have logs being put into firehose with below structure.

type A - {'log_type':'type_A_log',....other props....}

type B - {'log_type':'type_B_log',....other props....}

Using dynamic partitioning, I want to achieve the scenario where all logs of type A go into type_A_log directory inside s3 and type B log into type_B_log directory.

Can someone please help here ? I am going down a rabbithole.

Naxi
gefragt vor 2 Jahren138 Aufrufe
Keine Antworten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen