Questions tagged with Amazon Kinesis

Content language: English

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

How to create a kinesis firehose delivery stream with dynamic partitions enabled using python cdk?

I am trying to create a firehose delivery stream with dynamic partitions enabled. Below is what I have got so far. ``` analytics_delivery_stream = kinesisfirehose.CfnDeliveryStream( self, "AnalyticsDeliveryStream", delivery_stream_name='analytics', extended_s3_destination_configuration=kinesisfirehose.CfnDeliveryStream.ExtendedS3DestinationConfigurationProperty( bucket_arn=f'arn:aws:s3:::{analytic_bucket_name}', buffering_hints=kinesisfirehose.CfnDeliveryStream.BufferingHintsProperty( interval_in_seconds=60 ), dynamic_partitioning_configuration = kinesisfirehose.CfnDeliveryStream.DynamicPartitioningConfigurationProperty( enabled=True, retry_options=kinesisfirehose.CfnDeliveryStream.RetryOptionsProperty( duration_in_seconds=123 )), compression_format="UNCOMPRESSED", role_arn=firehose_role.role_arn, prefix="!{partitionKeyFromQuery:log_type}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/", error_output_prefix="errors/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}/", ) ) ``` When I run this, I get below error . `Processing Configuration is not enabled when DataPartitioning is enabled. ` I found below references to Processing Configuration in the docs ``` processing_configuration=kinesisfirehose.CfnDeliveryStream.ProcessingConfigurationProperty( enabled=False, processors=[kinesisfirehose.CfnDeliveryStream.ProcessorProperty( type="type", # the properties below are optional parameters=[kinesisfirehose.CfnDeliveryStream.ProcessorParameterProperty( parameter_name="parameterName", parameter_value="parameterValue" )] )] ), ``` I am not sure what values to put for **parameters** or **type** inside processing_configuration. I have logs being put into firehose with below structure. type A - {'log_type':'type_A_log',....other props....} type B - {'log_type':'type_B_log',....other props....} Using dynamic partitioning, I want to achieve the scenario where all logs of type A go into type_A_log directory inside s3 and type B log into type_B_log directory. Can someone please help here ? I am going down a rabbithole.
0
answers
0
votes
46
views
Naxi
asked 6 months ago

AWS IoT Greengrass (V2) and Video Streaming

Hello, The use case I have is this - There are two types of AWS IoT Greengrass V2 core devices that are implemented, which are connected (in the same private LAN network) in hub and spoke architecture. None of them are connected to client devices (Greengrass is being used because of its IPC and orchestration benefits): 1. [Spoke] AWS IoT Greengrass V2 core device is directly attached to a camera. The Video stream is sent to an Hub AWS IoT Greengrass V2 core device for ML processing (inference) that must be near-real time. 2. [Hub] AWS IoT Greengrass V2 core device that is processing and Fan-Out video streams: **A)** to ML inference interface (**local component of the hub**) **B)** to Kinesis Firehose (S3; to re-train the model) **C)** AWS Kinesis Video Stream (for human to view the video online) I have a couple of questions: 1. Is the architecture feasible? Make sense? 2. What is the best (performance and security wise) technology (open source, AWS component, protocol) to use in Spoke and Hub devices to send the video stream from the spokes to the hub (the video has to be high quality with minimal/no compression to keep the inference accuracy high)? 2. Can the Stream Manager component of AWS IoT Greengrass V2 core send (Hub) streams in fun-out mode (**e.g., to two different destinations concurrently, AWS Kinesis Firehose and AWS Kinesis Video Streams**)? Thank you, Yossi
1
answers
0
votes
138
views
yossico
asked 6 months ago