I'm trying to set up custom sources for Security Lake and have a Kinesis Firehose delivery stream configured to deliver parquet files into the Security Lake bucket under the ext/ prefix.
The problem I'm encountering is that the AWS::KinesisFirehose::DeliveryStream SchemaConfiguration (under CloudFormation AWS::KinesisFirehose::DeliveryStream Properties->ExtendedS3DestinationConfiguration->DataFormatConversionConfiguration->SchemaConfiguration) requires a table with schema matching the records in order to deliver the data, but Security Lake requires there to be records already there for Glue to crawl and create the table and schema which Firehose needs. It looks like it's not possible to directly stream data using Firehose when setting up Security Lake custom sources and that we need to move the records under the ext/ prefix using Glue or EMR.
Is my conclusion correct that this will not work? The fact that Security Lake uses a Glue crawler to create the table makes it less flexible when creating a custom source.
To summarize the catch 22: