I need to pre-process some data on S3 before the Glue Crawler crawls the data. For this I created an S3 Object Lambda to do the pre-processing. If I test the Object Lambda using the CLI, it provides the correct (processed) file.
I then tried to use the ARN of the Object Lambda as the data store. However the Include Path prompt will only accept a value starting with 's3://'.
I also tried to use an Access Point alias, but I do not see the alias for a Object Lambda Access Point. (If I use the alias for the supporting access point, I get the un-processed data from the S3 bucket.)
As a last resort, I also tried to use terraform to set the data store. This did work, but when I run the crawler I get an error. I assume this is also because the crawler is expecting a path starting with 's3://' and not an ARN.
Is there a different way to use an Object Lambda as the Data Store for a Glue Crawler?
