Cloudformation template to export the dataset into S3 Location

Hi All,

Created a Cloudformation template, which exports the subscribed dataset latest revision to S3 bucket. Lambda job looks for the latest revision and uses api 'ExportAssetsToS3' to write the data into s3 bucket. I am trying to achieve, lambda job to write the data to a location with prefix like - bucket/prefix/files .

Prefix is like current date and it should be changed for everyday.

In the ExportAssetsToS3, in AssetDestinations passed Key : '/2020-08-06/ and error came as below.

[ERROR] ValidationException: An error occurred (ValidationException) when calling the CreateJob operation: The request was rejected because it attempted to export distinct assets to the same target Amazon S3 object. When no key is specified in the request, Asset name is used as the default target key.
Traceback (most recent call last):
File "/var/task/index.py", line 69, in handler
{ 'AssetId': asset['Id'], 'Bucket': destination_bucket, 'Key': '/2020-08-06/' } for asset in assets_chunk
File "/var/runtime/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 626, in _make_api_call
raise error_class(parsed_response, operation_name)

Ex: Bucket - ABC
Prefix: /2020-08-06/
Final S3 location - ABC/2020-08-06/file.txt

Is there any approach to add the prefix to the S3 bucket.

Thanks,
Naveen J

Hi All,

Here is the Solution -

Change the code in export_job, AssetDestinations as below

'AssetDestinations': [
{ 'AssetId': asset['Id'], 'Bucket': destination_bucket, 'Key': '
abc/2020-08-06/' + asset['Name'] } for asset in assets_chunk ]

Thanks,
Naveen J

Edited by: naveen838 on Aug 7, 2020 9:33 AM

Edited by: naveen838 on Aug 7, 2020 9:33 AM