How to set Content-Type to application/json when uploading files via AWS IoT Greengrass Stream Manager to S3?

Hello there,

I understand that Stream Manager sets the System defined Content-Type to application/octet-stream instead of application/json, which prevents Athena from properly parsing the JSON files.

Kindly note that When a new file is uploaded to the s3 by default the "Content-Type" is set to "application/octet-stream" as system defined metadata and so Stream Manager sets the System defined Content-Type to application/octet-stream.

For your specific issue with Athena not properly parsing JSON files due to the incorrect Content-Type, here are some potential solutions:

Post-transfer metadata update: After Stream Manager uploads the files to S3, you could update the content type.

You can follow the below solution to do so .

1. Setup S3 event notifications for ‘PutObject’ events. - Amazon S3 Event Notifications - https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html

2. 2. Setup a Lambda function that updates object metadata. This Lambda could be invoked by S3 event notifications. Sample Lambda functions:
   • https://stackoverflow.com/questions/49766843/update-amazon-s3-object-meta-data-with-lambda-without-performing-a-object-copy
   • https://github.com/tooltwist/documentation/wiki/Configure-AWS-Lambda-to-Automatically-Set-Cache-Control-Headers-on-S3-Objects
3. 3. Each time an object is uploaded. A notification would be generated which would act as a trigger for the Lambda function that contains the code to update object metadata.

You also have the option to do this selectively for each object, or modify all the objects recursively in a particular prefix or entire bucket at once with a ‘--recursive’ flag, depending on your case.

Here is a another CLI command template for doing so: aws s3 cp <file_name> s3://bucket-name/prefix --metadata-directive REPLACE <all preferred metadata>

For example, if you wish to add/update the ‘content-type’ header to all objects in a prefix, you could use the ‘metadata-directive’ as shown in this command: aws s3 cp s3://your-bucket/path/yourfile.json s3://your-bucket/path/yourfile.json --content-type "application/json"

You can refer the below documentation for more information [+] https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html

----Woraround#2: Custom CLI/Python Script or AWS SDK custom code---

You can create a custom script that would automate updating of metadata for select objects that are uploaded to your bucket every-day.

Here’s an overview of this script would work:

1. List Objects by the date modified query: Sample CLI commands:

aws s3api list-objects-v2 --bucket aydavocartestbucket --query 'Contents[?contains(LastModified, 2021-04-07)].Key'

or

DATE=$(date +%Y-%m-%d) bucket=test-bucket-fh aws s3api list-objects-v2 --bucket "$bucket"
--query 'Contents[?contains(LastModified, '"$DATE"')].Key'

2. Use the output of last command, as an input for a copy operation with --metadata-directive “REPLACE” flag to update the objects with desired metadata

3. Automate the script with a CRON job to run at a desired frequency

• https://opensource.com/article/17/11/how-use-cron-linux

• CRON alternative for windows- schtasks: https://docs.microsoft.com/en-us/windows/win32/taskschd/schtasks

Alternatively, you could write a custom code or python script to filter the objects by the last modified date and update the object metadata by copying them over themselves.

I hope the above information serves you well.

Thank you.

Based on the information available, there isn't a direct configuration option in AWS IoT Greengrass Stream Manager to set the system-defined Content-Type metadata when uploading files to S3. The issue you're experiencing is a known behavior where certain AWS services like DataSync and Stream Manager often default to "application/octet-stream" as the ContentType for uploaded files.

For your specific issue with Athena not properly parsing JSON files due to the incorrect Content-Type, here are some potential solutions:

1. Post-transfer metadata update: After Stream Manager uploads the files to S3, you could implement a Lambda function triggered by S3 object creation events to update the ContentType metadata. You can use the AWS CLI or SDK to update the ContentType:

aws s3 cp s3://your-bucket/path/yourfile.json s3://your-bucket/path/yourfile.json --content-type "application/json"

2. Alternative approach: If the post-processing solution isn't feasible, you might need to consider alternative approaches for your data pipeline. You could:

• Continue using Lambda with IoT Rules but implement a local buffering mechanism
• Use AWS IoT Core's message queuing capabilities for offline scenarios
• Implement a custom solution that handles both buffering and proper metadata setting

3. Athena configuration: Check if there are any Athena/Glue crawler configurations that can be adjusted to handle files with incorrect Content-Type but valid JSON content.

The user-defined metadata you're setting in your S3ExportTaskDefinition is being correctly applied, but as you've observed, Athena appears to be using the system-defined Content-Type for determining how to parse the file.

Stream Manager's primary focus is on efficiently transferring data while preserving core file attributes, but it doesn't always preserve all metadata like MIME types exactly as needed for services like Athena.
Sources
DataSync: wrong ContentType for CSV file in S3 | AWS re:Post
Configure AWS IoT Greengrass stream manager - AWS IoT Greengrass