How to set or replace object meta data in AWS S3 batch Copy Operations ?
Hi. I am trying to learn what all best solution for doing data from Bucket-A to copy in Bucket-B with AWS S3 batch operations. Initially I did generate Manifesto file from Bucket Inventory Management, included all versions.
Now, while creating AWS S3 batch operation with Copy action or probably not sure which I can set to.
after setting up Source bucket, reports bucket path & destination bucket path.
When I reached to Operations section, I do thought that selecting Replace all object meta data with VersionId, possibly bring me old versionIds some where in my new bucket in terms of x-aws-meta-x-versionId. as screen shot attach.
But this couldnt helped me out, may be not sure how I can use this property. I tried several different patterns but eventually it treat it as a String value and not expression. So, please help me how I can achieve this use case with existing options ?
- Newest
- Most votes
- Most comments
Hi Dnyaneshwar,
If your question is simply copying data from one S3 bucket to another S3 bucket then follow this article: https://repost.aws/knowledge-center/move-objects-s3-bucket
If you want to set or replace object metadata in AWS S3 batch copy operations then follow below:
When creating a batch copy job in S3 Batch Operations, you can specify metadata to set or replace on the destination objects. The metadata can include both system-defined metadata like Content-Type as well as user-defined metadata.
Some options for setting metadata include:
Specify metadata in the job manifest file. The manifest is an XML or JSON file that lists the source objects and actions to take for each one. It allows setting metadata for each object copy.
Use the s3api copy-object command with the --metadata option. This allows replacing or adding metadata on the destination objects. You can run this command on multiple objects in batch mode.
Edit metadata after the copy operation using the s3api put-object-tagging and s3api put-object-acl commands. This performs the metadata changes after the copy is complete in two separate steps.
Few links: [1] Operations supported by S3 Batch Operations - Amazon Simple Storage Service - https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-operations.html [2] Editing object metadata in the Amazon S3 console - Amazon Simple Storage Service - https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-object-metadata.html
Hope this helps.
Relevant content
AWS OFFICIALUpdated 2 years ago- AWS OFFICIALUpdated 7 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 months ago
So, here is the case to elaborate more in detail.
Bucket-A ObjectKey: cefd83de-5b5e-4e34-b2ef-1a0499d13bbe
VersionIds: v7XbN99y_7Cw_BcYryxg6S7gabwHITBV 3vNisZsImeYQl1eIhd2ueQxUk2qrDB3j ISaKHenlzUktgB76OVFMun6GtSf6IawK Mai9HRaHVVk4Wk1ndIruBLa9CUc7t3Xu gZMc9nFi_pzgOqmxkRQbSY1UGDDKCNyg
On Migration or Copying or Replication Bucket-B ObjectKey: cefd83de-5b5e-4e34-b2ef-1a0499d13bbe
VersionIds: Object Meta Data (Key: x-amz-meta-x-version) sdfs_23d4e34f_dfsds_23dw443567Abc v7XbN99y_7Cw_BcYryxg6S7gabwHITBV Hhdfs_khljodf_dfsds_23dw443567kkk 3vNisZsImeYQl1eIhd2ueQxUk2qrDB3j sd4e34df34f_dfsds_23dw2343ffsdcfk ISaKHenlzUktgB76OVFMun6GtSf6IawK sdfs_dfdfcdf_dfsds_23dw443567thy0 Mai9HRaHVVk4Wk1ndIruBLa9CUc7t3Xu 4e34_dppfcdf_dfsds_23dw443567frmm gZMc9nFi_pzgOqmxkRQbSY1UGDDKCNyg This is what I am trying to achieve it, that link somehow the old versionId to the newly created versionIds respectively.
I think you are suggesting to use s3api copy-object command, but the concern here is wanted to leverage AWS S3 batch operations mostly, wanted to keeping plan-c for script based handling.
I was just following this re-post: https://repost.aws/questions/QUEiHluFqsQqOQtBrEQU07ow/s3-batch-operations-lambda-to-bulk-rename-objects where it is mentioned that manifesto JSON file can be made such changes to pick some values. @vatsravi any thought on this ?
This is how my CSV file looks like (Formatting may be not correct here but I hope its readable , please excuse me if this is not clean will update)
dev-djool-xyubd4 09c0f49d-6fde-4fb3-8b5d-0712a87b38a2 A0R3_1XRkN4trkDAAC0k_OQOsW2ZkE0c TRUE FALSE 2024-04-05T09:54:41.000Z 6c77fefb538025cabfaf8d9b38d00faa dev-djool-xyubd4 09c0f49d-6fde-4fb3-8b5d-0712a87b38a2 iWH_.ve8lcAcgqlujJtD6TTY3yQk_BuF FALSE FALSE 2024-04-05T09:54:41.000Z 6c77fefb538025cabfaf8d9b38d00faa
manifesto.json { "sourceBucket" : "dev-djool-xyubd4", "destinationBucket" : "arn:aws:s3:::dev-djool-s3-reports", "version" : "2016-11-30", "creationTimestamp" : "1712797200000", "fileFormat" : "CSV", "fileSchema" : "Bucket, Key, VersionId, IsLatest, IsDeleteMarker, LastModifiedDate, ETag", "files" : [ { "key" : "dev-djool-xyubd4/dev-djool-xyubd4-copy-job/data/4dba9155-e423-4352-b031-191c26e01cae.csv.gz", "size" : 29959, "MD5checksum" : "a50e27f226b1a68a7496466c9303bd6a" } ] }