How to specify metadata in the job manifest file.

Question

I am trying to use AWS S3 Batch operations to copy all versioned items from `Bucket-1` to `Bucket-2`.
I am using AWS S3 bucket Management > Inventory Management to generate Manifesto JSON file.

And then apply this JSON file in AWS S3 Batch Operation to copy data.

Additionally wanted to specify Object meta data from Manifesto file itself associated when item is copied to target location.

Query is How I can make changes in CSV file to mentioned to use this Object Meta data.
For example wanted to set `x-amz-meta-x-version` where the value should be old `versionId`

here is my JSON file looks like
**manifesto.json**
```
{
  "sourceBucket" : "dev-djool-xyubd4",
  "destinationBucket" : "arn:aws:s3:::dev-djool-s3-reports",
  "version" : "2016-11-30",
  "creationTimestamp" : "1712797200000",
  "fileFormat" : "CSV",
  "fileSchema" : "Bucket, Key, VersionId, IsLatest, IsDeleteMarker, LastModifiedDate, ETag",
  "files" : [ {
    "key" : "dev-djool-xyubd4/dev-djool-xyubd4-copy-job/data/4dba9155-e423-4352-b031-191c26e01cae.csv.gz",
    "size" : 29959,
    "MD5checksum" : "a50e27f226b1a68a7496466c9303bd6a"
  } ]
}

```
and my CSV file sample data

```
dev-djool-xyubd4	09c0f49d-6fde-4fb3-8b5d-0712a87b38a2	A0R3_1XRkN4trkDAAC0k_OQOsW2ZkE0c	TRUE	FALSE	2024-04-05T09:54:41.000Z	6c77fefb538025cabfaf8d9b38d00faa
dev-djool-xyubd4	09c0f49d-6fde-4fb3-8b5d-0712a87b38a2	iWH_.ve8lcAcgqlujJtD6TTY3yQk_BuF	FALSE	FALSE	2024-04-05T09:54:41.000Z	6c77fefb538025cabfaf8d9b38d00faa

```

Accepted Answer

> 💡 You can add the desired metadata fields to the `fileSchema` section of the JSON file. For example, to include a custom `x-amz-meta-x-version` field, you would update the `fileSchema` to include that field, and then populate the corresponding column in the CSV data file with the appropriate version IDs.

#### Example:
```json
{
  "sourceBucket" : "dev-djool-xyubd4",
  "destinationBucket" : "arn:aws:s3:::dev-djool-s3-reports",
  "version" : "2016-11-30",
  "creationTimestamp" : "1712797200000",
  "fileFormat" : "CSV",
  "fileSchema" : "Bucket, Key, VersionId, IsLatest, IsDeleteMarker, LastModifiedDate, ETag, x-amz-meta-x-version",
  "files" : [ {
    "key" : "dev-djool-xyubd4/dev-djool-xyubd4-copy-job/data/4dba9155-e423-4352-b031-191c26e01cae.csv.gz",
    "size" : 29959,
    "MD5checksum" : "a50e27f226b1a68a7496466c9303bd6a"
  } ]
}
```

#### Key Source:
- [User-defined object metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata)
- [Editing object metadata in the Amazon S3 console](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-object-metadata.html)

Answer

Hi

Please follow the steps outlined in this [re:Post thread](https://repost.aws/questions/QUwNmsKUz1RsqR1eCgBXGmrA/how-to-work-with-md5checksum-for-manually-edited-manifesto-json-file#ANlmxogDq6RFGHy91CW0PfWA) and let me know if you face any difficulty.

I'd be more than happy to assist if there are any additional questions or challenges you see while following the steps.

**Edit:**

Earlier I missed the x-amz-meta-x-version part, which you mentioned. To better understand, I would prefer to see the inventory configuration settings and also will see if this metadata is tied to all the objects in your s3 bucket/prefix. It'd be tedious to remove the metadata from each object and also it'd create a new version of the object if you modify/add/remove the metadata. Hence, I'd prefer to remove "x-amz-meta-x-version" from manifest.json fileschema, save this and get a new checksum and size of manifest.json and update those in manifest.checksum file. Upload these files in the respective lcation and then run your s3 batch job. Long term, if metadata of objects are supposed to be that way only, then you may need to figure out the purpose of this user defined metadata and is it really required.

Let me know if it doesn't work(I'm sure it'd solve your problem), happy t o help further.

Abhishek

How to specify metadata in the job manifest file.

Example:

Key Source:

Relevant content