Skip to content

AWS legal hold across S3 bucket

0

Hi, I am using nuget package for .net core project: AWSSDK.S3 Is it possible to apply legal hold across all the objects in a bucket in 1 API call? Or, is it possible to apply the same to a folder? Example, Suppose I have a bucket: my-bucket It has a folder: MyFiles with 10000 txt files in it. Instead of applying legal hold to each of them individually, can I do it in bulk? In my use case, I may have terabytes of data for legalhold. Hence the question of efficiency

asked 2 months ago70 views
3 Answers
0

Based on the AWS SDK for .NET, there is no single API call to apply legal hold across all objects in a bucket or to a folder. The PutObjectLegalHold operation works at the individual object level only.

To apply legal hold to multiple objects, you would need to call PutObjectLegalHoldAsync for each object individually. This means for your scenario with 10,000 files in the MyFiles folder, you would need to make 10,000 separate API calls.

For efficiency with large-scale operations like yours involving terabytes of data, you should consider:

  1. Parallel processing: Use asynchronous operations with Task.WhenAll or Parallel.ForEach to process multiple objects concurrently, which can significantly speed up the operation.

  2. Batch processing: Process objects in batches rather than all at once to manage memory and API rate limits effectively.

  3. S3 Inventory: Use S3 Inventory to get a list of all objects first, then process them systematically.

  4. Consider S3 Batch Operations: While not directly available through the PutObjectLegalHold API, AWS S3 Batch Operations is a service designed for performing large-scale batch operations on S3 objects, though you'd need to check if it supports legal hold operations for your use case.

The SDK provides the ModifyObjectLegalHold method that accepts the bucket name, object key, and hold status, but this must be called individually for each object you want to protect with a legal hold.
Sources
Use PutObjectLegalHold with an AWS SDK or CLI - Amazon Simple Storage Service
Use PutObjectLegalHold with an AWS SDK or CLI - AWS SDK Code Examples
Amazon S3 examples using SDK for .NET - AWS SDK Code Examples

answered 2 months ago
EXPERT
reviewed 2 months ago
0

Hello.

Instead of using the API, why not try using batch operations as described in the following document?
This allows you to process multiple objects at once.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-legal-hold.html

EXPERT
answered 2 months ago
0

To apply legal hold to all objects in an S3 bucket at scale, the easiest approach is to use S3 Batch Operations.

How it works

First, make sure your bucket has Object Lock enabled (this must be set at bucket creation) and that your IAM role includes the s3:PutObjectLegalHold and s3:GetBucketObjectLockConfiguration permissions.

Then, create a manifest listing all target objects — you can generate this using S3 Inventory or a CSV file — and submit a Batch Operations job with the operation type set to S3PutObjectLegalHold (ON). The job will run until it reaches a completed, canceled, or failed state.

AWS
SUPPORT ENGINEER
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.