Glue Job - S3 to S3

Here are some thoughts on handling large file transfers between S3 buckets in AWS Glue:

1. To copy/move an unzipped file from a tar file:

   • Use the tarfile module to open and extract the tar file
   • Loop through the extracted file objects and upload each one to the target S3 bucket using boto3 S3 client

import tarfile
import boto3

s3 = boto3.client('s3')

with tarfile.open('source.tar') as tar:
    for member in tar.getmembers():
        file_obj = tar.extractfile(member)
        s3.upload_fileobj(file_obj, 'target_bucket', member.name)

2. For large files, use multipart uploads:

   • Use s3.create_multipart_upload to initialize a multipart upload
   • Use s3.upload_part in a loop to upload file chunks in parallel
   • After all parts uploaded, use s3.complete_multipart_upload to finish

This allows you to upload large files efficiently in parallel.

3. Other methods to write to S3:

   • Use boto3 resource API instead of client (eg s3.Bucket(name).upload_fileobj())
   • Use AWS SDK for Python (Boto3) transfer manager which handles multipart and threading
   • Write to S3 from EMR using Hadoop distcp or Spark

4. To make it scalable and generic:

   • Parameterize bucket names, key prefixes
   • Put common logic in separate module/package
   • Use configuration files or environment variables for bucket names
   • Automate deployments of Glue job code

Sources [1] Copying an object using multipart upload - Amazon Simple Storage Service

[2] Efficient Amazon S3 Object Concatenation Using the AWS SDK for Ruby | AWS Developer Tools Blog

[3] Using multipart uploads with directory buckets - Amazon Simple Storage Service

[4] Uploading an object using multipart upload - Amazon Simple Storage Service

[5] Uploading Files to Amazon S3 | AWS Developer Tools Blog

[6] Uploading large objects to Amazon S3 using multipart upload feature and transfer acceleration

[7] Using high-level (s3) commands in the AWS CLI - AWS Command Line Interface