AWS Transfer Python Coding Question

Hi there,

Consider using S3 APIs directly instead of AWS Transfer

The AWS Transfer Family of services is typically a better fit when the object sender is unable to use S3 REST APIs to write directly to a bucket and instead must use protocols like SFTP (common in B2B scenarios).

While you can certainly use AWS Transfer services, it doesn't sound like you have a hard requirement to do so. Instead, it seems you're just looking for good ways to learn Python programming and cloud development. If that's the case, I would suggest you instead use the AWS SDK for Python (boto3) to read from /write to S3 directly. I think you'll find this much easier to code than putting the AWS Transfer service in the middle of things.

The other big benefit of using S3 APIs directly is that writing files directly to S3 is going to be less expensive. The two biggest reasons are:

1. AWS Transfer creates and manages a transfer endpoint for you. You pay an hourly cost for that endpoint even if it is not in use. Writing to S3 requires no such endpoint, thus no hourly cost.

2. AWS Transfer charges $0.04 per gigabyte transferred in either direction. If you use S3 APIs, there are no data transfer costs when writing to S3 (there is a small $0.005 per PutObject API call, though), and reading from S3 typically starts at ~$0.02 per gigabyte.

Refer to AWS's pricing pages for latest pricing:

• AWS S3 Pricing
• AWS Transfer Pricing

AWS Transfer workflow limits and alternatives

Many (but not all) of AWS service limits are adjustable. These limits are sometimes referred to as "service quotas", and these docs show that the limit of 10 workflows per account can be raised. I don't see any mention of steps per workflow, so that may be a hard limit. See here for how to request a limit increase.

I don't know what you're trying to accomplish with the AWS Transfer Workflows, but if you decide to send data to S3 directly, there are still ways to build your own workflows... a really common pattern is to create an S3 Event Notification. You can configure an S3 event to trigger when one or more of certain events, like object put or object delete, occur. You can choose the send the event to one of several AWS services (like AWS Lambda, SNS, SQS, or EventBridge) that you can use to build your own custom automation/workflow.

Python examples

There are lots of examples of Python+S3 out there, with a few examples below:

• boto3 - this is the name of the AWS SDK that you will import to make AWS API calls, including S3

• [AWS Python Code Examples for S3] - there are several examples, and it looks like object_wrapper.py is a good starting point for you.

• Tutorial: Using an Amazon S3 trigger to invoke a Lambda function - one of many examples out there of ways you can use S3 Events to build automatic responses / workflows to object changes in S3.

• There are a number of AWS and independent blogs and github projects out there, too. For example, Pushp Vashisht's blog [Use Python & Boto3 to Backup files/logs to AWS S3](https://medium.com/@pushp1997/use-python-boto3-to-backup-files-logs-to-aws-s3-df34cf9

Pricing Notes

Any reference to pricing is based on a quick review of the us-west-2 rates from AWS public documentation. AWS prices can vary by region and some services also offer automatic tiering (discounts) for high-volume usage.