By using AWS re:Post, you agree to the Terms of Use
/Is there an AWS service that can codelessly, serverlessly download an external file via HTTPS to an S3 bucket?/

Is there an AWS service that can codelessly, serverlessly download an external file via HTTPS to an S3 bucket?

0

Is there an AWS service that can codelessly, serverlessly download an external file via HTTPS to an S3 bucket?

Is there a way I can serverlessly, codelessly download a file from a given external HTTPS URL and save the downloaded file to a given S3 bucket?

Is there a way I can do this with Step Functions?

Is my only option to drop the codeless requirement and use Lambda?

1 Answers
0

Step Functions supports the ability to call HTTP endpoints through API Gateway, but does not currently support the ability to call generic HTTP endpoints. Here is a link to the documentation that talks about this. https://docs.aws.amazon.com/step-functions/latest/dg/connect-api-gateway.html

Here is a youtube video on how to upload files to S3 using AWS Lambda. If you are now leaning toward this approach. https://www.youtube.com/watch?v=mw_-0iCVpUc

Other options is to use Amazon Firehose. This is great for analytics use-cases and can handle larger volumes of data. It can also incorporate compression and Parquet conversions to make storing big data more cost-effective. https://aws.amazon.com/kinesis/data-firehose/features

answered 11 days ago
  • I have been combing through the Amazon Data Firehouse documentation and don't see where how to use Amazon Data Firehouse to do an HTTP GET to a generic HTTP endpoint. Where is that documented? Thank you.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions