By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Is there an AWS service that can codelessly, serverlessly download an external file via HTTPS to an S3 bucket?

0

Is there an AWS service that can codelessly, serverlessly download an external file via HTTPS to an S3 bucket?

Is there a way I can serverlessly, codelessly download a file from a given external HTTPS URL and save the downloaded file to a given S3 bucket?

Is there a way I can do this with Step Functions?

Is my only option to drop the codeless requirement and use Lambda?

Topics
ServerlessStorageApplication Integration
Tags
ServerlessAmazon Simple Storage ServiceAWS Step Functions
Language
English
rossmpersonal
asked 2 years ago507 views
1 Answer
0

Step Functions supports the ability to call HTTP endpoints through API Gateway, but does not currently support the ability to call generic HTTP endpoints. Here is a link to the documentation that talks about this. https://docs.aws.amazon.com/step-functions/latest/dg/connect-api-gateway.html

Here is a youtube video on how to upload files to S3 using AWS Lambda. If you are now leaning toward this approach. https://www.youtube.com/watch?v=mw_-0iCVpUc

Other options is to use Amazon Firehose. This is great for analytics use-cases and can handle larger volumes of data. It can also incorporate compression and Parquet conversions to make storing big data more cost-effective. https://aws.amazon.com/kinesis/data-firehose/features

AWS
pvillena
answered 2 years ago
  • rossmpersonal
    2 years ago

    I have been combing through the Amazon Data Firehouse documentation and don't see where how to use Amazon Data Firehouse to do an HTTP GET to a generic HTTP endpoint. Where is that documented? Thank you.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content