Is there an AWS service that can codelessly, serverlessly download an external file via HTTPS to an S3 bucket?
Is there an AWS service that can codelessly, serverlessly download an external file via HTTPS to an S3 bucket?
Is there a way I can serverlessly, codelessly download a file from a given external HTTPS URL and save the downloaded file to a given S3 bucket?
Is there a way I can do this with Step Functions?
Is my only option to drop the codeless requirement and use Lambda?
Step Functions supports the ability to call HTTP endpoints through API Gateway, but does not currently support the ability to call generic HTTP endpoints. Here is a link to the documentation that talks about this. https://docs.aws.amazon.com/step-functions/latest/dg/connect-api-gateway.html
Here is a youtube video on how to upload files to S3 using AWS Lambda. If you are now leaning toward this approach. https://www.youtube.com/watch?v=mw_-0iCVpUc
Other options is to use Amazon Firehose. This is great for analytics use-cases and can handle larger volumes of data. It can also incorporate compression and Parquet conversions to make storing big data more cost-effective. https://aws.amazon.com/kinesis/data-firehose/features
Relevant questions
Public access downloading from bucket
Accepted Answerasked a month agoIs there a serverless, codeless way I can send an email with an attachment?
asked 12 days agoS3 pre signed url with block public access on a bucket
Accepted Answerasked 2 years agoS3 file drop fire multiple Lambdas
Accepted Answerasked 3 years agoAppStream AppBlock Does Not Download From S3 On Instance Launch
asked a month agoBatch download files from multiple different folders in the same S3 bucket
asked 2 months agoIs there an AWS service that can codelessly, serverlessly download an external file via HTTPS to an S3 bucket?
asked 12 days agoHow to establish connection between AWS Fargate task to an external SFTP server?
asked 2 days agoHow can I (use AWS GoLang SDK to) create an io.writer (pipe) that will write buffers to a AWS S3 bucket file?
asked a month agoWhere is my Amazon Workspace Web temporary storage?
asked a month ago
I have been combing through the Amazon Data Firehouse documentation and don't see where how to use Amazon Data Firehouse to do an HTTP GET to a generic HTTP endpoint. Where is that documented? Thank you.