Skip to content

Transfer data from Snowflake to S3

0

We need to transfer data from Snowflake to S3. What are the options to do this using AWS services. We dont want to use Glue as it is costly. Provide other options to pull data from Snowflake and put it in S3 using AWS services.

2 Answers
2

There are a few options to transfer data from Snowflake to Amazon S3 using AWS services without relying on AWS Glue:

  1. AWS Transfer for SFTP: This fully managed SFTP service allows you to move file transfer workloads to AWS. You can set up an SFTP server endpoint, configure authentication for your SFTP clients, and select your target S3 buckets. Users can continue using their existing SFTP clients or applications to transfer data from Snowflake to S3.

  2. AWS DataSync: This service can be used to transfer objects between Amazon S3 compatible storage and various AWS storage services, including Amazon S3. While typically used with AWS Snowball Edge, it can be adapted for transferring data from Snowflake to S3. You would need to configure the necessary network connections and credentials to access both Snowflake and S3.

  3. AWS PrivateLink for Amazon S3: This option provides private connectivity between your VPC and Amazon S3. You can set up AWS PrivateLink to establish a secure connection from your Snowflake environment to S3. This method requires some additional configuration, including setting up VPC endpoints and potentially using Amazon EC2 instances as intermediaries for the data transfer process.

  4. Custom solution using AWS Lambda or Amazon EC2: You could develop a custom solution using AWS Lambda functions or EC2 instances to periodically extract data from Snowflake and upload it to S3. This would involve writing scripts to connect to Snowflake, retrieve the data, and then use the AWS SDK to upload it to S3.

Each of these options has its own trade-offs in terms of complexity, cost, and scalability. The best choice depends on your specific requirements, such as the volume of data, frequency of transfers, and security needs.
Sources
New Service: AWS Transfer for SFTP, a fully managed SFTP service for Amazon S3
Configuring transfers with S3 compatible storage on Snowball Edge - AWS DataSync
Using AWS PrivateLink for Amazon S3 for Private Connectivity Between Snowflake and Amazon S3 | AWS Partner Network (APN) Blog

answered 2 years ago

EXPERT

reviewed 2 years ago

0

For Appflow service, you can refer documentation : [+] Snowflake - : https://docs.aws.amazon.com/appflow/latest/userguide/snowflake.html these describes how to create a flow and use source as Snowflake and destination S3 to send the data using the Appflow service.

Connection details in Appflow: [+] Snowflake - Connection instructions: https://docs.aws.amazon.com/appflow/latest/userguide/snowflake.html#snowflake-setup

[+] Configuring Secure Access to Amazon S3 in the Snowflake documentation https://docs.snowflake.com/en/user-guide/data-load-s3-config

AWS
SUPPORT ENGINEER

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.