Ingesting data into AWS Data Lake using APIs
I have an AWS Data Lake that is ready to be used at the moment.
My use case for the Data Lake is to be able to, ingest data from different API connectors (coming from other data vendors and service providers). There are at least three different services such as CRM tools, data vendors and etc which I need to ingest data into AWS Data Lake.
All these services provide an API which can be used to connect to the data lake. The challenge is however to ensure that the process of ingesting/transforming and loading (ETL), must be automated and doesn't need manual interventions. Which means once connected, with every new data point that the vendors provide, it must automatically be loaded into the data lake.
My question is, what AWS services do I need to use to accomplish this ?
(1) Building an API connector from different data providers to Data lake ?
(2) Be able to do some ETL (or transform) the data before loading it into the Data Lake. Potentially this ETL service can sit between the API and the Data lake.
Hi, Depending on the services you are looking to extract data form, there could be easier method to extract your data from.
Alternatively if the SDK for the service is available in Python you could use AWS Glue to create the connector and also run the ETL and Finally write to S3. AWS Glue Studio has a marketplace that allows you to source pre-built connectors from AWS and third parties.
hope this helps,
Ingesting data into AWS Data Lake using APIsasked 5 months ago
Is it possible to give AWS SSO users Lake Formation data access?asked 7 months ago
data lake - data ingestion methodsAccepted Answerasked 4 years ago
Is it possible to specify DB snapshot in AWS Lake Formation?Accepted Answerasked a year ago
Access denied error while creating Data Filter in Lake Formationasked 6 months ago
Can I use aws glue crawlers to create master data in delta lake tables?asked a month ago
View data permissions granted with LF-TagsAccepted Answerasked 5 months ago
Using Amazon EBS for Data LakeAccepted Answerasked 5 months ago
Sagemaker and Data on DatabasesAccepted Answerasked 4 years ago
Updating data in governed tablesasked 7 months ago