Hi mr cloud.
I want to understand the best way to query RDS using a lambda function to fetch data and write to another AWS account S3 bucket.
For Fetching data I d recommend RDS Proxy so it handles automatically connection pool and more: https://github.com/aws-samples/amazon-rds-proxy-video-demo
Do we need to create view?
If you mean database view, no, no need.
how long lambda can be executed?
Max lambda execution is 15 minutes. Id though strongly recommend to keep it much shorter for that. If the workload is a long running one, lambda May not be best fit.
What should be batching strategy?
If you mean in regards database, Rds proxy will alleviate that otherwise sdk actions are available.
Can it write in parquet format?
Yes. Use python or node JS libraries/packages to perform that.
Hope it helps, and if does, I d appreciate to get accepted as answer so that community can benefit for clarity when asking similar questions, cheers!
- asked 4 years ago
- Accepted Answerasked 2 years ago
- How do I set up Amazon S3 event notifications to invoke a Lambda function that's in another AWS account?AWS OFFICIALUpdated a month ago
- AWS OFFICIALUpdated 3 months ago
- AWS OFFICIALUpdated 25 days ago
- EXPERTpublished 3 months ago
- A Brief Primer to Onboarding Data To a Healthcare and Life Sciences Data Mesh Leveraging AWS ServicesEXPERTpublished 7 months ago