By using AWS re:Post, you agree to the Terms of Use

Best way to run a web scraper from my S3 website and save the returned information to a database


I currently have a website hosted on S3 with a working RDS database connection. I would like to run a dynamic web scraper from this site that activates based on user input. Ideally, the user will click a button which will activate a search for a specific product across a specified list of retail sites. This data will be displayed on the site and also saved to the database. It doesn't matter if the data goes directly to the site or to the database. I would like to use an existing web scraping tool such as Scrapy. What is the best way to handle this?

1 Answer

One potential way to approach this, would be to set up the web scraper as a Lambda function, and expose the lambda to the website frontend via API gateway. The lambda can then write any output to s3 or to your RDS database.

answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions