Best way to run a web scraper from my S3 website and save the returned information to a database

0

I currently have a website hosted on S3 with a working RDS database connection. I would like to run a dynamic web scraper from this site that activates based on user input. Ideally, the user will click a button which will activate a search for a specific product across a specified list of retail sites. This data will be displayed on the site and also saved to the database. It doesn't matter if the data goes directly to the site or to the database. I would like to use an existing web scraping tool such as Scrapy. What is the best way to handle this?

1개 답변
2

One potential way to approach this, would be to set up the web scraper as a Lambda function, and expose the lambda to the website frontend via API gateway. The lambda can then write any output to s3 or to your RDS database.

AWS
bendutt
답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠