Usando AWS re:Post, accetti AWS re:Post Termini di utilizzo
AWS re:Postre:Post

Best way to run a web scraper from my S3 website and save the returned information to a database

0

I currently have a website hosted on S3 with a working RDS database connection. I would like to run a dynamic web scraper from this site that activates based on user input. Ideally, the user will click a button which will activate a search for a specific product across a specified list of retail sites. This data will be displayed on the site and also saved to the database. It doesn't matter if the data goes directly to the site or to the database. I would like to use an existing web scraping tool such as Scrapy. What is the best way to handle this?

Argomenti
ComputazionaliServerlessFront-end web e dispositivi mobiliNetworking e distribuzione di contenutiDatabase
Tag
Provisioning di siti webAWS LambdaGateway API di AmazonHosting di siti webDatabase
Lingua
English
AWS-User-0616660
posta 2 anni fa330 visualizzazioni
1 Risposta
2

One potential way to approach this, would be to set up the web scraper as a Lambda function, and expose the lambda to the website frontend via API gateway. The lambda can then write any output to s3 or to your RDS database.

AWS
bendutt
con risposta 2 anni fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande

Contenuto pertinente