Selenium and Web-Drivers in lambda function

0

Hello everyone, Our team is working on a web scraping project, that we'd like to upload to aws lambda. But, unfortunately, we've encountered some problems with web drivers. Based on our research, we are using: Python 3.8 Selenium v4.15.1 Chrome driver v121.0.6167.139 Headless-Chromium v1.0.0-57 (web drivers are packed together, and added to a layer through S3) as described in: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Synthetics_Library_python_selenium.html We've modeled our code according to those few tutorials we were able to find (such as https://www.youtube.com/watch?v=b49Y3NGJX68). But while trying to run even the simplest code, we encounter one of two errors: "Service /opt/chromedriver unexpectedly exited. Status code was: 127", or "Unable to locate or obtain driver for chrome".

We are using the following code structure: def lambda_handler(event, context):

logging.info('Web driver started')

# Define path
service = Service(executable_path=r'/opt/chromedriver')

# Define options
chrome_options = Options()

chrome_options.binary_location = '/opt/headless-chromium'

chrome_options.add_argument("--headless")

# Initialize Chrome
driver = webdriver.Chrome(service=service, options=chrome_options)
logging.info('Web driver defined')

# Open Google and then quit the driver
driver.get('https://www.google.ru')
logging.info('Web driver opened')

driver.quit()
logging.info('Web driver closed')

return "finished"

We'd be truly thankful if anyone could help us understand the reasons behind the mentioned errors.

  • Are the chromedriver and the headless-chromium in your Lambda layer built for the correct CPU architecture of your Lambda function? (i.e arm64 vs x86_64)

profile picture
Giorgi
asked 13 days ago98 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions