Chrome WebDriver and Selenium not working on EC2 Linux script

Question

I am working on developing a python script to automate extracting the shipment status for a batch of UPS tracking IDs.
ETL job extracts an excel file into S3 input bucket, which triggers a lambda function, which then reboots an EC2 instance.
This EC2 instance has my python script inside it.

The code is using selenium and chrome webdriver to web scrape the shipment status from the website. Everything is working fine locally on my computer but I'm facing this error while running the script on EC2 :
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome failed to start: exited normally.
(session not created: DevToolsActivePort file doesn't exist)
(The process started from chrome location /opt/google/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

I've made sure that the google chrome version that I installed matches with the chrome webdriver version that my code installs. The logging is correct in the terminal for it. No other errors in processing the file or anywhere else. Tried running the script without the root user settings just in case permissions is the issue, but that's not solving it either. Tried running the code in both headless and regular mode. Added all the necessary options to it. Do we need any specific permissions to run webdriver on EC2 ? None of this is working out.

Any help is appreciated!

re:Post Agent · Answer

It seems you're encountering a common issue when trying to run Selenium with Chrome WebDriver on an EC2 Linux instance. There are a few things you can try to resolve this:

1. Ensure you have all necessary dependencies installed. On a Linux EC2 instance, you might need to install additional libraries for Chrome to run properly. Try installing these packages:

sudo yum install -y libX11 libXcomposite libXcursor libXdamage libXext libXi libXtst cups-libs libXScrnSaver libXrandr alsa-lib pango atk at-spi2-atk gtk3

2. Use the '--no-sandbox' option when initializing Chrome. This is often necessary when running Chrome as a non-root user on Linux:

chrome_options.add_argument('--no-sandbox')
   chrome_options.add_argument('--disable-dev-shm-usage')

3. Make sure you're using a compatible version of ChromeDriver for your installed Chrome version. You can check the Chrome version with:

google-chrome --version

Then download the matching ChromeDriver version.

4. If you're still having issues, consider using AWS Device Farm for desktop browser testing instead of running Selenium directly on EC2. Device Farm provides a managed Selenium Grid that can be easier to set up and maintain. You would need to modify your RemoteWebDriver initialization to use the Device Farm WebDriver endpoint.

5. Another alternative is to use AWS CloudWatch Synthetics, which supports writing canary scripts using the Selenium test framework. This could potentially simplify your setup as it handles many of the environment configuration details for you.

If none of these solutions work, you might need to investigate further by checking EC2 instance logs or running Chrome with verbose logging enabled to get more detailed error information.

Remember to grant only the necessary permissions to your EC2 instance and follow AWS security best practices when configuring your setup.  
**Sources**  
[Library functions available for Python canary scripts using Selenium - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Synthetics_Canaries_Library_Python.html)  
[Migrating to Device Farm desktop browser testing from Selenium Grid - Device Farm desktop browser testing](https://docs.aws.amazon.com/devicefarm/latest/testgrid/getting-started-migration.html)  
[Runtime versions using Python and Selenium Webdriver - Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Synthetics_Library_python_selenium.html)  
[Migrating to Device Farm desktop browser testing from local Selenium WebDrivers - Device Farm desktop browser testing](https://docs.aws.amazon.com/devicefarm/latest/testgrid/getting-started-local.html)

Chrome WebDriver and Selenium not working on EC2 Linux script

Relevant content