AWS Lambda running Python Selenium Chrome driver fails with "The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed"

Hi All. I am trying to run a Docker image under AWS Lambda which contains a simple Python 3.11 script that uses Selenium Chrome driver to navigate to "https://demo.seleniumeasy.com/basic-first-form-demo.html", enter a couple of numbers and then click on "Get Total" Button. The Docker image runs fine locally, but it's failing when deployed to Lambda. Currently the Chrome options are:

   options = Options()

    # Trying more options
    options.add_argument("--no-sandbox")
    options.add_argument("--headless=new")
    options.add_argument('--ignore-certificate-errors')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--disable-gpu')
    options.add_argument('--disable-extensions')
    options.add_argument('--enable-automation')

    driver = webdriver.Chrome(options=options)

• The Lamdba error message (including stack trace) is:

{
  "errorMessage": "Message: unknown error: Chrome failed to start: crashed.\n  (chrome not reachable)\n  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\nStacktrace:\n#0 0x561223264d93 <unknown>\n#1 0x5612230332d7 <unknown>\n#2 0x56122305bab0 <unknown>\n#3 0x561223057a3d <unknown>\n#4 0x56122309c4f4 <unknown>\n#5 0x561223093353 <unknown>\n#6 0x561223062e40 <unknown>\n#7 0x561223064038 <unknown>\n#8 0x5612232b88be <unknown>\n#9 0x5612232bc8f0 <unknown>\n#10 0x56122329cf90 <unknown>\n#11 0x5612232bdb7d <unknown>\n#12 0x56122328e578 <unknown>\n#13 0x5612232e2348 <unknown>\n#14 0x5612232e24d6 <unknown>\n#15 0x5612232fc341 <unknown>\n#16 0x7f1dc4ea344b start_thread\n",
  "errorType": "WebDriverException",
  "requestId": "52e6b19a-7d62-40d0-9a49-a276e469cfe6",
  "stackTrace": [
    "  File \"/var/task/app1.py\", line 72, in lambda_handler\n    run_chrome()\n",
    "  File \"/var/task/app1.py\", line 46, in run_chrome\n    driver = webdriver.Chrome(options=options)\n",
    "  File \"/var/task/selenium/webdriver/chrome/webdriver.py\", line 45, in __init__\n    super().__init__(\n",
    "  File \"/var/task/selenium/webdriver/chromium/webdriver.py\", line 61, in __init__\n    super().__init__(command_executor=executor, options=options)\n",
    "  File \"/var/task/selenium/webdriver/remote/webdriver.py\", line 208, in __init__\n    self.start_session(capabilities)\n",
    "  File \"/var/task/selenium/webdriver/remote/webdriver.py\", line 292, in start_session\n    response = self.execute(Command.NEW_SESSION, caps)[\"value\"]\n",
    "  File \"/var/task/selenium/webdriver/remote/webdriver.py\", line 347, in execute\n    self.error_handler.check_response(response)\n",
    "  File \"/var/task/selenium/webdriver/remote/errorhandler.py\", line 229, in check_response\n    raise exception_class(message, screen, stacktrace)\n"
  ]
}

The Cloudwatch log shows:

START RequestId: 52e6b19a-7d62-40d0-9a49-a276e469cfe6 Version: $LATEST
TMP: /tmp
TMPDIR: /tmp
SE_CACHE_PATH: /tmp/chrome
os.name = 'posix'
/usr/bin/google-chrome: line 45: /dev/fd/62: No such file or directory
/usr/bin/google-chrome: line 46: /dev/fd/62: No such file or directory
Google Chrome 110.0.5481.177 
ChromeDriver 110.0.5481.77 (65ed616c6e8ee3fe0ad64fe83796c020644d42af-refs/branch-heads/5481@{#839})
[ERROR] WebDriverException: Message: unknown error: Chrome failed to start: crashed.
(chrome not reachable)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

NOTE that both "os.system('google-chrome --version')" and "os.system('chromedriver --version')" commands ran successfully

• Additionally I have set the following environment variables for the Lambda function:

  ENV VAR	VALUE
  DISPLAY	:0
  HOME	/tmp/fakehome
  SE_CACHE_PATH	/tmp/chrome
  TMP	/tmp
  TMPDIR	/tmp

• I've been working on this the last few days and have tried a multitude of combinations for the Chrome options. Any suggestions appreciated. Let me know if you need more info...

PS: I should add that I have a similar setup with a more complicated browsing session working fine on EC2 i.e. Python3.11, Selenium, Chrome tech stack (without Docker) runs fine on EC2 running Amazon Linux. But I am now trying to scale it up by using Lambda.