I get the follow error with my lambda function: Calling the invoke API action failed with this message: Network Failure

0

I have set up a VPC with outbound traffic. Any help would be massively appreciated

See below for the code:

import os
import time
from datetime import datetime
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

import boto3


def lambda_handler(event, context):
    options = Options()
    options.binary_location = '/opt/headless-chromium'
    options.add_argument('--headless')

    
     # Add your AWS S3 bucket name
    S3_BUCKET_NAME = "webscrapebucketxxxxxx22"

    # Initialize the S3 client
    s3 = boto3.client("s3")

    # Set the dictionary of website URLs you want to screenshot
    urls = {  "Google": "http://www.google.com"

    }
    
    # Set the path to your browser driver
    driver_path = "/opt/chromedriver"
    
 # Create a new instance of the browser driver with headless mode
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    driver = webdriver.Chrome(executable_path=driver_path, chrome_options=options)
    
    # Define a function to take a screenshot and save it to a file
    def take_screenshot(url, name):
        # Navigate to the website
        driver.get(url)
        
        # Wait for 30 seconds to ensure that the page is fully loaded
        time.sleep(30)
        
        # Get the current date as a string in the format YYYY-MM-DD
        now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
        
        # Generate the filename using the website name and the current date
        filename = f"{name}_{now}.png"
        
        # Save the screenshot to a file in the /tmp directory
        filepath = os.path.join("/tmp", filename)
        driver.save_screenshot(filepath)
        
        # Return the filepath for logging purposes
        return filepath
        
    def upload_to_s3(filepath):
        key = os.path.basename(filepath)
        s3.upload_file(filepath, S3_BUCKET_NAME, key)
        print(f"Uploaded {filepath} to s3://{S3_BUCKET_NAME}/{key}")
        
        
    # Loop through the dictionary of website URLs and take screenshots
    for name, url in urls.items():
        filepath = take_screenshot(url, name)
        upload_to_s3(filepath)
        print(f"Screenshot of {name} saved to {filepath}")
asked a year ago2986 views
4 Answers
0

Thanks for the quick replies! I originally didnt use a VPC but I was getting the same error which is why I tried the VPC approach.

If the VPC isn't required I'll remove it.

See below for the cloudwatch log snip, could this be an issue with the version of selenium or chrome webdriver I have used?

Cloudwatch log snip

answered a year ago
  • The error message "Chrome failed to start ~" appears in the log of the image.
    This error can be fixed by adding the following options.

    options.add_argument('--no-sandbox')
    
0

First, I do not see any reason to attach your function to a VPC. It does not seem you are actually accessing any private resources in your VPC.

If you do need the VPC for some reason, your function must be attached to a private subnet with a route to a NAT Gateway or to VPC endpoints to different AWS services you need to access (e.g., S3 in your case).

profile pictureAWS
EXPERT
Uri
answered a year ago
0

If Lambda is configured in a VPC, check the route table of the subnet where Lambda is configured to see if there is a route to the NAT Gateway.
Also verify that the security group's outbound rules allow it and that the network ACLs allow the necessary communications.
https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-networking.html

profile picture
EXPERT
answered a year ago
0

You may want to check if the Lambda function is configured with the correct security group and network ACL settings that allow outbound traffic.

Another thing to check is if your VPC has a NAT gateway or NAT instance set up for internet connectivity. Without a NAT, your Lambda function would not be able to communicate with resources outside of your VPC.

You can also check the AWS CloudWatch logs for your Lambda function to see if there are any additional error messages or clues as to why the network call failed.

hash
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content