How do I resolve the AWS Glue error "The specified subnet does not have enough free addresses to satisfy the request"?

Short description

When you run an AWS Glue ETL job, you might get one of the following errors:

• "The specified subnet does not have enough free addresses to satisfy the request. (Service: AmazonEC2; Status Code: 400; Error Code: InsufficientFreeAddressesInSubnet; Request ID: my_request_id)"
• "An error occurred calling o70.getDynamicFrame. Job 0 cancelled because SparkContext was shut down caused by threshold for executors failed after launch reached"

The preceding errors occur for the following reasons:

• The AWS Glue job uses more data processing units (DPUs) than there are available IP addresses.
• Other AWS services are using IP addresses in the same subnet.
• You didn't detach the elastic network interfaces after the job completes.

Resolution

Confirm how many DPUS the job uses, and reduce the number of DPUs

Make sure that the job is complete. Then, confirm how many DPUs the job used and determine the number of DPUs that you can remove.

To check how many DPUs the job used, complete the following steps:

1. Open the AWS Glue console.
2. In the navigation pane, choose Jobs.
3. Select the job, and then choose the History tab.
4. Check the number of DPUs in the Maximum capacity column.

Then, determine the number of DPUs to remove from the job.

The number of DPUs doesn't equal the number of network interfaces. 

Each worker requires one network interface, with the exception of the following worker types that each require an additional network interface:

• Standard worker type (1 DPU per worker)
• G1.X worker type (1 DPU per worker)
• G2.X worker type (2 DPUs)

For example, if you run a job with 20 workers on a G.2x, then use the following calculation to calculate the number of IP addresses:

AWS Glue 2.0/3.0/4.0: 40 DPU = 19 Workers (executors) + 1 driver = 20 IP addresses (no task runner in AWS Glue 2.0/3.0/4.0)

To reduce the number of DPUs, complete the following steps:

1. Open the AWS Glue console.
2. In the navigation pane, choose Jobs, and then select the job.
3. Choose the Action dropdown list, and then choose Edit job.
4. Expand the Security configuration, script libraries, and job parameters (optional) list.
5. In the Maximum capacity field, enter a lower number for the maximum number of DPUs that the job can use.
6. Save your changes, and then run the job again.

Check the number of available IP addresses, and delete unused network interfaces

First, check the number of available IP addresses in the subnet.

Complete the following steps:

1. Open the AWS Glue console.
2. In the navigation pane, choose Connections.
3. Select the connection that your job uses.
4. Choose the Action dropdown list, and then choose View details.
5. Note the subnet.
6. Open the Amazon VPC console.
7. In the navigation pane, choose Subnets.
8. In the Subnet dropdown list, choose the subnet that the AWS Glue connection uses.
9. On the Description tab, check the Available IPv4 Addresses field to view how many IP addresses are available in the subnet.
10. Verify that the subnet has more available IP addresses than the AWS Glue job requires.

If the subnet doesn't have enough available IP addresses, then delete any unused network interfaces.

Create and use a subnet with more available IP addresses

If the existing subnet doesn't have enough IP addresses, then create a new subnet in your Amazon VPC. You can use the original CIDR block, or add a new one to expand the range. Then, verify that the new subnet uses the same route table and network access control list (network ACL) rules as the original subnet. For example, if your previous subnet had a default route to an internet gateway, then your new subnet must route to an internet gateway.

Update the AWS Glue connection to use the new subnet

Complete the following steps:

1. Open the AWS Glue console.
2. In the navigation pane, choose Connections.
3. Select the connection that your AWS Glue job uses.
4. In the Action dropdown list, choose Edit connection.
5. On the Setup your connection's properties page, choose Next.
6. On the Setup access to your data store page, in the Subnet dropdown list, select the new subnet.
7. Choose Next, and then choose Finish.
8. Run the job again.

Related information

Defining job properties for Spark jobs

Setting up Amazon VPC for JDBC connections to Amazon RDS data stores from AWS Glue