How can I resolve the AWS Glue error "The specified subnet does not have enough free addresses to satisfy the request"?

Short description

You get these errors when there aren't enough IP addresses available for the AWS Glue job. Here are two common reasons why these errors might happen:

• When you run a job in a virtual private cloud (VPC) subnet, AWS Glue sets up elastic network interfaces that allow your job to connect securely to other resources in the VPC. Each elastic network interface gets a private IP address. If the elastic network interfaces aren't released when expected, then there might not be enough IP addresses available for the job. To resolve the error, confirm how many DPUs that the job used. Then, reduce the number of DPUs and run the job again. Or, delete unused elastic network interfaces.
• Multiple AWS services are using the same subnet. These services might be using many of the subnet's available IP addresses. To resolve the error, use a different subnet with more available IP addresses for the AWS Glue job.

Resolution

Use one of the following methods to resolve these errors.

Reduce the number of data processing units (DPUs) for the job

After the job run completes, check how many DPUs were used by the job:

1. Open the AWS Glue console.
2. On the navigation pane, choose Jobs.
3. Choose the job, and then choose the History tab. The Maximum capacity column shows the number of DPUs used for the job.

Do the following to reduce the number of DPUs for the job:

1. Decide how many DPUs to remove from the job. Keep in mind that the number of DPUs doesn't equal the number of elastic network interfaces. One elastic network interface is always attached to each worker. However, additional elastic network interfaces are also required for each job:
   Standard worker type (1 DPU per worker): one additional elastic network interface required
   G1.X worker type (1 DPU per worker): one additional elastic network interface required
   G2.X worker type (2 DPUs): one additional elastic network interface required For example:
   If you're running a job with 20 workers on a G.2x, you can calculate the number of IP addresses as follows:
   With AWS Glue 0.9/1.0: 41 DPU = 20 workers (executors) + 1 driver + 1 task runner = 22 IP addresses
   With AWS Glue 2.0/3.0: 40 DPU = 19 Workers (executors) + 1 driver = 20 IP addresses (no task runner in AWS Glue 2.0/3.0)
2. On the navigation pane, choose Jobs.
3. Choose the Action dropdown list, and then choose Edit job.
4. Expand the Security configuration, script libraries, and job parameters (optional) list.
5. In the Maximum capacity field, enter a lower number. This field sets the maximum number of DPUs that the job can use.
6. Save your changes, and then run the job again.

Delete unused elastic network interfaces

Check the number of available IP addresses in the subnet by doing the following:

1. Open the AWS Glue console.
2. In the navigation pane, choose Connections.
3. Select the connection that you want to check.
4. In the Action dropdown list, choose View details. Note the subnet.
5. Open the Amazon VPC console.
6. In the navigation pane, choose Subnets.
7. In the Subnet dropdown list, choose the subnet that the AWS Glue connection is using.
8. On the Description tab, check the Available IPv4 Addresses field. This field shows how many IP addresses are available in the subnet.

Be sure that the number of available IP addresses is more than the number of IP addresses required for the AWS Glue job.

If the number of available IP addresses is less than that required for your AWS Glue job to run, then delete the unused elastic network interfaces.

Create and use a subnet with more available IP addresses

Do the following to create a new subnet:

1. Create a new subnet in your VPC.
   Note: You can create a new subnet using the VPC's original CIDR blocks, or add additional CIDR blocks to your VPC to use with the new subnet.
2. Review the route tables and access control list (ACL) rules associated with the old subnet to be sure that the new subnet routes traffic the same way. For example, if your previous subnet had a default route configured to an internet gateway, then be sure that your new subnet has a similar default route.

Do the following to modify the AWS Glue connection to use the new subnet:

1. Open the AWS Glue console.
2. In the navigation pane, choose Connections.
3. Select the connection that your AWS Glue job is using.
4. In the Action dropdown list, choose Edit connection.
5. On the Set up your connection’s properties page, choose Next.
6. On the Set up access to your data store page, in the Subnet dropdown list, choose the new subnet.
7. Choose Next, and then choose Finish.
8. Run the job again.

Related information

Defining job properties for Spark jobs

Setting up a VPC to connect to JDBC data stores