Glue connections running out of IPs

Question

We recently experienced an issue where our glue connection ran out of IP addresses inside the subnet that it was created in.

Error message:
```
AWS Glue Error: The specified subnet does not have enough free addresses to satisfy the request
```

This was due to too many workers/dpus being assigned to the jobs in a single subnet.

I have two questions:
1. Since glue appears to take a **list** of connections, will it "roll-over" when one subnet is maxxed out and use the next connection in the list?  (at least this is what it shows in Terraform module--a list argument)

2. How many IP addresses does a G.2x worker use?  Since its 2 dpu, does it use 2 IPs?  Or is it variable?

Gonzalo Herreros · Accepted Answer

1. No, all the cluster will be in a single subnet from a single connection.   
2. One per worker, doesn't matter if the worker is 1DPU or 8

tjtoll · Answer

I only know the answer to #2: each DPU (or node) consumes an IP. We have a subnet dedicated to Glue has as many IPs as our Glue DPU quota.

rePost-User-senthilrathinam · Answer

Please refer to the solution outlined in this AWS Glue blog that employs a Private NAT Gateway approach to address this issue.

https://aws.amazon.com/blogs/big-data/scale-aws-glue-jobs-by-optimizing-ip-address-consumption-and-expanding-network-capacity-using-a-private-nat-gateway/

Glue connections running out of IPs

Add your answer

Relevant content