Mounting EFS in codebuild fails with "Error retrieving region"

Question

Hello,  
  
I have a buildspec that mounts an EFS drive. I get the following error:

```
[Container] 2019/12/17 05:03:23 Running command mount -t efs fs-11112222.efs.ap-south-1.amazonaws.com:/ /efs 
Error retrieving region 
```

I have changed the efs name in above code for this post.   
  
I have followed documentation here: https://docs.aws.amazon.com/codebuild/latest/userguide/sample-efs.html#sample-efs-create-acb   
  
Note that the documentation is asking to use amazon linux image in codebuild, but, uses apt-get in buildspec. That fails. I changed it to the amazonlinux equivalent yum commands.   
  
I have ensured that the codebuild has "PrivilegedMode" True.  
  
I am able to mount the EFS drive from an EC2 instance that is in the same subnet as the codebuild config.   
  
If I remove the EFS mounting instructions in buildspec, my build runs fine... Meaning, network and internet access looks fine.  
  
Please help!  
  
Best,  
Sachin  
  
Edited by: sdole on Dec 17, 2019 5:51 AM

Answer

I found this code in efs_utils. It seems like this code is looking for instance metadata service on codebuild. Maybe it cannot find that in docker?  
  
https://github.com/aws/efs-utils/blob/master/src/mount_efs/__init__.py#L130

Answer

Of the two changes you made, it's likely #2 is the only one that made a difference. When you mount an EFS file system via DNS, Route53 will return the IP address for the mount target in the same AZ as the client. If there is no mount target in the local AZ the mount will fail in order to avoid cross-az network charges. However, there is no problem with mount targets in extra AZs, they will simply be ignored by your client.  
  
The EFS mount helper forms the DNS name using the file system id and local region, with the latter looked up in EC2 instance metadata. Since your codebuild instance is running inside a container, and the metadata URL is different than with traditional EC2, this call is failing. As a workaround, you can edit the EFS mount helper configuration file (/etc/amazon/efs/efs-utils.conf) and hardcode region in the line that defines DNS name format:  
dns_name_format = {fs_id}.efs.{region}.amazonaws.com  
  
One reason you may decide to try this rather than using nfs-utils is if you want to take advantage of TLS encryption, IAM authorization, or EFS Access Points.

Answer

I found 2 problems and changed code to work around those problems.  
1. in ap-south-1, codebuild does not support ap-south-1c AZ, hence one of my mount points was probably not good for Codebuild.   
2. efs-utils codebase makes an instance metadata lookup for finding current region. Not sure whether or not Codebuild supports instance metadata lookup.  
  
To work around: I deleted the EFS mountpoint in ap-south-1c that codebuild does not support and switched to nfs-utils instead of efs-utils.  
  
Now, my setup works.  
  
I am disappointed about the snowflakish variation that ap-south-1c is not supported. How will I account for that in my automation? I will need to code in such one off exclusions. No?  
  
I opened a bug on efs-utils for them to fix the instance metadata lookup.   
https://github.com/aws/efs-utils/issues/37

Mounting EFS in codebuild fails with "Error retrieving region"

Relevant content