Skip to content

Sharing EFS file between services using AWS Copilot

0

Hello,

I'm facing an issue with EFS volume sharing between two services in my AWS Copilot application and would appreciate your guidance on the correct manifest configuration.

Goal: My goal is to have a Load Balanced Web Service (wp) create and populate an EFS volume, and have a separate Backend Service (ftp) mount that exact same volume to provide FTP access to the shared files.

Problem: Currently, the wp service works as expected. The ftp service deploys and mounts a volume at the specified path, but the directory always appears empty. This leads me to believe it's mounting a separate EFS Access Point instead of reusing the one from wp, even though I am using the from key in the manifest.

Could you please confirm the definitive storage configuration required in both service manifests to achieve this shared EFS volume correctly?


Service 1: wp (Source Service) Manifest

name: wp
type: Load Balanced Web Service

storage:
  volumes:
    sharedData:
      efs: true
      path: /shared/data
      read_only: false
#used a sidecar to manage groups and permissions
sidecars:
  multi-service-permissions:
    image: alpine:latest
    essential: false
    command: ["/bin/sh", "-c"]
    args:
      - |
        addgroup -g 2000 shared-web
        
        chown -R root:2000 /shared/data
        chmod -R 664 /shared/data
        find /shared/data -type d -exec chmod 775 {} \;
        
        find /shared/data -type d -exec chmod g+s {} \;
        
        echo "Multi-service permissions set"
    mount_points:
      - source_data: sharedData
        target: /shared/data
        container_path: /app/public/shared-data
        path: /app/public/shared-data

Service 2: ftp (Consumer Service) Manifest

name: ftp
type: Backend Service

storage:
  volumes:
    sharedData:
      from: 'wp.sharedData' # Mount from the wp service
      path: /shared/data

My Copilot application is named worpress-v2 and the environment is test.

Thank you for your help.


UPDATE: I've got the EFS sharing working using environment addons

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name.

Resources:
  # 1. A security group for the EFS mount targets to control access.
  EFSSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: "Security group for EFS mount targets"
      VpcId:
        Fn::ImportValue: 
          Fn::Sub: '${App}-${Env}-VpcId'
      Tags:
        - Key: Name
          Value: !Sub '${App}-${Env}-EFSSecurityGroup'

  # 2. The EFS file system itself.
  EFSFileSystem:
    Type: AWS::EFS::FileSystem
    Properties:
      Encrypted: true
      FileSystemTags:
        - Key: Name
          Value: !Sub '${App}-${Env}-SharedFileSystem'

  # 3. An ingress rule to allow traffic from your services to the EFS.
  # This rule connects your environment's main security group to the EFS security group.
  EFSSecurityGroupIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: "Allow inbound NFS traffic from Copilot services"
      GroupId: !Ref EFSSecurityGroup
      IpProtocol: tcp
      FromPort: 2049
      ToPort: 2049
      SourceSecurityGroupId: 
        Fn::ImportValue: 
          Fn::Sub: '${App}-${Env}-EnvironmentSecurityGroup'

  # 4. Mount targets for each PUBLIC subnet in the environment.
  # This ensures tasks running in public subnets can connect to EFS.
  MountTarget0:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref EFSFileSystem
      SubnetId: 
        Fn::Select:
          - 0
          - Fn::Split:
              - ','
              - Fn::ImportValue:
                  Fn::Sub: '${App}-${Env}-PublicSubnets'
      SecurityGroups:
        - !Ref EFSSecurityGroup

  MountTarget1:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref EFSFileSystem
      SubnetId: 
        Fn::Select:
          - 1
          - Fn::Split:
              - ','
              - Fn::ImportValue:
                  Fn::Sub: '${App}-${Env}-PublicSubnets'
      SecurityGroups:
        - !Ref EFSSecurityGroup
  
Outputs:
  # 5. Export the EFS ID so your services can reference it.
  SharedEFSID:
    Description: "The ID of the shared EFS file system."
    Value: !Ref EFSFileSystem
    Export:
      Name: !Sub '${App}-${Env}-SharedEFSID'

and I use placeholders in the efs for both servcies as `` volumes: orderwise: path: /shared/data efs: id: fs-placeholder read_only: false

then I use overrides by running copilot svc override --name svc_name for the two services and in the overrides/cfn.patches.yml I have

path: /Resources/TaskDefinition/Properties/Volumes/0/EFSVolumeConfiguration/FilesystemId value: Fn::ImportValue: !Sub '${AppName}-${EnvName}-SharedEFSID'

this allows them share the same efs and read from same location.

THE LAST PUZZLE: I cant write using the www-data user that the WordPress uses (only the root user `gid=0 uid=0` can write) . I've tried using sidecars but the permissions fail to apply for some reason, even when I log in as root using copilot svc exec and manually change the ownership to the shared-web user or www-data, it still has permission issues. How do i allow read/write permissions using the docker/manifest or entry files?

I have tried many variations of this for the sidecars

sidecars image: alpine:latest essential: false command: ["/bin/sh", "-c"] args: - | addgroup -g 2000 shared-web

    chown -R www-data:2000 /shared/data
    chmod -R 664 /shared/data
    find /shared/data -type d -exec chmod 775 {} \;
    
    find /shared/data -type d -exec chmod g+s {} \;
    
    echo "Multi-service permissions set"
mount_points:
  - source_volume: sharedData
    path: /shared/data
1 Answer
0

To share an EFS volume between two AWS Copilot services, you need to ensure that both services are referencing the same EFS file system and access point. Based on your manifests, there are a few adjustments needed to correctly share the EFS volume.

For your wp service (source service), your configuration looks mostly correct. You're creating an EFS volume named "sharedData" mounted at /shared/data. The sidecar for permission management is a good approach to ensure proper access rights.

For your ftp service (consumer service), the issue is likely in how you're referencing the shared volume. When using the from key, you should only specify the source service and volume name without additional EFS configuration. Including efs: true in the consumer service may be causing Copilot to create a new access point instead of using the existing one.

Here's the recommended configuration for your ftp service:

name: ftp
type: Backend Service

storage:
volumes:
sharedData:
from: 'wp.sharedData'
path: /shared/data

Remove the efs: true line since that's already defined in the source service. The from parameter tells Copilot to use the same EFS file system and access point that was created for the wp service.

Also, ensure that:

  1. Both services are deployed in the same environment
  2. The security groups for both services allow the necessary traffic for EFS (port 2049)
  3. The IAM roles for both services have the necessary permissions to access the EFS file system

If you're still experiencing issues after these changes, you might need to check the EFS access point configurations in the AWS console to verify that both services are indeed using the same access point. The permissions set by your sidecar container should be sufficient, but verify that the user/group IDs used by the ftp service have appropriate access to the shared files.
Sources
Amazon EFS volumes - AWS Batch
Amazon EFS volumes - Amazon Elastic Container Service

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.