By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

DataSync S3 to EFS "Success" but no files in EFS

0

Hello. I am new to AWS, so excuse any accidental simplifications/omissions.

Background: I have a windows desktop (not windows server). I have some CSV files which I want to run through a python container in Batch. Ultimately, I want to upload CSVs into EFS. The EFS will be mounted in the container/job and output a CSV from the container to EFS. I am okay with the Batch (for now).

Issue:

  • I have an S3 Bucket and an EFS created. The S3 contains the CSV file under /my_input_files/ folder. As Batch uses EFS, I would like to send the S3 data to EFS, then run the Batch.
  • I wanted to use DataSync to move S3 files to EFS. I used the DataSync wizard to set up a Task, using the S3 to EFS as sources/destinations, filling in the fields (using /my_input_files/ as the S3 source).
  • When I set the EFS destination something other than blank, I get an error of "cannot mount"
  • When I run the task, I get "Success". I created all the logs and save in S3, and I view them, but I cannot understand where the files are saved in the EFS
  • When I launch and EC2 instance and view the EFS, I cannot find the files anywhere.

I am not using an Agent or EC2 but just "Start" the DataSync task. I've tried looking at similar questions online, but the process described is a lot more convoluted, including using the Agent and setting up VPNs subnets etc, which did not seem relevant to my case (I am only passing data in same region between services on the same account).

Any help would be much appreciated.

Topics
StorageMigration & ModernizationComputeContainers
Tags
Amazon Simple Storage ServiceAWS DataSyncAWS BatchAmazon Elastic Container Service
Language
English
chrism
asked 2 months ago301 views
3 Answers
0
Accepted Answer

There does not seem to be a solution to this, as AWS support comes up with the same suggestions, and nothing works. As such, I am closing this down as no longer required.

chrism
answered 2 months ago
0

Hi,

This repo aws-samples/aws-s3-to-efs-examples has an example which should help you. For e.g. you can deploy the aws-s3-to-efs-via-datasync stack and then compare your current setup v/s this stack.

Thanks, Rama

profile pictureAWS
EXPERT
Ramakrishna_Natarajan
answered 2 months ago
  • chrism
    2 months ago

    Thanks Rama. I was hoping to confine my answer to the AWS console options, or the CLI. My stack is quite different, so I will not be able to follow this. I specifically want to understand why non-blanks options are producing the mount error, and why blanks are not saving the files anywhere. Thanks though.

0

Hi,

DataSync is task based data copy tool. Which means you must run the task to copy data from a source to destination location. Based on your explanation, It appears, that you first ran the task and then later you copied the files into S3 bucket. Have you tried re-running the task?

When you create an EFS location definition, mount path is required which usually be '/' root path of the EFS file system or you can provide any custom folder name [which must be created at the EFS before]. If you run task without the folder path not created at the destination, you would see the error stating unable to mount.

amadhuku
answered 2 months ago
  • chrism
    2 months ago

    Thanks. The S3 has the files already [sic: "I have an S3 Bucket and an EFS created. The S3 contains the CSV file under /my_input_files/ folder"].

    I do have the "/" but I cannot see the copied files even when successful. I tried using a named folder which I already created in the EFS. Still get the same error.

    I think I will leave it at this. AWS is not working for me. In Azure I did the same job much easier. I was able to set up an end-to-end Batch using containers in an hour, including data sync, without knowing anything about Azure. For the same job in AWS, I am still stuck on a simple data sync task and a week later, it is still not up and running.

    Thank you for the responses. I will close this down as a resolution is no longer required.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content