Logical Directories not visible if empty

I heard back from an aws ticket which indicated:

As you may not be aware of this, there is no such thing as "folder" in S3. When you upload a file to S3, such as 'prefix/key', the console interprets that as if the file 'key' was inside the 'prefix' folder. However, in reality it's just an object named 'prefix/key', and as soon as you delete the file, the "folder" is gone as well. You can workaround this by clicking the button "Create Folder" within your S3 bucket, which creates a 0 byte object with a forward slash at the end (e.g prefix/), thus it displays the "folder" irrespective of whether there are files inside of it, since you now have:

prefix/  
prefix/object1  
prefix/object2  

If object1 and object2 are deleted, prefix/ still remains to act as the "folder".

That said, from the information provided, it seems to me that the issue you're experiencing might be a result of that behavior. Therefore, I'd suggest trying and creating the 0 byte object in S3 in the location your logical directory '/in' is mapped to, and see if that resolves the issue.

I understand the concept of prefix and key, however, this detail seems to conflict with the intended use of logical directories as defined within the aws SFTP transfer service. The documentation suggests that logical directories can created with a mapping to an S3 location. It does not say anything about there having to be objects at that location for it to be visible.

I understand I can use the aws console to create "folders". Can the 0 byte files with names that end with '/' be created using the API? My initial attempts to do so using the CLI have failed. These folders are dynamically provisioned for each user when created and having to use the console to create a "folder" is not really a viable solution.

A workaround could also include writing a dummy file of some kind with the desired prefix so that the mapped folder is not empty and therefore is displayed by the SFTP properly. However, having to create dummy files all over the place for folders that might otherwise be empty seems like a cumbersome solution, not to mention that the file would be visible to the user, possibly causing confusion inside a folder that is expected to be empty.

Can you inquire with the SFTP team to find out if this is the intended user experience for the logical directories feature?

Thanks.

The ticket response from support was:

Since Transfer for SFTP is using S3 as the back end storage, we are limited to the rules and constraints that S3 forces on all bucket owners unfortunately. You are correct, that the user experience is not going to be the same as if the storage were traditional "file and folder" storage. The workarounds you describe are the exact workarounds we give to customer in circumstances similar to yours. Today, there is no option but to deal with the workarounds.

What I have done, however, is turned your case into a feature request for the Transfer team. We can't change the rules of S3, but we may be able to make the 0 byte solution more manageable or automated. I have no timeline for when this feature may be engineered or introduced, but at least your request is logged with any other customer looking for a similar solution.

I'm aware of the limitations of S3 and I don't think it is necessary to change its' behaviour, but rather, change the behaviour of the SFTP service.

I think the important part of logical or virtual directories is that they are "pretend" directories defined for display in the SFTP client. I think for this feature to be correct, I think they should implement the virtual folders in my SFTP client, regardless of what exists (or doesn't) in S3. If the virtual folder is mapped to a valid S3 location that contains objects, then display them, otherwise, just display an empty folder. They have control over the SFTP service and I imagine could do this without changing S3 at all.

Thanks.

My name is Brian and I am with the Transfer for SFTP team.

What you are running into at the moment with the directory not showing up if nothing exists would be the current expected behavior. Thank you for your suggestions on this which I have relayed to our Product Management team to be added to our back log of improvements.

As a workaround in the mean time, you can use the S3 api to create 0 byte objects as place holders for your directory. If using the cli you would want to use the s3api call instead of s3 so you can utilize the put-object operation.
EX:
aws s3api put-object --bucket bucketname --key path/to/folder/

You'll want to make sure that the end of the key name ends in a / for it to be considered a folder.
https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object.html

Was this ever changed? We are creating folders using an SFTP client and occasionally, the folders are disappearing. We have logging turned on and do not see any DELETEs pointed at the folders (only the files within them).

The behavior seems inconsistent.

Can someone provide an update on this? Has anything changed since 2019, or are the 0 byte workarounds still required for logically defined folder paths to be displayed in an SFTP client if the folder contains no files?

Hello,

For the logical mapping to appear the Target mush exist on the destination. That means that the folders (0 byte object) for the users is still required.

AWS Transfer Family also supports EFS and you can also configure logical directories. The difference with S3 (regarding the folders) is that if you delete the content of the folder it will not disappear and you will not lose the logical mapping since its a file system and the directories exist (but again the folders of the target are required the same way with the 0 byte object on s3).

For example if I have the following mapping with my EFS both folder1 and folder2 should exist in EFS and the same applies if I use S3 (with s3 if the zero byte object does not exist but you have an object with that prefix the mapping will show, however, if you delete that object and the prefix no longer exists the mapping will not show). If we use EFS and we delete the content of the folder1 it will not remove the folder and the mapping will still appear when we do a list operation.
[
{
"Entry": "/logical1",
"Target": "/fs-XXXXXXXX/folder1"
},
{
"Entry": "/logical2",
"Target": "/fs-XXXXXXXX/folder2"
}
]

Please check the links below for more information regarding AWS Transfer Family with EFS:

• https://docs.aws.amazon.com/efs/latest/ug/using-aws-transfer-integration.html
• https://aws.amazon.com/blogs/aws/new-aws-transfer-family-support-for-amazon-elastic-file-system/

Kind Regards,
Panagiotis

I know this question is old, but since someone recently asked about this again, I figured I would share the solution that worked for me, given that I had similar requirements - I wanted every SFTP mailbox to have predefined set of folders available to the user. It might help others head in the right direction.

My SFTP transfer family set up uses an API gateway/Lambda function combo to authenticate users. As part of the logic in the authenticating Lambda function, I added logic so that once a user is deemed valid, any required mailbox/S3 folders are created before returning a successful authentication response. This way, the user always has the expected folder structure upon connecting.

The way I created the empty folders is like other responses have suggested - creating an empty object with the key ending in slash, and the key being the full path to the folder (i.e. for expected folder structure "mytest/myfiles", create an empty S3 object with key "mytest/myfiles/".

As fazcc73 suggested, we did the same thing, within the custom identity-provider, we create folders using the above methods.

One thing to be aware of: If you are using S3:ObjectCreated events or similar to do any work when files are uploaded, don't forget that the zero byte files you create to imitate folders will also generate those ObjectCreated events, so your handler needs to be able to deal with those events gracefully, as well as legitimate file uploads.

Hello,

I know this question is too old, but I'm facing the same problem in my current implementation, so just want to know, is there a permanent solution in place for this? Or we still need to follow the workarounds like 0 byte object or create folder using lambda etc.?