By using AWS re:Post, you agree to the Terms of Use

s3 - date folder structure for etl data processes (airflow, etc)

0

Trying to verify the best practice. Say I have a "folder" structure like so:

s3://my-bucket/events/2022/09/01
s3://my-bucket/events/2022/09/02
...

As an example, On 9/3, the client should export into the "folder" s3://my-bucket/events/2022/09/02 (after the 9/2 day).

We have a client who is running their etl on 9/3 and then placing the files on 2022/09/03 folder and we're trying to convince them to put it in the 9/2 folder.

Am I correct is saying that this is the best practice?

1 Answer
0

Hello,

I would like to inform you that there is no general guidance or best practices prescribed by AWS for the query that you are asking. That being said, it is a subjective question and depends from use-case to use-case. Personally, I would agree with you in convincing your client to put the ETL processed data in the same named folder as the date on which the data was produced.

SUPPORT ENGINEER
answered 23 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions