s3 - date folder structure for etl data processes (airflow, etc)

Question

Trying to verify the best practice. 
Say I have a "folder" structure like so:

```
s3://my-bucket/events/2022/09/01
s3://my-bucket/events/2022/09/02
...
```
As an example, 
On 9/3, the client should export into the "folder" `s3://my-bucket/events/2022/09/02`  (after the 9/2 day).

We have a client who is running their etl on 9/3 and then placing the files on `2022/09/03` folder and we're trying to convince them to put it in the **9/2** folder.

Am I correct is saying that this is the best practice?

Answer

Hello,

I would like to inform you that there is no general guidance or best practices prescribed by AWS for the query that you are asking.  That being said, it is a subjective question and depends from use-case to use-case. Personally, I would agree with you in convincing your client to put the ETL processed data in the same named folder as the date on which the data was produced.

s3 - date folder structure for etl data processes (airflow, etc)

相关内容