What's the load time for FSx for Lustre from S3?

Question

A customer is fronting S3 with FSx to interact with Sagemaker more efficiently. They will probably use scratch mode so they can turn on/off frequently, but I want to understand how long that initial load would be from S3 to see if it just makes sense to go the persistent route. I see in the FAQs that 'When linked to an S3 bucket, an FSx for Lustre file system transparently presents S3 objects as files and allows you to write changed data back to S3.' I'm assuming that means there is an initial load time from S3 to Lustre? Does it happen all at once or on an object-by-object basis, i.e. only when a certain object is requested for the first time?

What type of information can I give the customer about this initial latency around creating the file system?

Accepted Answer

It will take a few minutes to create the filesystem metadata representation of the S3 Bucket/prefix.  The amount of time to create the filesystem depends on the number of objects in the filesystem.  Once the file system is created, the data is lazy-loaded in the scratch FSx for Lustre based on file access, so the first access will be typical S3 latencies with subsequent access at sub-MS latencies for the scratch tier.  Alternately, the customer can pre-populate the data (or a subset of the data) after the file system is created using the Lustre HSM commands:  https://docs.aws.amazon.com/fsx/latest/LustreGuide/preload-file-contents-hsm.html

What's the load time for FSx for Lustre from S3?

Relevant content