By using AWS re:Post, you agree to the Terms of Use

What value should I set for directory_path for the Amazon SageMaker SDK with FSx as data source?

1

What value should I set for the directory_path parameter in FileSystemInput for the Amazon SageMaker SDK?

Here is some information about my Amazon FSx for Lustre file system:

  • My FSx ID is fs-0684xxxxxxxxxxx.
  • My FSx has the mount name lhskdbmv.
  • The FSx maps to an Amazon S3 bucket with files (without extra prefixes in their keys)

My attempts to describe the job and the results are the following:

Attempt 1:

fs = FileSystemInput(
    file_system_id='fs-0684xxxxxxxxxxx',
    file_system_type='FSxLustre',
    directory_path='lhskdbmv',
    file_system_access_mode='ro')

Result:

estimator.fit(fs) returns ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: FileSystem DirectoryPath 'lhskdbmv' for channel 'training' is not absolute or normalized. Please ensure you don't have a trailing "/", and/or "..", ".", "//" in the path.

Attempt 2:

fs = FileSystemInput(
    file_system_id='fs-0684xxxxxxxxxxx',
    file_system_type='FSxLustre',
    directory_path='/',
    file_system_access_mode='ro')

Result:

ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: The directory path for FSx Lustre file system fs-068406952bf758bac is invalid. The directory path must begin with mount name of the file system.

Attempt 3:

fs = FileSystemInput(
    file_system_id='fs-0684xxxxxxxxxxx',
    file_system_type='FSxLustre',
    directory_path='fsx',
    file_system_access_mode='ro')

Result:

ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: FileSystem DirectoryPath 'fsx' for channel 'training' is not absolute or normalized. Please ensure you don't have a trailing "/", and/or "..", ".", "//" in the path.

1 Answer
0
Accepted Answer

The directory_path parameter must point to /mountname/path/to/specific/folder/in-file-system. The value of mountname is returned in the CreateFileSystem API operation response. It is also returned in the response of the describe-file-systems AWS Command Line Interface (AWS CLI) command and the DescribeFileSystems API operation.

For your use case, the response might look similar to the following: mountName = lhskdbmv

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions