By using AWS re:Post, you agree to the Terms of Use

Questions tagged with Amazon SageMaker Model Building

Sort by most recent
  • 1
  • 12 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

No such file or directory: '/opt/ml/input/data/test/revenue_train.csv' Sagemaker [SM_CHANNEL_TRAIN]

I am trying to deploy my RandomForestClassifier on Amazon Sagemaker using Python SDK. I have been following this example https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-script-mode/sagemaker-script-mode.ipynb but keep getting an error that the train file was not found. I think the file were not uploaded to the correct channel. When I run the script as follows it works fine. ``` ! python script_rf.py --model-dir ./ \ --train ./ \ --test ./ \ ``` This is my script code: ``` # inference functions --------------- def model_fn(model_dir): clf = joblib.load(os.path.join(model_dir, "model.joblib")) return clf if __name__ =='__main__': print('extracting arguments') parser = argparse.ArgumentParser() # hyperparameters sent by the client are passed as command-line arguments to the script. parser.add_argument('--max_depth', type=int, default=2) parser.add_argument('--n_estimators', type=int, default=100) parser.add_argument('--random_state', type=int, default=0) # Data, model, and output directories parser.add_argument('--model-dir', type=str, default=os.environ.get('SM_MODEL_DIR')) parser.add_argument('--train', type=str, default=os.environ.get('SM_CHANNEL_TEST')) parser.add_argument('--test', type=str, default=os.environ.get('SM_CHANNEL_TEST')) parser.add_argument('--train-file', type=str, default='revenue_train.csv') parser.add_argument('--test-file', type=str, default='revenue_test.csv') args, _ = parser.parse_known_args() print('reading data') train_df = pd.read_csv(os.path.join(args.train, args.train_file)) test_df = pd.read_csv(os.path.join(args.test, args.test_file)) if len(train_df) == 0: raise ValueError(('There are no files in {}.\n').format(args.train, "train")) print('building training and testing datasets') attributes = ['available_minutes_100','ampido_slots_amount','ampido_slots_amount_100','ampido_slots_amount_200','ampido_slots_amount_300','min_dist_loc','count_event','min_dist_phouses','count_phouses','min_dist_stops','count_stops','min_dist_tickets','count_tickets','min_dist_google','min_dist_psa','count_psa'] X_train = train_df[attributes] X_test = test_df[attributes] y_train = train_df['target'] y_test = test_df['target'] # train print('training model') model = RandomForestClassifier( max_depth =args.max_depth, n_estimators = args.n_estimators) model.fit(X_train, y_train) # persist model path = os.path.join(args.model_dir, "model_rf.joblib") joblib.dump(model, path) print('model persisted at ' + path) # print accuracy and confusion matrix print('validating model') y_pred=model.predict(X_test) print('Confusion Matrix:') result = confusion_matrix(y_test, y_pred) print(result) print('Accuracy:') result2 = accuracy_score(y_test, y_pred) print(result2) ``` the error is raised in the train_df line of the script (FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/input/data/test/revenue_train.csv'). I tried specifying the input parameters: ``` # change channel input dirs inputs = { "train": "ampido-exports/production/revenue_train", "test": "ampido-exports/production/revenue_test", } from sagemaker.sklearn.estimator import SKLearn enable_local_mode_training = False hyperparameters = {"max_depth": 2, 'random_state':0, "n_estimators": 100} if enable_local_mode_training: train_instance_type = "local" inputs = {"train": trainpath, "test": testpath} else: train_instance_type = "ml.c5.xlarge" inputs = {"train": trainpath, "test": testpath} estimator_parameters = { "entry_point": "script_rf.py", "framework_version": "1.0-1", "py_version": "py3", "instance_type": train_instance_type, "instance_count": 1, "hyperparameters": hyperparameters, "role": role, "base_job_name": "randomforestclassifier-model", 'channel_input_dirs' : inputs } estimator = SKLearn(**estimator_parameters) estimator.fit(inputs) ``` but i still get the error FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/input/data/test/revenue_train.csv
1
answers
0
votes
35
views
asked a month ago

SageMaker XGBoost Parquet Example Code Fails and Errors out. Bug?

Hi, I'm trying to run the SageMaker XGBoost Parquet example [linked here](https://sagemaker-examples.readthedocs.io/en/latest/introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.html). I followed the exact same steps but using my own data. I uploaded my data, converted it to a pandas df. The train_df shape is (15279798, 32) while the test_df shape is (150848, 32). I then converted it to parquet files and uploaded it to an S3 bucket - per example instructions. My error is as follows: ``` Failure reason AlgorithmError: framework error: Traceback (most recent call last): File "/miniconda3/lib/python3.7/site-packages/sagemaker_xgboost_container/data_utils.py", line 422, in _get_parquet_dmatrix_pipe_mode data = np.vstack(examples) File "<__array_function__ internals>", line 6, in vstack File "/miniconda3/lib/python3.7/site-packages/numpy/core/shape_base.py", line 283, in vstack return _nx.concatenate(arrs, 0) File "<__array_function__ internals>", line 6, in concatenate ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 32 and the array at index 1 has size 9 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_trainer.py", line 84, in train entrypoint() File "/miniconda3/lib/python3.7/site-packages/sagemaker_xgboost_container/training.py", line 94, in main train(framework.tr ``` But I'm confused because the train and test are the same shape and I added no extra code. My code below: ``` # requires PyArrow installed train.to_parquet("Xgb_train.parquet") test.to_parquet("Xgb_test.parquet") %%time sagemaker.Session().upload_data( "Xgb_train.parquet", bucket=bucket, key_prefix=prefix + "/" + "Ptrain" ) sagemaker.Session().upload_data( "Xgb_test.parquet", bucket=bucket, key_prefix=prefix + "/" + "Ptest" ) container = sagemaker.image_uris.retrieve("xgboost", region, "1.2-2") %%time import time from time import gmtime, strftime job_name = "xgboost-parquet-example-training-" + strftime("%Y-%m-%d-%H-%M-%S", gmtime()) print("Training job", job_name) # Ensure that the training and validation data folders generated above are reflected in the "InputDataConfig" parameter below. create_training_params = { "AlgorithmSpecification": {"TrainingImage": container, "TrainingInputMode": "Pipe"}, "RoleArn": role, "OutputDataConfig": {"S3OutputPath": bucket_path + "/" + prefix + "/single-xgboost"}, "ResourceConfig": {"InstanceCount": 1, "InstanceType": "ml.m5.2xlarge", "VolumeSizeInGB": 20}, "TrainingJobName": job_name, "HyperParameters": { "max_depth": "5", "eta": "0.2", "gamma": "4", "min_child_weight": "6", "subsample": "0.7", "objective": "reg:linear", "num_round": "10", "verbosity": "2", }, "StoppingCondition": {"MaxRuntimeInSeconds": 3600}, "InputDataConfig": [ { "ChannelName": "train", "DataSource": { "S3DataSource": { "S3DataType": "S3Prefix", "S3Uri": bucket_path + "/" + prefix + "/Ptrain", "S3DataDistributionType": "FullyReplicated", } }, "ContentType": "application/x-parquet", "CompressionType": "None", }, { "ChannelName": "validation", "DataSource": { "S3DataSource": { "S3DataType": "S3Prefix", "S3Uri": bucket_path + "/" + prefix + "/Ptest", "S3DataDistributionType": "FullyReplicated", } }, "ContentType": "application/x-parquet", "CompressionType": "None", }, ], } client = boto3.client("sagemaker", region_name=region) client.create_training_job(**create_training_params) print(client) status = client.describe_training_job(TrainingJobName=job_name)["TrainingJobStatus"] print(status) while status != "Completed" and status != "Failed": time.sleep(60) status = client.describe_training_job(TrainingJobName=job_name)["TrainingJobStatus"] print(status) ```
1
answers
0
votes
83
views
asked 6 months ago
  • 1
  • 12 / page