Training a classifier on parquet with SageMaker ?



What parquet data loading logic is known to work well to train with SageMaker on parquet? ml-io? pyarrow? any examples? That would be to train a classifier, either logistic regression, XGBoost or custom TF.

  • Hi, I'm facing the same issue but for testing. It doesn't seem that testing in Sagemaker accepts PyArrow or parquet files. Do you know if Sagemaker does accept parquet files for testing or only training? If not, whats the go around?

asked 3 years ago334 views
1 Answer
Accepted Answer

XGBoost as a framework container (v0.90+) can read parquet for training (see example notebook).
The full list of valid content types are CSV, LIBSVM, PARQUET, RECORDIO_PROTOBUF (see source)

Uber Petastorm for reading parquet into Tensorflow, Pytorch, and PySpark inputs.
As XGBoost accepts numpy, you can convert from PySpark to numpy/pandas using the mentioned PyArrow.

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions