Training a classifier on parquet with SageMaker ?

Question

Hi,

What parquet data loading logic is known to work well to train with SageMaker on parquet? ml-io? pyarrow? any examples? That would be to train a classifier, either logistic regression, XGBoost or custom TF.

Accepted Answer

XGBoost as a framework container (v0.90+) can read parquet for training (see example [notebook][1]).  
The full list of valid content types are CSV, LIBSVM, PARQUET, RECORDIO_PROTOBUF (see [source][2])

Additionally:  
[Uber Petastorm][3] for reading parquet into Tensorflow, Pytorch, and PySpark inputs.   
As XGBoost accepts numpy, you can convert from PySpark to numpy/pandas using the mentioned PyArrow.

[1]: https://github.com/awslabs/amazon-sagemaker-examples/blob/caf9363c0242d0da2de7f5765e7318fd843ce4c3/introduction_to_amazon_algorithms/xgboost_abalone/xgboost_parquet_input_training.ipynb
  [2]: https://github.com/aws/sagemaker-xgboost-container/blob/5e778770e009ce989e288e7bbc1255556129e75b/src/sagemaker_xgboost_container/data_utils.py#L40
  [3]: https://github.com/uber/petastorm

Training a classifier on parquet with SageMaker ?

Relevant content