By using AWS re:Post, you agree to the Terms of Use

xgboost sagemaker batch transform job output in multiple lines



I've just trained a churn prediction model with XGBoost algorithm, based on the SageMaker example notebooks. I've created SageMaker batch transformation jobs using this model using input from CSV file with multiple records, however the output file is a single record CSV containing all the inferences in a single comma separated row. The result is that I'm not able to use the "Join source" feature with "Input - Merge input data with job output" since the input and output files must match the number of records. I've tried with different batch job configurations but I always get the same single line output file.

Do you know if is there any configuration that allows me to merge input and output in order to have a direct association between an input column with its inference result? Is this a restriction from the XGBoost algorithm built-in implementation?

asked 2 years ago145 views
1 Answer
Accepted Answer

Sounds like a configuration issue, this algorithm should be able to output proper output CSVs.

Are you using accept="text/csv" and assemble_with="Line" on your Transformer? Is your strategy set to SingleRecord or MultiRecord?

And split_type="Line", content_type="text/csv" on the .transform() call?

I have had custom algorithms accidentally output row vectors instead of column vectors for multi-record batches in the past (because they gave a 1D output which the default serializer interpreted as a row), but not built-in algorithms.

Dropping to SingleRecord could be a last resort (forcing Batch Transform itself to handle the serialization), but would decrease efficiency/speed.

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions