xgboost sagemaker batch transform job output in multiple lines

Question

Hello,

I've just trained a churn prediction model with XGBoost algorithm, based on the SageMaker example notebooks.  I've created SageMaker batch transformation jobs using this model using input from CSV file with multiple records, however the output file is a single record CSV containing all the inferences in a single comma separated row.  The result is that I'm not able to use the "Join source" feature with "Input  - Merge input data with job output" since the input and output files must match the number of records. I've tried with different batch job configurations but I always get the same single line output file.

Do you know if is there any configuration that allows me to merge input and output in order to have a direct association between an input column with its inference result? Is this a restriction from the XGBoost algorithm built-in implementation?

Accepted Answer

Sounds like a configuration issue, this algorithm should be able to output proper output CSVs.

Are you using `accept="text/csv"` and `assemble_with="Line"` on your `Transformer`? Is your `strategy` set to `SingleRecord` or `MultiRecord`?

And `split_type="Line"`, `content_type="text/csv"` on the `.transform()` call?

I have had custom algorithms accidentally output row vectors instead of column vectors for multi-record batches in the past (because they gave a 1D output which the default serializer interpreted as a row), but not built-in algorithms.

Dropping to `SingleRecord` could be a last resort (forcing Batch Transform itself to handle the serialization), but would decrease efficiency/speed.

xgboost sagemaker batch transform job output in multiple lines

Relevant content