SageMaker debugger built in rule CreateXgboostRule not generating report as expected

Question

I'm currently working with a SageMaker hosted XGBoost model; I've added the built in rule "CreateXgboostRule" to generate a training report, however, only the ProfilerReport is generated in the S3 rule-output folder - the expected result based on the dev doc is for a CreateXGBoostRule folder as well within this same folder.

The code I'm using is based directly on the example provided in: https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-training-xgboost-report.html

```
import boto3
import sagemaker
from sagemaker.estimator import Estimator
from sagemaker import image_uris
from sagemaker.debugger import Rule, rule_configs

rules=[
    Rule.sagemaker(rule_configs.create_xgboost_report())
]

region = boto3.Session().region_name
xgboost_container=sagemaker.image_uris.retrieve("xgboost", region, "1.2-1")

estimator=Estimator(
    role=sagemaker.get_execution_role()
    image_uri=xgboost_container,
    base_job_name="debugger-xgboost-report-demo",
    instance_count=1,
    instance_type="ml.m5.2xlarge",
    
    # Add the Debugger XGBoost report rule
    rules=rules
)

estimator.fit(wait=False)
```
I've tried rewriting the estimator a number of ways, verified "rules" is receiving an array of objects, tried different versions of XGBoost within the region, but everything still results in the built in rule only creating the ProfilerReport with no CreateXGBoostRule directory under rule-output.

Any ideas would be greatly appreciated! Thanks.

Darya · Answer

Hi Dennis,
I tried a code above in AWS SageMaker notebook and didn't get any ProfilerReport as well.

The possible issue is that there is no data to train hence nothing to report.
To prove this hypothesis I decided to take a sample XGBoost notebook and to add report functionality in it.
This [link ](https://docs.aws.amazon.com/sagemaker/latest/dg/howitworks-nbexamples.html) tells how to access sample notebooks. I used "xgboost_customer_churn.ipynb".

Here some changes that I made:

1. In order to be able to run it smoothly, instead of "1.6-1" XGBoost image I set the "1.2-1":

```
container = sagemaker.image_uris.retrieve("xgboost", sess.boto_region_name, "1.2-1")
```
2. I added reporting to the training cell:

```
from sagemaker.debugger import Rule, rule_configs

rules=[
    Rule.sagemaker(rule_configs.create_xgboost_report())
]
sess = sagemaker.Session()

xgb = sagemaker.estimator.Estimator(
    container,
    role,
    instance_count=1,
    instance_type="ml.m4.xlarge",
    output_path="s3://{}/{}/output".format(bucket, prefix),
    sagemaker_session=sess,
    base_job_name="debugger-xgboost-report-demo",
    rules=rules
)

xgb.set_hyperparameters(
    max_depth=5,
    eta=0.2,
    gamma=4,
    min_child_weight=6,
    subsample=0.8,
    verbosity=0,
    objective="binary:logistic",
    num_round=100,
)

xgb.fit({"train": s3_input_train, "validation": s3_input_validation})
```
All the rest I run without changes.

However, it was a little confusing to fing a ProfilerReport. In this particular example it had the path: `sagemaker-region-11122233/sagemaker/DEMO-xgboost-churn/output/debugger-xgboost-report-demo-2022-00-000/rule-output/CreateXgboostReport/`

In order to get this path you can use (run this code in the SageMaker notebook):

`xgb.output_path` - gives the first part of the path (including the bucket name)

`xgb.latest_training_job.job_name` - gives the second part of the path (training job name)

If you combine these two, you will get the full path towards the `rule-output` directory.

I hope this helps.

SageMaker debugger built in rule CreateXgboostRule not generating report as expected

Relevant content