aws sagemaker PySparkProcessor "print statements" not showing

Question

Hello,
How can I view print statement results from a PySparkProcessor script run through SageMaker jobs? I created a script to collect and manipulate data from s3, then save back the results to s3. Nothing is saved to s3 in the end. It is run from a SageMaker jupyter notebook.

I searched for my print statements in the job logs, but did find them.

Amazon SageMaker > Processing jobs > [job name-id] > View logs > Log stream > ...

Here is an excerpt of my code. Any other config info which would be helpful? Any tips? Thanks.
```
spark_processor = PySparkProcessor(
        base_job_name= job_name,
        framework_version="2.4",...

...

spark_processor.run(
        submit_app="src/preprocess.py",
        arguments=['--s3_output_path', save_location,
...                  ],
        spark_event_logs_s3_uri = s3_log_location,
        logs=True,
        wait=False,
    )

```

Njabs · Answer

Hi there,

As per the documentation for the PySparkProcessor class's `run` method, the definition for the log parameter is as follows: "Whether to show the logs produced by the job. Only meaningful when wait is True (default: True)" [1].

In your case I see you have set wait to False. May you please confirm if you see logs if you set the wait parameter to True.

```
spark_processor.run(
        submit_app="src/preprocess.py",
        arguments=['--s3_output_path', save_location,
...                  ],
        spark_event_logs_s3_uri = s3_log_location,
        logs=True,
        wait=True,
    )
```

Reference

[1] https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.spark.processing.PySparkProcessor

aws sagemaker PySparkProcessor "print statements" not showing

Contenus pertinents