1개 답변
- 최신
- 최다 투표
- 가장 많은 댓글
0
Hi there,
As per the documentation for the PySparkProcessor class's run
method, the definition for the log parameter is as follows: "Whether to show the logs produced by the job. Only meaningful when wait is True (default: True)" [1].
In your case I see you have set wait to False. May you please confirm if you see logs if you set the wait parameter to True.
spark_processor.run(
submit_app="src/preprocess.py",
arguments=['--s3_output_path', save_location,
... ],
spark_event_logs_s3_uri = s3_log_location,
logs=True,
wait=True,
)
Reference
관련 콘텐츠
- AWS 공식업데이트됨 3년 전
- AWS 공식업데이트됨 2년 전
Thanks for the help. Yes, I tried setting 'logs' and 'wait' to 'True'. I am still unable to find my debug statements. Below is a snippet from the submitted job script. Any other suggestions? Thanks.
def main(): print("*** in main", flush=True) logging.info("*** in main info") logging.debug("*** in main debug") ... if name == "main": main()
Below are the areas I am looking for the debug statements.