- 최신
- 최다 투표
- 가장 많은 댓글
First, make sure you have the proper AWS credentials for the worker running the spark-submit job. This will depend on what you're using to run the task (for example, Glue Job execution role, Fargate execution role, EC2 instance profile, etc.). Once you have this you can set the Amazon S3 bucket you want to save to as the output path parameter. You can use Spark's 'save' method to write the results to this output path. For example:
val outputDataFrame: data = // your data
outputDataFrame.write.parquet("s3://yourbucket/output")
Depending on where you run your job, you can gather your application logs in CloudWatch. EC2, Glue, Fargate, EKS, ECS all integrate with Amazon CloudWatch so you can enable the execution role to write job to CloudWatch. You can find your application logs there. It's then up to you if you want to send those logs to other storage destinations like S3, Splunk, DataDog, etc.
관련 콘텐츠
- AWS 공식업데이트됨 2년 전
- AWS 공식업데이트됨 일 년 전
- AWS 공식업데이트됨 3년 전