I followed the walkthrough in this AWS Big Data Blog post to connect the Apache Spark Troubleshooting Agent for Amazon EMR to AWS DevOps Agent as an MCP capability provider:
https://aws.amazon.com/blogs/big-data/accelerate-apache-spark-debugging-on-amazon-emr-with-aws-devops-agent/
Setup completed successfully (private connection, capability provider, agent space) and I submitted the same deliberately-failing demo job (customer_events_aggregator.py, spark.executor.pyspark.memory=256m). I'm seeing two issues I'd like clarification on.
Issue 1: Agent's own investigation reaches a wrong root cause, apparently due to a permissions gap
When I asked AWS DevOps Agent to investigate the CloudWatch alarm, it concluded the failure was due to ApplicationMaxCapacityExceededException (disk: 42 GB) from dynamic allocation requesting additional executors. When I checked the IAM role actually used for the agent's native AWS API calls (DevOpsAgentRole-AgentSpace-*, auto-provisioned when the agent space was created), it does not have S3 permissions scoped to the demo bucket, and I could reproduce an AccessDenied from that role when trying to read the job's log objects directly.
Separately, the capability provider's SigV4 signing role (the one created by the CloudFormation template, e.g. TroubleshootingRoleArn) does have s3:GetObject/s3:ListBucket on the demo bucket, and is presumably what the MCP server itself uses server-side to read the Spark History Server data.
Question: if the MCP tool (analyze_spark_workload) has its own S3 access via that role, why does the agent's investigation still appear to fall back to a symptom-level/incorrect conclusion instead of the tool's deeper analysis? Is there a scenario where the agent doesn't invoke the MCP tool's findings as authoritative, or falls back to native-tool reasoning when something in the MCP response is incomplete? Is this expected/documented behavior, and is there a way to see (or force) which source the final root cause narrative was actually built from?
Issue 2: Discrepancy between the actual raw-log evidence and the Root Cause tab's stated config
By manually pulling the executor's archived stderr from S3 (SPARK_EXECUTOR/<id>/archived/stderr/stderr_0.gz), I found the definitive cause: PySpark sets Setting mem limits to 134217728 of max 134217728 (128 MiB = half of spark.executor.pyspark.memory=256m, split across spark.executor.cores=2), the worker process is killed for exceeding that pool, and the JVM reports it as java.net.SocketException: Connection reset in stage 3.0 (repeated for all 4 task attempts) — not a JVM heap OOM.
However, both the blog's own "Root cause" screenshot and my own agent's Root Cause tab attribute the failure to "only 1GB executor memory" (i.e., spark.executor.memory=1g), which is a separate, unrelated JVM heap setting from the actual spark.executor.pyspark.memory=256m pool that was exhausted.
Question: is analyze_spark_workload conflating spark.executor.memory and spark.executor.pyspark.memory when generating the human-readable root cause summary? Is this a known limitation, and is there a more precise field in the MCP tool's structured output that correctly distinguishes JVM heap exhaustion from PySpark worker subprocess memory exhaustion?
Environment:
Region: us-east-1
Demo stack: spark-troubleshooting-demo (from the blog's CloudFormation template)
Any clarification from the AWS DevOps Agent / EMR Serverless team would be appreciated. Happy to share full logs/screenshots if useful.
