Sagemaker Studio in VPC Only fails to create Notebook job with "Unexpected error occurred during creation of job."

Question

The CloudWatch logs show following error
```
    Traceback (most recent call last):
      File "/opt/conda/envs/studio/lib/python3.9/site-packages/jupyter_scheduler/handlers.py", line 194, in post
        job_id = await ensure_async(self.scheduler.create_job(CreateJob(**payload)))
      File "/opt/conda/envs/studio/lib/python3.9/site-packages/jupyter_server/utils.py", line 182, in ensure_async
        result = await obj
      File "/opt/conda/envs/studio/lib/python3.9/site-packages/sagemaker_scheduling/logging.py", line 109, in wrapper
        raise excep
      File "/opt/conda/envs/studio/lib/python3.9/site-packages/sagemaker_scheduling/logging.py", line 105, in wrapper
        return await func(*args, **kwargs)
      File "/opt/conda/envs/studio/lib/python3.9/site-packages/sagemaker_scheduling/scheduler.py", line 210, in create_job
        s3_file_uploader = await self._prepare_job_artifacts(
      File "/opt/conda/envs/studio/lib/python3.9/site-packages/sagemaker_scheduling/scheduler.py", line 168, in _prepare_job_artifacts
        input_uri = S3URI(runtime_environment_parameters.s3_input)
      File "/opt/conda/envs/studio/lib/python3.9/site-packages/sagemaker_scheduling/runtime_environment_parameters.py", line 40, in s3_input
        return self.parameters.get(RuntimeEnvironmentParameterName.S3_INPUT.value)
    AttributeError: 'NoneType' object has no attribute 'get'
[E 2023-03-16 15:36:01.070 SchedulerApp] 'NoneType' object has no attribute 'get' Traceback (most recent call last): File "/opt/conda/envs/studio/lib/python3.9/site-packages/jupyter_scheduler/handlers.py", line 194, in post job_id = await ensure_async(self.scheduler.create_job(CreateJob(**payload))) File "/opt/conda/envs/studio/lib/python3.9/site-packages/jupyter_server/utils.py", line 182, in ensure_async result = await obj File "/opt/conda/envs/studio/lib/python3.9/site-packages/sagemaker_scheduling/logging.py", line 109, in wrapper raise excep File "/opt/conda/envs/studio/lib/python3.9/site-packages/sagemaker_scheduling/logging.py", line 105, in wrapper return await func(*args, **kwargs) File "/opt/conda/envs/studio/lib/python3.9/site-packages/sagemaker_scheduling/scheduler.py", line 210, in create_job s3_file_uploader = await self._prepare_job_artifacts( File "/opt/conda/envs/studio/lib/python3.9/site-packages/sagemaker_scheduling/scheduler.py", line 168, in _prepare_job_artifacts input_uri = S3URI(runtime_environment_parameters.s3_input) File "/opt/conda/envs/studio/lib/python3.9/site-packages/sagemaker_scheduling/runtime_environment_parameters.py", line 40, in s3_input return self.parameters.get(RuntimeEnvironmentParameterName.S3_INPUT.value) AttributeError: 'NoneType' object has no attribute 'get'
```
Also all the "advanced options" are missing in the "create notebook job" dialogue.

The setting is isolated VPC with permissions updated accordingly: https://docs.aws.amazon.com/sagemaker/latest/dg/scheduled-notebook-policies.html

The VPC endpoints for S3, SageMaker API and Runtime, SSM, STS, Metrics, Logs, ECR API and ECR DKR are deployed in the VPC. Notebooks are working fine.

Any idea what could be wrong?

Accepted Answer

Thanks for reporting the issue.

Can you try also configuring following two vpc endpoints? 
1. Amazon EC2
2. Amazon EventBridge
Also, if you have s3 vpc gateway endpoint policy with fine control of allowed s3 bucet, please allow s3 access for sagemakerheadlessexecution-prod-* like below. 
FYI - you can find more reference link from https://docs.aws.amazon.com/sagemaker/latest/dg/create-notebook-auto-execution-advanced.html

```
{
   "Action":[
      "s3:*"
   ],
   "Resource":[
      "arn:aws:s3:::sagemakerheadlessexecution-prod-*",
      "arn:aws:s3:::sagemakerheadlessexecution-prod-*/*"
   ],
   "Effect":"Allow",
   "Sid":"SCTASK14554266"
}
```

Sagemaker Studio in VPC Only fails to create Notebook job with "Unexpected error occurred during creation of job."

Relevant content