How can I troubleshoot issues associated with connecting Git repositories to Amazon SageMaker notebook instances?

5 minute read
1

I'm trying to add a Git repository to an Amazon SageMaker notebook instance, but I'm getting one or more of the following errors:

"fatal: unable to access <Git repository URL>: the requested URL returned error: 403" "fatal: unable to access <Git repository  URL> : failed to connect to …: port 443: connection timed out" "fatal: unable to access <Git repository URL> could not resolve the host: …" "fatal: could not read Username for <Git repository URL>: terminal prompts disabled" "fatal: could not read Password for <Git repository URL>: terminal prompts disabled" "fatal: authentication failed for <Git repository URL>" "No space left on device while cloning Git repo. Please try using a larger volume"

Resolution

To add a new Git repository to your SageMaker notebook instance, see Add a Git repository to your Amazon SageMaker account. If you're trying to add a Git repository that's not hosted on AWS CodeCommit, you might get the following errors.

Errors associated with connectivity

If there are connectivity issues, you might get one of the following errors:

  • "fatal: unable to access <Git repository URL>: the requested URL returned error: 403"
  • "fatal: unable to access <Git repository URL>: failed to connect to …: port 443: connection timed out"
  • "fatal: unable to access <Git repository URL> could not resolve the host: …"

To resolve these errors, do the following:

  • Verify the notebook instance connectivity.
  • Verify the accuracy of the Git repository URL.

Verify the notebook instance connectivity

Be sure that notebook instance has successfully connected to the Git repository. This repository might be located either on an on-premises Git repository server or a public cloud server, such as GitHub, Bitbucket, or GitLab.

Check the network configuration that your notebook instance is using. The notebook instance might have one of the following network configurations:

  • The instance isn't attached to an Amazon Virtual Private Cloud (VPC).
  • Your Amazon VPC has direct internet access
  • Your VPC doesn't have direct internet access.

If you use a VPC, be sure that the subnets, security groups, network access control Lists (ACLs), NAT gateway, internet gateway, and route tables are configured correctly.

Verify the accuracy of the Git repository URL

Be sure that the Git repository URL that you're adding is in the following format:

  • GitHub URL: https://github.com/<workspace>/<repository.git>
  • Bitbucket URL: https://bitbucket.org/<workspace>/<repository.git>
  • GitLab URL: https://gitlab.com/<workspace>/<repository.git>

Be sure to replace the following:

  • workspace with the Git workspace name
  • repository.git with the code repository name that has the *.git file extension.

For example, if you're using Bitbucket as a Git repository where the name of your workspace is myworkspace, and your Git repository is myrepository.git, then you must use the following Git repository URL: https://bitbucket.org/myworkspace/myrepository.git

Errors associated with authentication

If there are authentication issues, you might get one of the following errors:

  • "fatal: could not read Username for <Git repository URL>: terminal prompts disabled"
  • "fatal: could not read Password for <Git repository URL>: terminal prompts disabled"
  • "fatal: authentication failed for <Git repository URL>"

To resolve these errors, do the following:

  • Verify the authentication details.
  • Verify that the AWS Identity and Access Management (IAM) user or roles has the required permissions to access AWS Secrets Manager.

Verify that the authentication details are accurate

Be sure that the username and password are accurate in Secrets Manager.

  • GitHub: If you have enabled two-factor authentication, then you must use a personal access token instead of a password. After this token is generated, you must update the password in Secrets Manager with this new token.
  • Bitbucket: Bitbucket doesn't support account passwords for Git authentication. It's a best practice to create App passwords using your Personal settings. After creating the App password, you must update the password in Secrets Manager.
  • GitLab: If you have enabled two-factor authentication, then you must use a personal access token instead of a password. After this token is generated, you must update the password in Secrets Manager with this new token.

Verify that the notebook instance's IAM role has the required permissions

Verify that the IAM role used for the notebook instance has the secretsmanager:GetSecretValue permission attached in its IAM policy. If the IAM role doesn't have the required permissions, then you can add the following policy to the IAM role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": [
        "example-secret-ARN"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "secretsmanager:ListSecrets",
      "Resource": "*"
    }
  ]
}

Note: example-secret-ARN is the ARN in Secrets Manager that has the secrets (username and password) for the Git repository authentication.

Errors associated with storage

If there are storage related issues, you might get the following error:

No space left on device while cloning Git repo. Please try using a larger volume

To resolve this error, verify that the notebook instance has enough storage. Be sure that the size of the free space in the notebook instance’s volume is larger than that of the Git repository that you're using to clone. If you run out of disk space, specify a larger value for VolumeSizeInGB. For more information, see Customize your notebook volume size, up to 16 TB, with Amazon SageMaker.


Related information

Associate Git repositories with SageMaker notebook instances

Using AWS CodeCommit with interface VPC endpoints

AWS OFFICIAL
AWS OFFICIALUpdated a year ago