Why am I receiving the errors UPSTREAM_ERROR, UPSTREAM_NOT_FOUND, or CLIENT_UNAUTHORIZED when accessing my Lightsail instance using the browser-based SSH console?

Short description

If you receive an error when you connect to your Lightsail instance, and the error relates to SSH, then you might see one of the following messages:

• "Your instance encountered an error and has closed the connection. Try again or contact customer support. UPSTREAM_ERROR [515]"
• "An error occurred and we were unable to connect or stay connected to your instance. If this instance has just started up, try again in a minute or two. UPSTREAM_NOT_FOUND [519]"
• "Login failed. If this instance has just started up, try again in a minute or two. CLIENT_UNAUTHORIZED [769]"

The UPSTREAM_ERROR [515] and UPSTREAM_NOT_FOUND [519] errors indicate that your Lightsail instance is can't be reached when using SSH. The following are common reasons for these errors:

• Instance boot failures, instance status check failures. or resource over-utilization on the instance.
• An OS-level firewall is blocking SSH port access.
• The default SSH port (22) is changed to a different one.
• The SSH service is down.

The CLIENT_UNAUTHORIZED [769] error indicates that there's an SSH authentication issue in your Lightsail instance. The following are common reasons for this error:

• Misconfiguration of the Lightsail system key in /etc/ssh/lightsail_instance_ca.pub.
• You upgraded your Ubuntu instance from an older version to 20.04 or later.

Resolution

Instance boot failures, instance status check failures, or resource over-utilization on the instance

View the status check metrics of your instance to determine if the instance failed the system status check or the instance status check.

System status check

If the system status check failed, then the instance's underlying hardware is unhealthy. To fix the issue, stop and then start the instance. This migrates the instance to healthy hardware.

Warning: Be aware that the public IP address of the instance changes on every stop and start of the instance. If you want the IP address to stay the same on every stop and start of the instance, then attach a static IP address.

Instance status check

If the instance status check failed, then an operating system-level issues might be causing boot errors. Or, the instance's resources, such as CPU or memory, might be over-utilized. For troubleshooting steps, see How can I troubleshoot common issues that cause my Lightsail instance to be unresponsive?

An OS-level firewall is blocking SSH port access

Browser-based SSH access uses the Amazon internal IP address to connect to your Lightsail Instance. Some OS-level firewalls and access control files allow SSH access only to one set of IP addresses and deny all others. In this case, you receive the UPSTREAM_NOT_FOUND [519] or UPSTREAM_ERROR [515] error. The same issue occurs if a firewall is blocking SSH access completely.

Note: Examples of OS-level firewalls include Iptables and UFW, while an example of a control file is /etc/hosts.deny (Hosts Access Control in cPanel based Instances).

To resolve this issue, do one of the following:

• If SSH access to the instance using the terminal or PuTTY application works, then log in to the instance and remove the deny rules from firewall and /etc/hosts.deny file.
• If you have a cPanel instance, then access the WHM console to remove the firewall rules.
• If you can't connect to the instance by using SSH from the terminal or PuTTY application, then deactivate the firewall using a launch script. The launch script can be added only while launching an instance. To use a launch script to deactivate OS-level firewalls, Iptables, and UFW, do the following:

1.    Open the Amazon Lightsail console.

2.    Create a manual snapshot of the instance.

3.    On the Snapshots tab, under Manual snapshots, select the three dots next to the new snapshot.

4.    Select Create new instance.

5.    Select the same Availability Zone as the previous instance.

6.    Select Add launch script, and then add the following script.

Note: The following example deactivates the UFW firewall, flushes all the iptable chains or firewall rules, and deactivates the /etc/hosts.deny file by renaming it.

sudo ufw disable
sudo iptables -F
sudo mv /etc/hosts.deny /etc/hosts.deny_backup
sudo touch /etc/hosts.deny
sudo systemctl enable sshd
sudo systemctl restart sshd

7.    Choose a new instance plan, or use the same plan as the previous instance.

8.    Enter a name for the instance, and then select Create instance.

After the new instance starts running, wait for 10 to 15 minutes, and then try to connect to it using the browser-based SSH console.

Note: If the previous instance had a static IP address, then you can use it on the new instance. Detach the static IP address, and then attach it to the new instance from the Networking tab. For more information, see Static IP addresses in Amazon Lightsail.

The SSH service is down

If the SSH service isn't running or active on the instance, then the SSH connection fails and you receive the UPSTREAM_NOT_FOUND [519] error. To troubleshoot this issue, configure the AWS Systems Manager Session Manager service for your Lightsail instance. After configuring Session Manager, access the Instance without the SSH service, and then fix the SSH issue.

Basic troubleshooting steps for SSH issues include:

• Review the SSH authentication logs in /var/log/auth.log or /var/log/secure file, depending on the OS distribution, to identify the errors.
• Test the SSH configuration file syntax and then correct any errors.

sudo sshd -t
sudo systemctl restart sshd

Misconfiguration of the Lightsail system key /etc/ssh/lightsail_instance_ca.pub

Lightsail uses the system key /etc/ssh/lightsail_instance_ca.pub to activate browser-based SSH access. If this file is missing, then SSH authentication fails and you receive the **CLIENT_UNAUTHORIZED [769]**error. The same error occurs when the Lightsail system key isn't specified in the TrustedUserCAKeys parameter in the SSH configuration file /etc/ssh/sshd_config.

If the SSH connection to the instance through the terminal or PuTTY application works, then log in to the instance and confirm that the /etc/ssh/sshd_config file exists and that it contains the ssh-rsa key.

If the file is missing, then recreate the file using the following steps:

1.    Run the following command and confirm that you get the ssh-rsa key in the command output:

sudo cat /var/lib/cloud/instance/user-data.txt | grep ^ssh-rsa

2.    If the command returns the ssh-rsa key in the output, then run the following commands to copy it to /etc/ssh/lightsail_instance_ca.pub:

sudo sh -c "cat /var/lib/cloud/instance/user-data.txt | grep ^ssh-rsa > /etc/ssh/lightsail_instance_ca.pub"
    sudo sh -c "echo >> /etc/ssh/sshd_config" 
    sudo sh -c "echo 'TrustedUserCAKeys /etc/ssh/lightsail_instance_ca.pub' >> /etc/ssh/sshd_config"
    sudo systemctl restart sshd

3.    If you don't get an ssh-rsa key from step 1, or if you can't SSH to the instance using the terminal or PuTTY application, then create a manual snapshot of the instance and launch a new instance with that snapshot. The Lightsail system key is added to the server automatically at instance launch.

You've upgraded from an older Ubuntu version to 20.04 or later

For instances running Ubuntu 20.04 or later, you must allow certificate authorities (CAs) to use the ssh-rsa algorithm to sign certificates. Otherwise, authentication fails and you receive the CLIENT_UNAUTHORIZED [769] error. CAs are allowed to use the ssh-rsa algorithm by default in Ubuntu 20.04 instances. However, if you upgrade your older Ubuntu version to 20.04, then you must manually allow the CAs. To do this, follow these steps:

1.    Connect to the instance using an SSH client such as the terminal or PuTTY application.

2.    Open the SSH configuration file (/etc/ssh/sshd_config).

3.    Add CA parameters to the file, depending on your version of Ubuntu:

       If the OS version is upgraded to Ubuntu 20.04:  Manually enter the CASignatureAlgorithms parameter line into the sshd_config file:

$ sudo vi /etc/ssh/sshd_config
$ cat /etc/ssh/sshd_config | egrep "CASignature"
CASignatureAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa

       If the OS version is upgraded to Ubuntu 22.04:  Manually enter the CASignatureAlgorithms and PubkeyAcceptedAlgorithms parameter lines into the sshd_config file:

$ sudo vi /etc/ssh/sshd_config
$ cat /etc/ssh/sshd_config | egrep "CASignature|PubkeyAccepted"
CASignatureAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
PubkeyAcceptedAlgorithms +ssh-rsa-cert-v01@openssh.com,ssh-rsa

4.    Run the following command to validate the sshd configuration file content:

$ sudo sshd -T

5.    Run the following command to restart the sshd service:

$ sudo systemctl restart sshd

---