By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

After stopping and starting ec2, there is no ssh connection and the volume is not mounted on another device.

0

The ec2 instance used as t2.xlarge was using two root volumes and two data volumes as ebs. The root of the ebs is standard type and the data volume is st1. To change the type from t2.xlarge to t3.xlarge, the instance was stopped and restarted after changing the type. However, the ssh connection did not work. I checked the network and firewall, it was the same as before, and it was not blocked or connected. (connection rejected error)

So, after creating a new instance, I confirmed that the ssh connection was working, and after disconnecting the newly created ebs, I tried to connect to the previous ebs, but the connection was not possible. I thought it was strange, so I attached the newly created volume again, connected the old root volume as an additional volume, and mounted it, and it worked normally. But the problem was the ebs I was using as the data volume. I also tried to mount it to see if the data remained properly, but "can't read superblock on /dev/nvme1n1p1." error message was generated. So I did xfs_repair the volume, but the result message could not find valid secondary bad block came out and it still couldn't be mounted. Another attempt was to create a snapshot of the volume and attach and mount the volume created as a snapshot to a new instance, but the same error occurred.

What is the cause of this and what is the solution? Is there a way to keep backups to prevent this from happening? It's not my mistake, but it seems to be a problem with AWS.

Topics
ComputeStorage
Tags
Amazon EC2Amazon Elastic Block Store
Language
English
joker
asked 8 months ago334 views
1 Answer
0

Lots to unpack here. To answer your last question first, you can setup automated snapshots to backup your EBS volumes regularly (weekly, daily, hourly, or whatever you require) using Amazon Data Lifecycle Manager https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html

Migrating from t2 to t3 isn't as straightforward as appears at face value. They use different virtualisation (t2 uses Xen, t3 uses Nitro) so the new instance would need the correct ENA and NVMe drivers https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/resize-limitations.html (it's a bit late now, but ec2 modify-instance-attribute --instance-id INSTANCE_ID --ena-support might have helped you here).

So that would explain why your t3.xlarge wouldn't boot.

As for the items around mounting disks, for the first one you tried you say "connection wasn't possible", was the (old) volume in the same AZ as the (new) instance? It's amazing how easy it is to overlook that.

Then you could mount the old root volume as an additional disk on the new instance, but not the data volume which gave you bad superblock when you tried to xfs_repair it. Two things come to mind here, first is it actually an xfs filesystem, and second could it have been part of a logical volume group on the original host?

profile picture
EXPERT
Steve_M
answered 8 months ago
  • joker
    8 months ago

    Knowing the difference between the t2 and t3 generations, I tried attaching the previous volume to the t2 type instance again, but ssh access is still not possible. And the newly created volume and instance are in the same region. The root volume is ext4 and the data volume is xfs. In the case of the root volume, it is mounted, but ssh access is not available when creating an instance with the root volume.

  • joker
    8 months ago

    Also it could be a uuid problem https://repost.aws/en/knowledge-center/ebs-mount-volume-issues I tried the method in , but it still doesn't work.

  • Steve_M EXPERT
    8 months ago

    When we're back on the t2.micro and it's not booting, can you get the system log, Console -> EC2 -> select instance -> Action -> Monitor & troubleshoot -> Get system log

    And/or you might be able to Console -> EC2 -> select instance -> Action -> Monitor & troubleshoot -> Get instance screenshot depending on the region

    Is the data volume all on one partition on that disk, and is not a LV that spans multiple disks?

  • joker
    8 months ago

    Both the root volume and the data volume are composed of different ebs and different partitions respectively. In the case of system log, most of them are start and stop, and there is no case of instance screenshot.

    The root volume is mounted and there is no problem, but the biggest problem is that the xfs type data volume is not mounted.

  • Steve_M EXPERT
    8 months ago

    So did you change the UUID of the old root filesystem (like in the link in your previous comment)? If you did, but the /etc/fstab entry still has the old UUID then that may be why the instance won't boot from the old disk.

    If xfs_repair fails to fix the old data volume, what about xfs_check and xfs_info?

    And what is the output of file -s /dev/[data_volume_device]?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content