By using AWS re:Post, you agree to the Terms of Use

Questions tagged with Amazon Elastic Block Store

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

EC2 - Could not set DHCPv4 address: Connection timed out (sa-east-1a)

Our c6i.2xlarge 3-year reserved instance, running for its first 5 days, generated this log entry **_Could not set DHCPv4 address: Connection timed out_** on Jan 28 02:59:51 UTC, followed by **_Failed_** and **_Configured_**. From there on, the machine became unresponsive and AWS finally raised a StatusCheckFailed_Instance at 06:59 UTC. At 09:06 UTC machine was stopped and restarted through the Console. I found these apparently related issues, but still clueless: [CoreOS goes offline on DHCP failure on Amazon VPC](https://github.com/coreos/bugs/issues/2020) [CoreOS on EC2 losing network connection once a day](https://github.com/coreos/bugs/issues/1551) The box is running MySQL 5.7.36 and Memcache 1.5.6 on top of Ubuntu 18.04. I would be thankful if someone could help me identify the **root cause** of this issue, and: 1. Could this be related to ntp-systemd-netif.service ? 2. This instance type has a separate channel for EBS, but with network down, and no customers making requests (no usage logs on the application machine, except the "MySQL connection timeouts"), what would explain a surge on EBS disk reads? CloudWatch graphs below. 3. We have an EFS disk attached to this instance, that started failing at 04:04 UTC _probably_ related to network failure. No errors reported at EFS sa-east São Paulo status page. Jan 28 02:17:01 ip-172-xxx-xxx-xxx CRON[18179]: pam_unix(cron:session): session opened for user root by (uid=0) Jan 28 02:17:01 ip-172-xxx-xxx-xxx CRON[18180]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 28 02:17:01 ip-172-xxx-xxx-xxx CRON[18179]: pam_unix(cron:session): session closed for user root Jan 28 02:29:11 ip-172-xxx-xxx-xxx systemd-networkd[728]: ens5: Configured Jan 28 02:29:11 ip-172-xxx-xxx-xxx systemd-timesyncd[623]: Network configuration changed, trying to establish connection. Jan 28 02:29:12 ip-172-xxx-xxx-xxx systemd-timesyncd[623]: Synchronized to time server 169.254.169.123:123 (169.254.169.123). Jan 28 02:29:12 ip-172-xxx-xxx-xxx systemd[1]: Started ntp-systemd-netif.service. Jan 28 02:59:51 ip-172-xxx-xxx-xxx systemd-timesyncd[623]: Network configuration changed, trying to establish connection. Jan 28 02:59:51 ip-172-xxx-xxx-xxx systemd-networkd[728]: **ens5: Could not set DHCPv4 address: Connection timed out** Jan 28 02:59:51 ip-172-xxx-xxx-xxx systemd-networkd[728]: **ens5: Failed** Jan 28 02:59:51 ip-172-xxx-xxx-xxx systemd-networkd[728]: **ens5: Configured** Jan 28 02:59:51 ip-172-xxx-xxx-xxx systemd-timesyncd[623]: Synchronized to time server 169.254.169.123:123 (169.254.169.123). Jan 28 02:59:51 ip-172-xxx-xxx-xxx systemd-timesyncd[623]: Network configuration changed, trying to establish connection. Jan 28 02:59:51 ip-172-xxx-xxx-xxx systemd-timesyncd[623]: Synchronized to time server 169.254.169.123:123 (169.254.169.123). Jan 28 03:00:01 ip-172-xxx-xxx-xxx systemd[1]: **Started ntp-systemd-netif.service.** Jan 28 03:01:21 ip-172-xxx-xxx-xxx systemd-udevd[503]: seq 16407 '/kernel/slab/proc_inode_cache/cgroup/proc_inode_cache(4935:ntp-systemd-netif.service)' is taking a long time Jan 28 03:01:28 ip-172-xxx-xxx-xxx systemd-udevd[503]: seq 16408 '/kernel/slab/:A-0000040/cgroup/pde_opener(4935:ntp-systemd-netif.service)' is taking a long time Jan 28 03:01:34 ip-172-xxx-xxx-xxx systemd-udevd[503]: seq 16409 '/kernel/slab/kmalloc-32/cgroup/kmalloc-32(4935:ntp-systemd-netif.service)' is taking a long time Jan 28 03:01:40 ip-172-xxx-xxx-xxx systemd-udevd[503]: seq 16410 '/kernel/slab/kmalloc-4k/cgroup/kmalloc-4k(4935:ntp-systemd-netif.service)' is taking a long time Jan 28 03:17:03 ip-172-xxx-xxx-xxx CRON[18284]: pam_unix(cron:session): session opened for user root by (uid=0) Jan 28 03:17:12 ip-172-xxx-xxx-xxx CRON[18285]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 28 03:19:34 ip-172-xxx-xxx-xxx snapd[6419]: autorefresh.go:530: Cannot prepare auto-refresh change: Post https://api.snapcraft.io/v2/snaps/refresh: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jan 28 03:19:34 ip-172-xxx-xxx-xxx CRON[18284]: pam_unix(cron:session): session closed for user root Jan 28 03:28:44 ip-172-xxx-xxx-xxx snapd[6419]: stateengine.go:149: state ensure error: Post https://api.snapcraft.io/v2/snaps/refresh: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Jan 28 03:36:35 ip-172-xxx-xxx-xxx systemd[1]: Starting Ubuntu Advantage Timer for running repeated jobs... Jan 28 04:01:18 ip-172-xxx-xxx-xxx systemd[1]: **Started ntp-systemd-netif.service.** Jan 28 04:03:09 ip-172-xxx-xxx-xxx systemd-udevd[503]: seq 16496 '/radix_tree_node(4961:ntp-systemd-netif.service)' is taking a long time Jan 28 04:04:00 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:06:13 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:06:26 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:09:14 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:09:26 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:12:15 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:12:26 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:12:36 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:15:15 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:15:26 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:15:34 ip-172-xxx-xxx-xxx kernel: nfs: server fs-0ac698ea1xxxxxxxx.efs.sa-east-1.amazonaws.com not responding, timed out Jan 28 04:16:39 ip-172-xxx-xxx-xxx sshd[4657]: pam_unix(sshd:session): session closed for user ubuntu Jan 28 04:17:30 ip-172-xxx-xxx-xxx systemd-logind[974]: Failed to abandon session scope, ignoring: Connection timed out Jan 28 04:18:00 ip-172-xxx-xxx-xxx systemd-logind[974]: Removed session 27. [Cloud Watch Graphs](https://ibb.co/7tydsyQ) Thanks!
0
answers
0
votes
114
views
asked 8 months ago

EC2 instance fails to start after changing instance type from t2 to t3

I tried to update an EC2 instance from t2 to t3. Since the AZ I was running the instance in did not support t3 instances. I stopped the instance, created an image, and then tried to create an instance from that image in us-east-1c. The instance is running RockyOS v. 8.5. The instance did not start. Using the serial console, it appears as though the EBS volume was not detected. I verified that the ENA and NMVE drivers were installed. I tried a series of experiments, where I created new instances from the original AMI and I was able to create t2 instances, stopped them, created images, and then created new t3 instances without issue. The main difference of course is that the production instance has a lot more data on it, has been updated via dnf update, etc. I suppose I could just create a brand new t3 instance and migrate the data over, but I would like to understand why I wasn't able to convert the instance from t2 to t3. Some more information: The reason that the experiments worked, was because the original AMI was based on RockyOS v. 8.4. This version allows me to migrate between t2 and t3 versions without any issues. The production instance was updated at some point to version 8.5 and for some reason this version does not boot on t3 (nitro) instance types. I repeated my experiment, and launched the original AMI in a t2, did an upgrade, then after changing the instance type to t3, the instance does not boot. While this doesn't provide a solution to the problem, at least not it is reproducible. So, what is it about Rocky OS v. 8.5 that prevents the migration to a nitro instance? The modinfo ena and modinfo nvme both show the drivers are present
2
answers
0
votes
175
views
asked 9 months ago

EBS Volumes on RAID0 to gain perfomance

Hi, I have to setup a Domino Server, that uses a "translog" path. On the Domino Server documentation, it is suggested to dedicate a specific disk with his own controller to this path directory. I am not sure what method to use, either to use a normal GP3 disk, and add more IOPs if I see they are needed, or using ESB (algo GP3) on Raid0. Problem that I see with Raid0 is that I do not known how easy is to increase volumes on a RAID0 Configuration. On a normal (single) ESB volume, starting with a "small" size of the volume (ie, 100Gb), and then, increase it, is easy, is a trivial thing, and I can add also more IOPs / trhoughput on a very easy way, so I guess I can get the same (or very similar) perfomance that I will get with RAID0. I am aware that will RAID0 I will be able to "double" the perfomance, because I will get accumulatted IOPs values, so the maximum of IOPs obtained on RAID0 will be always greater, but not sure if I will need to increase the IOPS to a ratio that can not be obtained also on a single GP3 Disk. Moreover, my concern on RAID0 is about how easy will be to increase volume on it.. Can I increase a ESB volume that is being part of a RAID0 the same way that a normal EBS volume ? Do I need to have both ESB volumes at the same size ? Is the administrative additional task on a RAID0 woth the less (ie, complexity added for increasing volumes, for snapshoots, backups/recoverys, etc) ? In summary... When is bettert to have a single EBS volume with the number of IOPS you need, or a RAID0 config ?
2
answers
0
votes
82
views
asked 9 months ago