AWS Volume Juggling for Impaired Nodes and Node Health Check

Luke Hebert

July 28, 2021 - 3 minutes read - 445 words

When an EC2 instance fails its status checks, the quickest path to a fix often involves pulling the root volume off the impaired node and inspecting it from a healthy rescue instance. This walkthrough covers the full detach, mount, diagnose, and restore workflow so you can get a stuck instance back in service without spinning up a whole new environment.

The instance status checks monitor the health of the instance by sending an ARP request to the Network Interface Card (NIC) on the instance from the host, which means that a failure points to an internal issue with the instance itself [1].

First, try a simple reboot to see if the instance moves to a healthy underlying node. If that does not resolve the issue, stop the impaired instance, detach its EBS root volume, and attach it to another instance in the same Availability Zone to review the logs and determine what is causing the problem [2]. Below are the steps for attaching the EBS root volume to a rescue instance for review.

Using a Rescue Instance

Open the Amazon EC2 console.
Choose Instances from the navigation pane, then select the instance that is in emergency mode or otherwise impaired.
Stop the instance.
Detach the Amazon EBS root volume (/dev/xvda or /dev/sda1) from the stopped instance.
Launch a new EC2 instance in the same Availability Zone as the impaired instance. This new instance becomes your rescue instance (a t2.micro should be sufficient).
Attach the root volume you detached in Step 4 to the rescue instance as a secondary device.

Note

You can use different device names when attaching secondary volumes.

Connect to your rescue instance using SSH.
Create a mount point directory for the volume attached in Step 6. In the example below, the mount point is /mnt/rescue.
```
$ sudo mkdir /mnt/rescue
```
Mount the volume at the directory you created in Step 8.
```
$ sudo mount /dev/xvdf /mnt/rescue
```

Note

The device (/dev/xvdf in the example above) might be attached to the rescue instance under a different name. Run lsblk to view available disk devices and their mount points so you can confirm the correct device name.

With the volume mounted, review the logs and other files from the impaired instance under /mnt/rescue/. If you can identify and correct a configuration or service issue, apply the fix now to restore the instance to normal service.
Once the issue is resolved, unmount the volume.
```
$ sudo umount /mnt/rescue
```
Detach the volume from the rescue instance.
Attach the volume back to the original instance, then start the instance and confirm it boots successfully.

References

[1] AWS EC2 Instance Status Checks

[2] EC2 Linux Emergency Mode