Fix EC2 NVMe Mount Issues After Reboot Using UUIDs (AWS EBS Guide)

-

Fix EC2 NVMe Mount Mismatch After Reboot — The UUID Way

When working with multiple EBS volumes on an EC2 instance—especially NVMe-backed ones—you might notice that after a reboot, your mount points suddenly don't match the volumes you expected.

This guide explains the root cause, shows how to fix it using UUIDs, and shares some patterns and pro tips around volume device names.

The Problem

After rebooting, your mount points might look correct on the surface but are actually pointing to the wrong disks.

Before reboot:

/dev/nvme1n1 → 300G → /opt/puppetlabs
/dev/nvme2n1 → 50G  → /var/log/puppetlabs
/dev/nvme3n1 → 30G  → /etc/puppetlabs

After reboot:

/dev/nvme1n1 → 30G  → /opt/puppetlabs ❌
/dev/nvme2n1 → 50G  → /var/log/puppetlabs ✅
/dev/nvme3n1 → 300G → /etc/puppetlabs ❌

This can cause services to fail or logs to be written in the wrong location.

Root Cause: NVMe Device Name Reordering

Device names like /dev/nvme1n1 are assigned dynamically by the Linux kernel at boot time. They depend on:

  • Volume size (smaller volumes may initialize faster)
  • Attachment order (not guaranteed by EC2)
  • IO path timing

So what was /dev/nvme1n1 yesterday might be /dev/nvme3n1 after a reboot. Spooky, right?

Observation

Many sysadmins (like me!) have noticed this pattern:

/dev/nvme1n1   → 30G
/dev/nvme2n1   → 50G
/dev/nvme3n1   → 300G

Smaller volumes often get lower nvmeXn1 numbers. But don't rely on it — it's not guaranteed.

The Permanent Fix: Use UUIDs in /etc/fstab

Step 1: List Your Volumes and UUIDs

lsblk -f

Sample output:

nvme1n1  xfs  0749b056...  /opt/puppetlabs  # 30G
nvme2n1  xfs  2a545650...  /var/log/puppetlabs  # 50G
nvme3n1  xfs  6d6bbea5...  /etc/puppetlabs  # 300G

Step 2: Identify What Each Volume Contains

mkdir /mnt/test1 /mnt/test2 /mnt/test3
mount /dev/nvme1n1 /mnt/test1
mount /dev/nvme2n1 /mnt/test2
mount /dev/nvme3n1 /mnt/test3

ls /mnt/test1
ls /mnt/test2
ls /mnt/test3

Step 3: Update /etc/fstab

Use UUIDs instead of /dev/nvmeXn1.

UUID=6d6bbea5-ff46-47f9-8ae2-b0281579e999   /opt/puppetlabs     xfs   defaults,nofail   0 2
UUID=2a545650-9df2-4811-b400-5f9bc25f5467   /var/log/puppetlabs xfs   defaults,nofail   0 2
UUID=0749b056-7c18-4fea-88ba-b7196a5dafcf   /etc/puppetlabs     xfs   defaults,nofail   0 2

Step 4: Apply Changes

umount /opt/puppetlabs
umount /etc/puppetlabs
mount -a

Verify everything is correct:

df -hT

Then reboot:

sudo reboot

Bonus: Identify EBS Volumes from Linux

Use the following command to match NVMe devices with EBS volume IDs:

sudo nvme list

This helps map /dev/nvme1n1 to vol-xxxxxxxx in your AWS console.

Pro Tips

  • Always use UUIDs or LABELs for mounting volumes — never /dev/nvmeXn1
  • Backup your /etc/fstab before editing:
cp /etc/fstab /etc/fstab.backup
  • Use xfs_admin -L to label volumes with readable names:
sudo xfs_admin -L optdata /dev/nvmeXn1

Conclusion

Dynamic device naming is one of those silent little issues in cloud infrastructure that can cause major headaches. But by switching to UUID-based mounts, you ensure your volumes are mounted correctly every time — no matter how the system orders them.

Keep calm and UUID on 💚

Comments

Post a Comment

Popular Posts

Puppet Code Deploy Troubleshooting & Resolution Guide

-
Initial Problem I was setting up Puppet Enterprise and trying to push code using the trusted puppet code deploy command. But instead of a clean deployment, I got this surprise: puppet code deploy --dry-run And boom — this lovely error shows up: [POST /deploys][500] Errors while collecting a list of environments to deploy (exit code: 1). ERROR -> failed to stat '/var/opt/lib/pe-puppet/.gitconfig' So... what gives? Turns out this issue is part permissions, part SSH trust, and part “who ran this command and as which user.” But don’t worry — I’ll walk through all root causes and their real-world fixes step by step. Root Cause #1: Directory Permission What Happened: The directory /var/opt/lib was locked down with drwx------ (700) and owned by root . User pe-puppet couldn’t even step inside it. Since r10k runs as pe-puppet , it failed trying to read .gitconfig . The Fix: chmod 711 /var/opt/lib Why: This lets pe-puppet enter the folder ...
Read more

Fix: SSH Permission Denied Issue | Real Solution

-
1. Verify SSH Command and Key Usage Use the correct private key with -i : ssh -i ~/.ssh/your_private_key user@host Ensure the key is not passphrase-protected (or enter the passphrase correctly). 2. Check Key Generation and Deployment Regenerate Keys (if unsure): ssh-keygen -t ed25519 -C "your_email@example.com" # Prefer ed25519 Ed25519 uses elliptic curve cryptography (Edwards-curve Digital Signature Algorithm) and provides 128-bit security (equivalent to RSA-3072 or RSA-4096) with a 256-bit key. Copy the Public Key to the Server: ssh-copy-id -i ~/.ssh/your_public_key user@host If ssh-copy-id isn’t available, manually append the public key to ~/.ssh/authorized_keys on the server. 3. Fix File/Directory Permissions On the Server: chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys chmod go-w ~ # Home directory should not be world-writable On the Client (Local Machine): chmod 600 ~/.ssh/your_private_key 4. Check SSH Server Configuration Edit /etc/ssh/s...
Read more

Linux Process Termination Signals Explained with Examples

-
Linux uses signals to communicate with processes, allowing them to terminate, pause, or handle specific events. Below are key termination signals, examples of how to use them, and their effects. 1. SIGINT (2) : Interrupt Signal Default Action : Terminate the process. Trigger : Press Ctrl+C in the terminal. Use Case : Gracefully stop a running command (e.g., a script or program). Example: # Start a long-running process sleep 100 # Press Ctrl+C to send SIGINT and terminate it Handling SIGINT in a Bash Script: #!/bin/bash trap 'echo "SIGINT caught! Exiting..."; exit' SIGINT echo "Running... Press Ctrl+C to test SIGINT" while true; do sleep 1 done Output: Running... Press Ctrl+C to test SIGINT ^CSIGINT caught! Exiting... 2. SIGQUIT (3) : Quit Signal Default Action : Terminate and generate a core dump. Trigger : Press Ctrl+\ in the terminal. Use Case : Debugging (generates a core dump for post-mortem analysis). ...
Read more