Oracle ASM Disks Not Visible After Reboot - Diagnosis and Fix

-

Problem Overview

After a scheduled reboot of a RHEL 8 server running Oracle ASM, the Oracle instance failed to start. Investigation revealed that ASM disks were not visible under /dev/oracleasm, and oracleasm listdisks returned no results.

Initial Diagnosis

Symptoms Observed:

cd /dev/oracleasm
ls   # → No output
oracleasm status
# Checking if ASM is loaded: no  
# Checking if /dev/oracleasm is mounted: no
oracleasm scandisks
# Unable to instantiate disk "MIM_DATA_QA_ORC05_01"

oracleasm listdisks showed no known disks.

Deep-Level Checks

  • Confirmed disk devices present using lsblk (e.g., /dev/sde1, /dev/sdf1, etc.)
  • Kernel module oracleasm not loaded (lsmod | grep oracleasm returned nothing)

Root Cause Identified

The Oracle ASM kernel module (oracleasm) was not auto-loaded after reboot, preventing ASMLib from recognizing ASM disks.

Resolution Steps

  1. Reinstalled ASMLib kernel module: 
    [root@harq1orc30 yum.repos.d]# yum reinstall kmod-redhat-oracleasm-2.0.8-18.2.el8_10.x86_64
Updating Subscription Management repositories.
Rhel8-Baseos-virtual                                                                                                          23 MB/s |  89 MB     00:03
Rhel8-Appstream-virtual                                                                                                      7.4 MB/s |  72 MB     00:09
Rhel8-supplementary-virtual                                                                                                  350 kB/s | 396 kB     00:01
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                      68 kB/s | 2.9 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                         65 kB/s | 2.6 kB     00:00
Red Hat Satellite Tools 6.8 for RHEL 8 x86_64 (RPMs)                                                                          49 kB/s | 2.0 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - Supplementary (RPMs)                                                                  53 kB/s | 2.3 kB     00:00
Dependencies resolved.
=============================================================================================================================================================
 Package                                   Architecture               Version                                 Repository                                Size
=============================================================================================================================================================
Reinstalling:
 kmod-redhat-oracleasm                     x86_64                     8:2.0.8-18.2.el8_10                     Rhel8-Baseos-virtual                      46 k

Transaction Summary
=============================================================================================================================================================

Total download size: 46 k
Installed size: 143 k
Is this ok [y/N]: y
Downloading Packages:
kmod-redhat-oracleasm-2.0.8-18.2.el8_10.x86_64.rpm                                                                            72 kB/s |  46 kB     00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                         72 kB/s |  46 kB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                     1/1
  Reinstalling     : kmod-redhat-oracleasm-8:2.0.8-18.2.el8_10.x86_64                                                                                    1/2
  Running scriptlet: kmod-redhat-oracleasm-8:2.0.8-18.2.el8_10.x86_64                                                                                    1/2
  Running scriptlet: kmod-redhat-oracleasm-8:2.0.8-18.2.el8_10.x86_64                                                                                    2/2
  Cleanup          : kmod-redhat-oracleasm-8:2.0.8-18.2.el8_10.x86_64                                                                                    2/2
  Running scriptlet: kmod-redhat-oracleasm-8:2.0.8-18.2.el8_10.x86_64                                                                                    2/2
  Verifying        : kmod-redhat-oracleasm-8:2.0.8-18.2.el8_10.x86_64                                                                                    1/2
  Verifying        : kmod-redhat-oracleasm-8:2.0.8-18.2.el8_10.x86_64                                                                                    2/2
Installed products updated.

Reinstalled:
  kmod-redhat-oracleasm-8:2.0.8-18.2.el8_10.x86_64

Complete!
  2. Loaded the module: 
    modprobe oracleasm
  3. Verified module load: 
    [root@harq1orc30 yum.repos.d]# lsmod | grep asm
oracleasm              69632  0
  4. Initialized ASMLib: 
    [root@harq1orc30 yum.repos.d]# oracleasm init
Mounting ASMlib driver filesystem: /dev/oracleasm
  5. Scanned and listed ASM disks: 
    [root@harq1orc30 yum.repos.d]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "MIM_FRA_QA_ORC05_01"
Instantiating disk "MIM_DATA_QA_ORC05_01"
Instantiating disk "MIM_HAS_QA_ORC05_01"
Instantiating disk "MIM_DATA_QA_ORC05_02"
Instantiating disk "MIM_DATA_QA_ORC05_03"
Instantiating disk "MIM_DATA_QA_ORC05_04"
Instantiating disk "MIM_DATA_QA_ORC05_05"
Instantiating disk "MIM_CTLLOG2_QA_ORC05_01"
Instantiating disk "MIM_DATA_QA_ORC05_10"
Instantiating disk "MIM_CTLLOG1_QA_ORC05_01"
Instantiating disk "MIM_DATA_QA_ORC05_08"
Instantiating disk "MIM_DATA_QA_ORC05_07"
Instantiating disk "MIM_DATA_QA_ORC05_06"
Instantiating disk "MIM_DATA_QA_ORC05_09"

[root@harq1orc30 yum.repos.d]# oracleasm listdisks
MIM_CTLLOG1_QA_ORC05_01
MIM_CTLLOG2_QA_ORC05_01
MIM_DATA_QA_ORC05_01
MIM_DATA_QA_ORC05_02
MIM_DATA_QA_ORC05_03
MIM_DATA_QA_ORC05_04
MIM_DATA_QA_ORC05_05
MIM_DATA_QA_ORC05_06
MIM_DATA_QA_ORC05_07
MIM_DATA_QA_ORC05_08
MIM_DATA_QA_ORC05_09
MIM_DATA_QA_ORC05_10
MIM_FRA_QA_ORC05_01
MIM_HAS_QA_ORC05_01

[root@harq1orc30 yum.repos.d]# oracleasm querydisk /dev/sde1
Device "/dev/sde1" is marked an ASM disk with the label "MIM_FRA_QA_ORC05_01"

Final Summary

StepActionOutcome
Post-rebootASM disks missingOracle down, no /dev/oracleasm/*
Diagnosisoracleasm module not loadedConfirmed via lsmod
FixReinstalled and reloaded moduleoracleasm successfully loaded
RecoveryRe-scanned ASM disksDisks reappeared in list
ResultOracle ASM operationalServices restored without reboot

Lessons Learned

  • Verify both physical (block device) and logical (ASM) disk visibility
  • Use oracleasm init and scandisks after confirming device presence
  • ASMLib can be recovered without reboot by reloading the kernel module

Why Was Reinstallation Necessary?

The oracleasm kernel module likely failed to load after reboot due to one or more of the following:

  • Kernel upgrade: A new kernel was active post-reboot, but the oracleasm module was not built or linked for it.
  • Corruption or removal: The module may have been accidentally removed or damaged during patching.
  • Initramfs omission: oracleasm might not have been included in the initial RAM filesystem used at boot.
  • Systemd changes: Legacy init scripts were deprecated, and auto-load configuration wasn't updated properly for systemd environments.

Reinstalling the kmod-redhat-oracleasm package fixed the issue by:

  • Installing the correct driver version for the current kernel.
  • Ensuring the module was registered with modprobe and depmod.
  • Restoring the ability of ASMLib to recognize and manage ASM disks.

Recommendation: Always verify the compatibility of ASM modules after kernel updates and consider embedding them into initramfs using dracut.

Comments

Post a Comment

Popular Posts

Why SELinux Is Blocking Your Service (and How to Fix It)

-
Troubleshooting SELinux-Related Issues in Linux Running services with SELinux enabled can sometimes lead to unexpected denials or blocked actions—especially if SELinux was previously disabled or the service is configured in a non-standard way. In most cases, SELinux issues are the result of misconfigured contexts, policies, or port bindings. Here's a step-by-step approach to diagnose and fix SELinux-related problems while keeping your system secure. Check SELinux Status and Mode Start by verifying that SELinux is active on your system: # sestatus Look for: SELinux status: enabled Current mode: enforcing or permissive To test if SELinux is causing a problem, temporarily switch to permissive mode: # setenforce 0 # Logs denials but doesn’t block actions # setenforce 1 # Reverts to enforcing mode If the issue disappears in permissive mode, SELinux is likely the root cause. Look for SELinux Denials i...
Read more

Puppet Code Deploy Troubleshooting & Resolution Guide

-
Initial Problem I was setting up Puppet Enterprise and trying to push code using the trusted puppet code deploy command. But instead of a clean deployment, I got this surprise: puppet code deploy --dry-run And boom — this lovely error shows up: [POST /deploys][500] Errors while collecting a list of environments to deploy (exit code: 1). ERROR -> failed to stat '/var/opt/lib/pe-puppet/.gitconfig' So... what gives? Turns out this issue is part permissions, part SSH trust, and part “who ran this command and as which user.” But don’t worry — I’ll walk through all root causes and their real-world fixes step by step. Root Cause #1: Directory Permission What Happened: The directory /var/opt/lib was locked down with drwx------ (700) and owned by root . User pe-puppet couldn’t even step inside it. Since r10k runs as pe-puppet , it failed trying to read .gitconfig . The Fix: chmod 711 /var/opt/lib Why: This lets pe-puppet enter the folder ...
Read more

Fix: SSH Permission Denied Issue | Real Solution

-
1. Verify SSH Command and Key Usage Use the correct private key with -i : ssh -i ~/.ssh/your_private_key user@host Ensure the key is not passphrase-protected (or enter the passphrase correctly). 2. Check Key Generation and Deployment Regenerate Keys (if unsure): ssh-keygen -t ed25519 -C "your_email@example.com" # Prefer ed25519 Ed25519 uses elliptic curve cryptography (Edwards-curve Digital Signature Algorithm) and provides 128-bit security (equivalent to RSA-3072 or RSA-4096) with a 256-bit key. Copy the Public Key to the Server: ssh-copy-id -i ~/.ssh/your_public_key user@host If ssh-copy-id isn’t available, manually append the public key to ~/.ssh/authorized_keys on the server. 3. Fix File/Directory Permissions On the Server: chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys chmod go-w ~ # Home directory should not be world-writable On the Client (Local Machine): chmod 600 ~/.ssh/your_private_key 4. Check SSH Server Configuration Edit /etc/ssh/s...
Read more