Ilmar Kerm

Oracle, databases, Linux and maybe more

Before storage device can be removed (or unassigned) from server, it must first be removed from Linux. This is even more necessary if the server is running Oracle ASMLib, because if there are any removed or read-only storage devices present when running service oracleasm scandisks, the command will just hang and server load will start to increase (maximum I’ve seen was 3500).

Here you find RedHat instructions on how to cleanly remove storage devices in RHEL6. A lot of steps, especially when running multipath and having multiple paths to each device.

Since we do database replication and cloning to test using storage replication, then assigning and unassigning storage devices from test servers is quite frequent and if not done properly and cleanly, we have to reboot the server. Here is a little script to help with that task. It takes ASMLib disk names to be removed as an argument, finds all paths to the device and removes them from Linux.

Tested in OEL5 (using RHEL kernel) and OEL6 (using UEK2 kernel). This script only deals with dm-* and sd* devices, so before using please check and modify the script for your environment. Before removing the storage devices from Linux, make sure that the disk groups are dismounted from ASM and they are not used.

#!/bin/bash -l

# This script removes all underlying block devices of the specified ASMLIB disks from linux

if [ $# -eq 0 ]; then
  echo "No command line arguments found"
  echo "List all the ASMLIB disks to be removed as command line arguments"
  exit 1
fi

# Recursively go into /sys/block/devicename/slaves to get all the underlying disk devices of the multipath devices
scan_dev_paths() {
  for f in $(ls -1 "/sys/block/$1/slaves" ); do
    if [ ${f:0:2} == "dm" ]; then
      # In case the main dm- device was a partition need to go recursively down
      scan_dev_paths "$f"
    elif [ ${f:0:2} == "sd" ]; then
      devices=("${devices[@]}" "$f")
    fi
  done
}

cd /dev/oracleasm/disks
for disk in "$@"; do
  # Check if the requested block device exists
  if [ -b "$disk" ]; then
    declare -a devices
    # Take the device minor number
    # Using /dev/dm-* because /sys/dev does not exist in OEL5 using RHEL kernel
    minorhex=$(stat -c %T "$disk")
    minor=$(printf '%d' "0x${minorhex}")
    blockdev="dm-$minor"
    # Check if the underlying dm- device exists
    if [ -b "/dev/$blockdev" ]; then
      # Just to verify query the detected block devices
      asmlibmark=$(/sbin/service oracleasm querydisk "/dev/dm-$minor")
      echo "$asmlibmark"
      if [[ "$asmlibmark" == *"is marked an ASM disk"* ]]; then
        scan_dev_paths "$blockdev"
      else
        echo "Block device /dev/$blockdev does not seem to be ASMLIB disk, you shouldn't see this message, so exiting"
        exit 1
      fi
    else
      echo "Block device /dev/$blockdev not found, you shouldn't see this message, so exiting"
      exit 1
    fi
    # What devices did we find?
    echo "Devices found: ${devices[@]}"
    for d in "${devices[@]}"; do
      echo "Removing $d"
      /sbin/blockdev --flushbufs "/dev/$d"
      echo 1 > "/sys/block/$d/device/delete"
    done
    unset devices
  else
    echo "ASMLIB disk $disk not found"
  fi
done

# Execute multipath to get rid of the multipath devices and rescan asmlib
/sbin/multipath
/sbin/service oracleasm scandisks
/sbin/service oracleasm listdisks

Categories