mdadm RAID array with failed drive set to inactive after boot

Written by Claudio Kuenzler - 0 comments

Published on September 21st 2023 - Listed in Linux Hardware

---

A physical drive (a Western Digital Green SSD) died in a physical machine running Debian Linux. Not even BIOS was able to detect the drive anymore. No worries, I thought, this is a RAID-1 array and data is not lost.

However after a reboot of the machine I was surprised to find the raid array in question (md3) to be inactive:

root@debian ~ # cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md3 : inactive sdb1[2](S)
      976630488 blocks super 1.2

md2 : active raid1 sda3[3] sdc3[2]
      927803096 blocks super 1.2 [2/2] [UU]
      bitmap: 3/4 pages [12KB], 131072KB chunk

md1 : active raid1 sda2[3] sdc2[2]
      24397824 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[3] sdc1[2]
      24396800 blocks super 1.2 [2/2] [UU]

unused devices: <none>

Although the md3 array shows the remaining raid drive (sdb1), the whole array is set to inactive. Let's take a look at details:

root@debian ~ # mdadm --detail /dev/md3
/dev/md3:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 1
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 1

              Name : debian:3  (local to host debian)
              UUID : b02f1c28:8bebd054:13c57e21:2fb06992
            Events : 27114

    Number   Major   Minor   RaidDevice

       -       8       17        -        /dev/sdb1

mdadm is able to detect the superblock. That's a good sign.

Activate raid array

Let's activate this raid array:

root@debian ~ # mdadm --run /dev/md3
mdadm: started array /dev/md/3

root@debian ~ # cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md3 : active (auto-read-only) raid1 sdb1[2]
      234298968 blocks super 1.2 [2/1] [U_]
      bitmap: 0/2 pages [0KB], 65536KB chunk

md2 : active raid1 sda3[3] sdc3[2]
      927803096 blocks super 1.2 [2/2] [UU]
      bitmap: 3/4 pages [12KB], 131072KB chunk

md1 : active raid1 sda2[3] sdc2[2]
      24397824 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[3] sdc1[2]
      24396800 blocks super 1.2 [2/2] [UU]

unused devices: <none>

The array is now listed as active, but set to auto-read-only. At least mdadm now correctly detected a drive failure or missing drive in the array ([U_]).

Let's make the array operative (read-write) again:

root@debian ~ # mdadm --readwrite /dev/md3

root@debian ~ # cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md3 : active raid1 sdb1[2]
      234298968 blocks super 1.2 [2/1] [U_]
      bitmap: 0/2 pages [0KB], 65536KB chunk

md2 : active raid1 sda3[3] sdc3[2]
      927803096 blocks super 1.2 [2/2] [UU]
      bitmap: 3/4 pages [12KB], 131072KB chunk

md1 : active raid1 sda2[3] sdc2[2]
      24397824 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[3] sdc1[2]
      24396800 blocks super 1.2 [2/2] [UU]

unused devices: <none>

The array is now fully working again - at least with one drive. The next step is obviously to replace the failed drive in the software raid using mdadm.

Why was the array set to inactive at boot time?

Of course that was my very first question to myself. It turns out that I forgot to add the md3 array to /etc/mdadm/mdadm.conf when I created the md3 array a while ago. This file is read by mdadm at boot time and activates the listed raid arrays automatically.

To update the conf file with the currently active raid arrays, do:

root@debian ~ # mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Then manually edit /etc/mdadm/mdadm.conf with your favourite editor and make sure there are no duplicates. In my case this results in:

root@debian ~ # cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/0  metadata=1.2 UUID=607be06b:83859600:ad9c1374:7da35f4a name=debian:0
ARRAY /dev/md/1  metadata=1.2 UUID=5eb2b49f:00bfefbf:6940df99:cc963ac2 name=debian:1
ARRAY /dev/md/2  metadata=1.2 UUID=e275a9ab:78beea51:ca120a41:232d0b58 name=debian:2
ARRAY /dev/md/3  metadata=1.2 UUID=b02f1c28:8bebd054:13c57e21:2fb06992 name=debian:3

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.