Recovering a RAID1 disk from a corrupt array

I recently had our backup server at work die. Upon investigation the Kubuntu 8 server had had it’s OS drive and one of the RAID1 data disks corrupted to such an extent it couldn’t boot. The bad data disk was so bad that it couldn’t even be assigned a hardware address under Linux or Windows.

The only hope remained with the remaining RAID1 disk.

As the server had been set up by a previous administrator as a software RAID, I thought it would be a nice simple job of sticking the drive into a good computer and copying files off.

Alas no.

Windows had no luck at all. This was due to the drive being formatted in ext2, and there was no way I could get drivers for a ext2 drive to be read by Windows XP or 7.

The Linux mount command faired little better. This returned:

Error: Unknown filesystem type ‘linux_raid_member’

Essentially the problem was that we are trying to load only 1 disk from a RAID array and getting nowhere without the other RAID members. But we couldn’t use them because of corruption.

The way around this was to create an imaginary mirrored disk to the good data disk, mount the psuedo-array, and copy the files off. Here’s how I did it.

Open a console window in linux (I’m using Ubuntu 10.10) and attach the drive using a USB caddy (SATA or IDE cable is possible just be careful on the next step)

Check the drives log file to see the address of the last added drive

dmesg

It should say something like

[ 1531.730022] usb 1-6: new high speed USB device using ehci_hcd and address 7
[ 1531.882748] scsi7 : usb-storage 1-6:1.0
[ 1532.882157] scsi 7:0:0:0: Direct-Access     WDC WD25 00AAJB-00WGA0    2C01 PQ: 0 ANSI: 2 CCS
[ 1532.882739] sd 7:0:0:0: Attached scsi generic sg3 type 0
[ 1532.883565] sd 7:0:0:0: [sdd] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[ 1532.884502] sd 7:0:0:0: [sdd] Write Protect is off
[ 1532.884507] sd 7:0:0:0: [sdd] Mode Sense: 00 38 00 00
[ 1532.884510] sd 7:0:0:0: [sdd] Assuming drive cache: write through
[ 1532.886140] sd 7:0:0:0: [sdd] Assuming drive cache: write through
[ 1532.886147]  sdd: sdd1
[ 1532.895509] sd 7:0:0:0: [sdd] Assuming drive cache: write through
[ 1532.895514] sd 7:0:0:0: [sdd] Attached SCSI disk
[ 1533.081700] mdadm[2657]: segfault at 0 ip 0000000000415332 sp 00007fff56a6d7a0 error 4 in mdadm[400000+2a000]
[ 1578.911665] md: bind<sdd1>

Note the last line. Our drive is called sdd1 Next, get the multi-disk administer program (aka mdadm)

sudo apt-get install mdadm

Examine the current system for any RAID arrays. If nothing is found then it’s returned blank, as was the case for me

mdadm –examine –scan

Now we create a forced RAID1 from one real disk (and an imaginary one) (requires super user)

sudo mdadm –create –level=1 –force –raid-devices=1 /dev/md1 /dev/sdd1

Change the last word (in bold) for whatever your drive is called.

Mount the created RAID array

sudo mount /dev/md1 /mnt

Check status and file directory of mount

ls /mnt

Copy files to new drive (ie all of mounted array to other mounted disk under /media)

sudo cp -r /mnt /media/NewDisk/Recovered

Feel free to also use cp -r -v if you want to see that you are actually copying things. Otherwise you’ll just have a flashing cursor for a few hours.

Alternatively, you can now use your normal file browser and go to File System > mnt and copy and paste out in the normal way (Ctrl +C or right click then Copy).

Hope it helps!

What distinguishes you from other developers?

I've built data pipelines across 3 continents at petabyte scales, for over 15 years. But the data doesn't matter if we don't solve the human problems first - an AI solution that nobody uses is worthless.

Are the robots going to kill us all?

Not any time soon. At least not in the way that you've got imagined thanks to the Terminator movies. Sure somebody with a DARPA grant is always going to strap a knife/gun/flamethrower on the side of a robot - but just like in Dr.Who - right now, that robot will struggle to even get out of the room, let alone up some stairs.

But AI is going to steal my job, right?

A year ago, the whole world was convinced that AI was going to steal their job. Now, the reality is that most people are thinking 'I wish this POC at work would go a bit faster to scan these PDFs'.

When am I going to get my self-driving car?

Humans are complicated. If we invented driving today - there's NO WAY IN HELL we'd let humans do it. They get distracted. They text their friends. They drink. They make mistakes. But the reality is, all of our streets, cities (and even legal systems) have been built around these limitations. It would be surprisingly easy to build self-driving cars if there were no humans on the road. But today no one wants to take liability. If a self-driving company kills someone, who's responsible? The manufacturer? The insurance company? The software developer?