Replacing a Linux RAID Drive

NAS drives

I have been running a software RAID array at home for some time now. It’s a single network storage where I consolidate all my files. I manage this array manually using the mdadm command. Some people choose to buy a NAS storage box which hides all of the implementation details behind a nice Web GUI, but it’s essentially the same thing under the hood.

It operates with 4 drives using Linux software RAID 5, which means it can tolerate a single drive failure, but failures don’t always take out an entire drive. They usually manifest as bad sectors in a drive. As an illustration, the RAID 5 array below can still operate properly (meaning no data loss, yet) with bad sectors on two of its drives:

RAID 5 array with damaged blocks

As long as the other drives in the array doesn’t develop bad sectors in the same stripe, the data can still be reconstructed from the remaining good blocks. This means that you can somewhat leave the drive as it is for a period without replacement, but of course you are taking a risk.

I thought I’d share my experiences with drive replacements thus far.

Detecting Drive Problems

Most Linux distributions provide the raid-check script for periodic RAID scrubbing. This is basically a background cron job that tells the kernel to start checking the RAID array. For RHEL/CentOS systems, this should occur every weekend.

During this scrubbing process, all drives within the array are read and their parity blocks are computed, to ensure that everything tallies.

It is during this verification process that sometimes causes hard drive errors to show up. Typically when a drive encounters a problem during read, the hardware returns an error, which will then be logged by Linux. They can look like these:

ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata3.00: irq_stat 0x40000001
ata3.00: failed command: READ DMA EXT
ata3.00: cmd 25/00:00:d8:10:27/00:02:05:00:00/e0 tag 8 dma 262144 in
         res 51/40:1f:b8:12:27/00:00:05:00:00/e0 Emask 0x9 (media error)
ata3.00: status: { DRDY ERR }
ata3.00: error: { UNC }
ata3.00: configured for UDMA/133
ata3: EH complete
 .
 . (repeats)
 .
sd 2:0:0:0: [sdc]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 2:0:0:0: [sdc]  Sense Key : Medium Error [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
        72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
        05 27 12 b8
sd 2:0:0:0: [sdc]  Add. Sense: Unrecovered read error - auto reallocate failed
sd 2:0:0:0: [sdc] CDB: Read(10): 28 00 05 27 10 d8 00 02 00 00
end_request: I/O error, dev sdc, sector 86446776

Continue reading

Netbooting Your Raspberry Pi

A very long time ago, I set up and played around with diskless machines. These are basically PCs can boot up an operating system fully without hard disks. All the operating system files come from a server on the network. It was amazing (well, to me at least)!

Back then, Ethernet cards used to have a DIP/PLCC socket, which allowed you to insert an EEPROM on which you burn a boot ROM. Fortunately I didn’t have to do any of that because the network cards at that time already came with PXE ROMs built-in, just as they do today. To activate this, you just need to select the network card’s option ROM in the boot order, or make it higher up in the boot priority.

3Com network card with boot ROM socket marked

As part of the boot process, the network card will request an address from the DHCP server, which also tells the client where it can find the TFTP server with the next boot stage. The ROM will download this file from the TFTP server and start executing it.

That’s how Linux ultimately gets started from the network.

An announcement was made recently on the Raspberry Pi blog that you can achieve total network boot, just like on the PC, without any SD cards.

Continue reading

Bruteforcing LUKS Volumes Explained

Some weeks back, we were forced to reboot one of our server machines because it stopped responding. When the machine came back up, we were greeted with a password prompt to decrypt the partition. No problem, since we always used a password combination (ok, permutation) that consisted of a few words, something along the lines of “john”, “doe”, “1954”, and the server’s serial number. Except that it didn’t work, and we forgot the permutation rules AND whether we used “john” “doe” or “jack” “daniels”.

All the search results for bruteforcing LUKS are largely the same — “use cryptsetup luksOpen --test-passphrase“. In my case, the physical server is in the server room, and I don’t want to stand in front of the rack trying to figure all this out. My question is, can I do this offline on another machine? None of those blog entries were helpful in this regard.

The LUKS Header

To answer this question, I took a look at the LUKS header. This header is what provides multiple “key slots”, allowing you to specify up to 8 passwords or key files that can decrypt the volume. cryptsetup is the standard userspace tool (and library) to manipulate and mount LUKS volumes. Since LUKS was designed based on TKS1, the TKS1 document referenced by the cryptsetup project was very helpful. After consulting the documentation & code, I came up with the following diagram that describes the LUKS key verification process:

LUKS encryption flowchart

Continue reading

Creating Minimal Throw-away CentOS 6 VMs

Whether you are using CentOS for a build server or simply testing out a new configuration, you can quickly create a VM (virtual machine) that is under 1GB. You can do this without downloading any special tools or ISO files — just the CentOS installation DVD and VirtualBox (or VMware if you prefer).

I like the text-based console, so you won’t be getting a GUI or fancy Linux desktop with this one. Given its small size, you could also archive the entire environment (or even several of them) for future use without having to waste gigabytes of free space. These environments also serve as a base which can be upgraded or added onto to provide more functionality later.

Continue reading

Encrypt All the Drives

I have always been an advocate on storage security (all types of security, actually). I like how iOS devices keep all files encrypted, even if you do not set a passcode on the device. They do this to facilitate quick erasure of files on the device — to erase all the data, they simply wipe the master key.

Erasing magnetic storage media isn’t difficult, but it is time-consuming. For solid state media such as SSDs and flash drives, the wear-leveling makes it difficult to ensure that all flash blocks have been securely overwritten. The answer to this is to encrypt everything.

Encrypt all the drives!! (meme)

Recently I have been busy building a Linux-based NAS and I decided to put this to practice.

Continue reading