Replacing a Linux RAID Drive

NAS drives

I have been running a software RAID array at home for some time now. It’s a single network storage where I consolidate all my files. I manage this array manually using the mdadm command. Some people choose to buy a NAS storage box which hides all of the implementation details behind a nice Web GUI, but it’s essentially the same thing under the hood.

It operates with 4 drives using Linux software RAID 5, which means it can tolerate a single drive failure, but failures don’t always take out an entire drive. They usually manifest as bad sectors in a drive. As an illustration, the RAID 5 array below can still operate properly (meaning no data loss, yet) with bad sectors on two of its drives:

RAID 5 array with damaged blocks

As long as the other drives in the array doesn’t develop bad sectors in the same stripe, the data can still be reconstructed from the remaining good blocks. This means that you can somewhat leave the drive as it is for a period without replacement, but of course you are taking a risk.

I thought I’d share my experiences with drive replacements thus far.

Detecting Drive Problems

Most Linux distributions provide the raid-check script for periodic RAID scrubbing. This is basically a background cron job that tells the kernel to start checking the RAID array. For RHEL/CentOS systems, this should occur every weekend.

During this scrubbing process, all drives within the array are read and their parity blocks are computed, to ensure that everything tallies.

It is during this verification process that sometimes causes hard drive errors to show up. Typically when a drive encounters a problem during read, the hardware returns an error, which will then be logged by Linux. They can look like these:

ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata3.00: irq_stat 0x40000001
ata3.00: failed command: READ DMA EXT
ata3.00: cmd 25/00:00:d8:10:27/00:02:05:00:00/e0 tag 8 dma 262144 in
         res 51/40:1f:b8:12:27/00:00:05:00:00/e0 Emask 0x9 (media error)
ata3.00: status: { DRDY ERR }
ata3.00: error: { UNC }
ata3.00: configured for UDMA/133
ata3: EH complete
 .
 . (repeats)
 .
sd 2:0:0:0: [sdc]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 2:0:0:0: [sdc]  Sense Key : Medium Error [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
        72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
        05 27 12 b8
sd 2:0:0:0: [sdc]  Add. Sense: Unrecovered read error - auto reallocate failed
sd 2:0:0:0: [sdc] CDB: Read(10): 28 00 05 27 10 d8 00 02 00 00
end_request: I/O error, dev sdc, sector 86446776

Continue reading

Raspberry Pi Zero Wireless

On the 5th birthday of the Raspberry Pi last week, the foundation announced a new addition to the family — the Raspberry Pi Zero W. The W stands for Wireless.

I got my hands on one, from the fine folks at Pimoroni. (And no they didn’t pay me to say this.)

The Pi Zero W board

It has the same specs as the Raspberry Pi Zero, namely the 1GHz single-core CPU and 512 MB of RAM. It still has the two micro USB port — one for power and another for OTG, which means you can get it to behave like USB devices when plugged into a PC. The big difference is that they have added WiFi and Bluetooth capability to this small board by squeezing some space out from between the processor and the power circuitry. The size of the board and the placement of connectors remain the same, even the test points on the back.

I’m excited for anything that has processing power, HDMI connectivity and WiFi.

WiFi + Bluetooth

BCM43438 wireless chipset

The 802.11n WiFi and Bluetooth 4.1 functionality comes from the Broadcom BCM43438 (now known as the Cypress CYW43438). This is the same chipset that was used in the Pi 3. The wireless chipset connects via SDIO, so your network traffic does not have to contend for the USB bus bandwidth.

Continue reading

Cracking iTunes Backup Passwords with hashcat

Following the recent announcement of LUKS support in hashcat, I noticed that there have been some commits to support iTunes Backup passwords as well.

This is only useful if the backup was encrypted by setting a backup password on the iOS device. If the backup is not encrypted then all the files are in clear and there is nothing to bruteforce.

The keys used to encrypt the backup are stored in the BackupKeyBag, which can be found in the Manifest.plist file. This keybag is a binary blob, the format of which has already been documented by researchers from Sogeti ESEC Lab.

I have written a simplified script which dumps the BackupKeyBag.
You will need the Python bindings from libplist for the script to work. If you cannot get it to work, you can try the Perl script from philsmd instead.

Speeding up iOS Backups

iOS device backups usually take a while, depending on how much storage has been used on your device.

The iOS backup process is driven by the device itself, through the BackupAgent process. This process treats the host PC like a dumb disk store, by sending it commands like DLMessageCreateDirectory, DLMessageUploadFiles, DLMessageRemoveFiles, DLMessageGetFreeDiskSpace, etc. so that it can determine what has been backed up previously and what to send/update for incremental backups.

For password cracking, we don’t need the entire 64 GB (or God forbid, 128 GB) of data on the iOS device. We just need the Manifest.plist, which is typically less than 50 KB. But because the backup process is controlled by the device and not the PC, we can’t simply ask it to send over that single file. Sometimes when we setup a VM with libimobiledevice, we might also not have allocated such a large virtual disk. Of course when I say “we”, I really mean “I”.

Continue reading

Seeedstudio Fusion PCB Review

Fusion PCB is a PCB service from Seeedstudio. They have been offering PCB prototyping service since I made my first board in 2011. It has recently been revamped a little, tweaking prices and options, as well as integrating an online Gerber viewer from EasyEDA. I was invited to give Seeedstudio’s revamped Fusion PCB service a try, and since I had some boards in the pipeline for manufacture, I thought why not?

You can configure various options for the PCB, such as board thickness, copper pour and surface finish. You can also make flex PCBs or aluminium for better heat sinking, as opposed to regular FR4. These options will of course come at a price. However, you can select various colours for your PCB at no additional cost.

The Boards

I ordered 2 sets of boards in total. I’ve decided to opt for an ENIG finish for the TIL311 display boards, just because it looks nicer in gold. The boards are manufactured with black solder mask, making the gold pads stand out better.

I’ll describe the display board in a separate post after I’ve assembled it. For now, here’s what 4 of the boards look like, component side up:

TIL311 display PCBs

Like most PCB prototyping services, they track your order by printing some kind of order identifier onto each PCB. Usually they try to put this identifier underneath a component like an IC so it gets hidden when the board is fully populated, but sometimes they put it somewhere prominent, like under your product name. On this board, the identifier sits under IC4 but for the other board, it was under the product name.

The PCBs arrived in a shrink-wrapped bubbly packaging to protect the boards. There was also a desiccant thrown in for one set of the boards to keep it dry.

PCBs arrived in bubbly shrink-wrap

Continue reading

Raspberry Pi Zero as Multiple USB Gadgets

In case you haven’t heard, the Raspberry Pi Zero is the smallest, most low-cost device in the Raspberry Pi family, but it’s also the hardest to find. It has two Micro-B USB ports, one for power and another functions as a dual-role USB OTG port.

Raspberry Pi Zero, back side

One of the more interesting uses for the Raspberry Pi Zero is to get it to behave as a USB device, just like your USB flash drive, for example.

There have been several guides written already, such as the Adafruit one, but most of them were based on the old kernel gadget drivers, like g_serial and g_ether. It still works, but not as flexible and likely to be deprecated in future.

Continue reading