Automated backup of an external drive on Linux

The following steps set up a backup for a particular USB drive every time it is inserted. Very useful for avoiding the perils of working directly from a flash drive on multiple computers.
 

Create the backup script that uses duplicity:

/usr/local/bin/usb_drive_backup.sh

#!/bin/bash

DRIVE_NAME=USBDRIVE
BACKUP_DIR=/home/username/BACKUPS/USBDriveBackup

sleep 4

duplicity incremental /media/username/USBDRIVE file://$BACKUP_DIR --no-encryption >> $BACKUP_DIR/log.txt
if [ $? -eq 0 ]; then
    echo "## Backup complete `date`" >> $BACKUP_DIR/log.txt
    echo
fi

/usr/local/bin/usb-backup-in_udev

#!/bin/bash

/usr/local/bin/usb_drive_backup.sh &

 
 
Set up the udev rule to run the backup script every time the drive is inserted:

Find the device ID

$ lsusb
Bus 002 Device 002: ID 0930:1400 Toshiba Corp. Memory Stick 2GB
...

/etc/udev/rules.d/00-usb-backup.rules

ACTION=="add", ATTRS{idVendor}=="0930", ATTRS{idProduct}=="1400", ENV{XAUTHORITY}="/home/username/.Xauthority", ENV{DISPLAY}=":0", OWNER="username", RUN+="/usr/local/bin/usb-backup-in_udev"

 
 
Restoring the backup:

duplicity restore file:///home/username/BACKUPS/USBDriveBackup ./RestoredFiles/ --no-encryption
Advertisements

Corrupt USB Flash Drive Data Recovery on Mac OS X Mavericks

An oldy but a goody…

  • Identify the device node
    Remove the drive from the computer then run
    $ ls /dev/rdisk*
    Insert the drive and re-run the command, for me a new device /dev/rdisk3 shows up
  • Checkout what might be wrong
    $ sudo dmesg | tail
    ...
    disk3s1: I/O error.
    0 [Level 3] [Facility com.apple.system.fs] [ErrType IO] [ErrNo 5] [IOType Write] [PBlkNum 14988] [LBlkNum 0]
    0 [Level 3] [Facility com.apple.system.fs] [DevNode /dev/disk3s1] [MountPt /Volumes/Lexar]
  • Make an image of the disk
    $ dd bs=64k if=/dev/rdisk3 of=./recovered.dmg conv=noerror,sync
    478816+0 records in
    478816+0 records out
    31379685376 bytes transferred in 1278.080407 secs (24552200 bytes/sec)
  • Try mount the created image in Finder (unlikely)
  • Install foremost for data carving
    $ brew install foremost
    ==> Downloading http://foremost.sourceforge.net/pkg/foremost-1.5.7.tar.gz
    ######################################################################## 100.0%
    ==> make mac
    🍺 /usr/local/Cellar/foremost/1.5.7: 5 files, 84K, built in 29 seconds
  • Run foremost on the image
    $ foremost -i ./recovered.dmg -o ./recovered
  • Browse the glorious recovered data!

Laptop SATA Harddrive Data Recovery

The story:
My friend recently had her laptop stop booting and took it to a repair shop that advised the hard drive was dying and replaced it. As in my last post on hard drive recovery, the technicians said they couldn’t get any data off it. Thankfully this is another success story of recovering data, this time using open source Linux tools 🙂

The victim:
A 500GB 2.5inch SATA hard drive from a laptop running Windows 7.

The tools:

The process:
I am running Ubuntu 10.04 and copying all the data between external drives.

  • Put the drive into an external eSATA enclosure
  • Purchase a new external hard drive with more space that the entire broken drive (1TB to accommodate a 500GB image and then recovering data from that image). Plug in the blank target hard drive. Mine is mounted to /media/Elements.
  • Determine the device file of the broken hard drive by first listing the existing drives:
    ls /dev/disk/by-id/ -la
    Then insert the broken hard drive and repeat the command to see what new drive has shown up. Mine is /dev/sdc
  • Install GNU ddrescue:
    sudo apt-get install gddrescue
  • Make an image of the drive using ddrescue (hdImage), with 3 retries for each error (-r 3), and noting the error locations in a log file (hdImageLog):
    sudo ddrescue -r 3 /dev/sdc /media/Elements/hdImage /media/Elements/hdImageLog
  • For me this process took about 20 hours. I left it over night the first day and got 250GBs. I then had to use the computer so killed ddrescue with Ctrl+C. Because of the log file I could easily restart the imaging process later with the same command:
    # sudo ddrescue -r 3 /dev/sdc /media/Elements/hdImage /media/Elements/hdImageLog

    Press Ctrl-C to interrupt
    Initial status (read from logfile)
    rescued: 289344 MB, errsize: 39337 kB, errors: 221
    Current status
    rescued: 500091 MB, errsize: 16330 kB, current rate: 0 B/s
    ipos: 22699 MB, errors: 270, average rate: 3603 kB/s
    opos: 22699 MB, time from last successful read: 34.9 m
    Splitting failed blocks...

    The second time I ran ddrescue it went back over the error blocks, tried to split them and recover more data in between. You can see that in the first run about 40MB of errors were read, but that went down to about 16MB. I left ddrescue for about 6 hours splitting failed blocks and then just killed it again because it wasn’t making any more progress. I figured about 4.6*10^-5 % errors was good!

  • Now I had an image, I tried just mounting it as is:
    sudo mount -t ntfs -o r,force,loop hdImage mnt
    to no avail.
  • I then decided to just try some data carving software to extract any files that I could, without getting the directory structure back. I tried both Photorec and Foremost on the same image but Foremost worked best in my case.
    sudo apt-get install foremost
    sudo foremost -i /media/Elements/hdImage -o /media/Elements/recovered/

    This takes a few hours and in the end I recovered about 40,000 files!
  • Happy that I’d got as much data as I can off the image, I tried to repair the filesystem using testdisk but didn’t have any luck in this case.
  • Resources:
    https://help.ubuntu.com/community/DataRecovery
    http://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html
    http://linux.die.net/man/1/foremost
    http://www.cgsecurity.org/wiki/PhotoRec_Step_By_Step

Dead iMac Hard Drive Recovery

This is a tale of loss, love but in the end, happiness. My girlfriends 20″ iMac hard drive recently died, although the computer was still under warranty the hard drive could not be easily recovered by ‘MagnumMac’ and if it is replaced under warranty you don’t get to keep the old drive! The hard drive contained the last 3 years photos and Uni work. There was no choice but to pay for a replacement and keep the broken drive. (For anyone who’s iMac starts going slow for a couple of months, then started making a clicking sound, a ‘plonking’ sound and the inability to boot will follow.) These are the steps I used to recover the drive, only loosing about 20 files in 160GB. Note that this drive was VERY dead, it would still spin but would not mount, it had a hardware fault and would ‘plonk’ off if you tried to do so.

  • Put the drive into an external SATA enclosure and connect it to a computer with more free space than the entire drive capacity.
  • Hopefully the disk will mount, if not open /dev directory and try find it. For me it was rdisk3. Now create an image of the disk using dd, from the terminal type:

    dd bs=64k if=/dev/rdisk3 of=/recovered.dmg conv=noerror,sync

    The original post I found used a blocksize of 512, but this was going to take about 25 days to copy using USB, I used 64k which only took about 3 hours. You will end up with a .dmg file.

  • Get DiskWarrior , it REALLY WORKS so I would advise buying it if you have the money. To use DiskWarrior the .dmg must be mounted. As mine wouldn’t mount I had to attach it using the command:

    hdiutil mount -nomount -readwrite /recovered.dmg

  • From DiskWarrior use the “Rebuild” option on the image which will now be in the popup menu. It only takes a minute and will come up with a Report. From the report click “Preview” and it will mount your files as a drive on your desktop. BACKUP YOUR DATA and enjoy!

Websites which made this all possible:
http://www.macosxhints.com/article.php?story=20050302225659382

http://www.friday.com/bbum/2006/02/20/recovering-disk-images-with-diskwarrior/