Automated backup of an external drive on Linux

The following steps set up a backup for a particular USB drive every time it is inserted. Very useful for avoiding the perils of working directly from a flash drive on multiple computers.
 

Create the backup script that uses duplicity:

/usr/local/bin/usb_drive_backup.sh

#!/bin/bash

DRIVE_NAME=USBDRIVE
BACKUP_DIR=/home/username/BACKUPS/USBDriveBackup

sleep 4

duplicity incremental /media/username/USBDRIVE file://$BACKUP_DIR --no-encryption >> $BACKUP_DIR/log.txt
if [ $? -eq 0 ]; then
    echo "## Backup complete `date`" >> $BACKUP_DIR/log.txt
    echo
fi

/usr/local/bin/usb-backup-in_udev

#!/bin/bash

/usr/local/bin/usb_drive_backup.sh &

 
 
Set up the udev rule to run the backup script every time the drive is inserted:

Find the device ID

$ lsusb
Bus 002 Device 002: ID 0930:1400 Toshiba Corp. Memory Stick 2GB
...

/etc/udev/rules.d/00-usb-backup.rules

ACTION=="add", ATTRS{idVendor}=="0930", ATTRS{idProduct}=="1400", ENV{XAUTHORITY}="/home/username/.Xauthority", ENV{DISPLAY}=":0", OWNER="username", RUN+="/usr/local/bin/usb-backup-in_udev"

 
 
Restoring the backup:

duplicity restore file:///home/username/BACKUPS/USBDriveBackup ./RestoredFiles/ --no-encryption

Advertisements

Corrupt USB Flash Drive Data Recovery on Mac OS X Mavericks

An oldy but a goody…

• Identify the device node
  Remove the drive from the computer then run
  $ ls /dev/rdisk*
  Insert the drive and re-run the command, for me a new device /dev/rdisk3 shows up
• Checkout what might be wrong
  $ sudo dmesg | tail
...
disk3s1: I/O error.
0 [Level 3] [Facility com.apple.system.fs] [ErrType IO] [ErrNo 5] [IOType Write] [PBlkNum 14988] [LBlkNum 0]
0 [Level 3] [Facility com.apple.system.fs] [DevNode /dev/disk3s1] [MountPt /Volumes/Lexar]
• Make an image of the disk
  $ dd bs=64k if=/dev/rdisk3 of=./recovered.dmg conv=noerror,sync
478816+0 records in
478816+0 records out
31379685376 bytes transferred in 1278.080407 secs (24552200 bytes/sec)
• Try mount the created image in Finder (unlikely)
• Install foremost for data carving
  $ brew install foremost
==> Downloading http://foremost.sourceforge.net/pkg/foremost-1.5.7.tar.gz
######################################################################## 100.0%
==> make mac
🍺 /usr/local/Cellar/foremost/1.5.7: 5 files, 84K, built in 29 seconds
• Run foremost on the image
  $ foremost -i ./recovered.dmg -o ./recovered
• Browse the glorious recovered data!

greping for text in source files only

Searching source and header files is my most common grep so I like to set up the following aliases:

alias grepch="grep --include=*.[ch] --include=*.cpp -nr"
alias grepc="grep --include=*.[c] --include=*.cpp -nr"
alias greph="grep --include=*.[h] -nr"

Using git as a subversion interface – git-svn

Git is my favourite source control system but unfortunately some companies still use historic subversion repos that we have to put up with. This is my basic workflow when dealing with svn.

Importing the repo

Cloning the svn repo to git:
git svn clone https://address/to/svn/repo

For a very large repo this periodically failed with:
error: git-svn died of signal 13
To continue from the same spot just run:
git svn fetch

This happened quite a few times when cloning from a slow server that took all day.

Note that git-svn doesn’t recursively clone svn externals (nested repositories). There are a few complex solutions on stackoverflow, haven’t picked my favourite yet.

Pulling changes

Instead of a git pull I do:

git stash; git svn fetch; git svn rebase; git stash pop

Commiting changes

For simple projects I think a major advantage of git is the ability to easily queue commits from the command line. I use my normal process of the interactive prompt to add files and a commit message:

git add -i
git commit -m "Message"

Then instead of a git push I do:

git stash; git svn dcommit; git svn rebase; git stash pop

Bash – Loops using pipe and read

Perform some operation for every line in the output of a command, e.g. add .txt to every file in a directory:

ls /var/* | while read fname; do cp $fname $fname.txt; done

Instead of:

for fname in `ls /var/*`; do cp $fname $fname.txt; done

Neat trick I actually picked up in a job interview. Easier to read if the original command is long as it avoids the backticks, and bash is all about readability, right?? 🙂

Bash String Manipulation

Bash has some convenient string manipulation functions. I often use substring removal which is very handy for trimming suffixes and prefixes from things like file names. These operations all evaluate to a new string and don’t modify the original variable.

Suffix removal
Remove the longest match from end:
    output=${string%%suffix}
Remove the shortest match from end:
    output=${string%suffix}

Prefix removal
Remove the longest match from front:
    output=${string##prefix}
Remove the shortest match from front:
    output=${string#prefix}

Great reference for more string operations: http://tldp.org/LDP/abs/html/string-manipulation.html

Mozilla Game On: Hackable Javascript + HTML5 Game

I have submitted my entry to the Mozilla Game On competition under the ‘hackable games’ category. It can be played online now at: http://austinmarton.github.com/phys-ed/

I developed a basic platformer written from scratch in Javascript using HTML5 canvas and some jQuery. It has a ‘mod mode’ in which the user can view some of what is going on behind the scenes and change some of the physical parameters. Of course I ran out of time in the end but the competition was a great push to at least get a playable prototype out there.

It is released under GPLv3 so please check out or even fork the source code on github!