Bash String Manipulation

Bash has some convenient string manipulation functions. I often use substring removal which is very handy for trimming suffixes and prefixes from things like file names. These operations all evaluate to a new string and don’t modify the original variable.

Suffix removal
Remove the longest match from end:
    output=${string%%suffix}
Remove the shortest match from end:
    output=${string%suffix}

Prefix removal
Remove the longest match from front:
    output=${string##prefix}
Remove the shortest match from front:
    output=${string#prefix}

Great reference for more string operations: http://tldp.org/LDP/abs/html/string-manipulation.html

Receiving raw packets in Linux without pcap

Since there was a lot of interest in my post on sending raw Ethernet packets, this is an example of receiving packets on a raw socket. A compiling version can be found on github: https://gist.github.com/2862515. The application that I developed this for was to receive packets sent to a specific MAC address. The main sections are explained below:

Open raw socket listening for ETHER_TYPE

	if ((sockfd = socket(PF_PACKET, SOCK_RAW, htons(ETHER_TYPE))) == -1) {
		perror("listener: socket");	
		return -1;
	}

Set interface to promiscuous mode

	strncpy(ifopts.ifr_name, ifName, IFNAMSIZ-1);
	ioctl(sockfd, SIOCGIFFLAGS, &ifopts);
	ifopts.ifr_flags |= IFF_PROMISC;
	ioctl(sockfd, SIOCSIFFLAGS, &ifopts);

Allow socket to be reused

	if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &sockopt, sizeof sockopt) == -1) {
		perror("setsockopt");
		close(sockfd);
		exit(EXIT_FAILURE);
	}

Bind to device

	if (setsockopt(sockfd, SOL_SOCKET, SO_BINDTODEVICE, ifName, IFNAMSIZ-1) == -1)	{
		perror("SO_BINDTODEVICE");
		close(sockfd);
		exit(EXIT_FAILURE);
	}

Receive on socket

	numbytes = recvfrom(sockfd, buf, BUF_SIZ, 0, NULL, NULL);
	printf("listener: got packet %lu bytes\n", numbytes);

Print the data

	printf("\tData:");
	for (i=0; i<numbytes; i++) printf("%02x:", buf[i]);
	printf("\n");

Close socket

	close(sockfd);

Laptop SATA Harddrive Data Recovery

The story:
My friend recently had her laptop stop booting and took it to a repair shop that advised the hard drive was dying and replaced it. As in my last post on hard drive recovery, the technicians said they couldn’t get any data off it. Thankfully this is another success story of recovering data, this time using open source Linux tools 🙂

The victim:
A 500GB 2.5inch SATA hard drive from a laptop running Windows 7.

The tools:

The process:
I am running Ubuntu 10.04 and copying all the data between external drives.

  • Put the drive into an external eSATA enclosure
  • Purchase a new external hard drive with more space that the entire broken drive (1TB to accommodate a 500GB image and then recovering data from that image). Plug in the blank target hard drive. Mine is mounted to /media/Elements.
  • Determine the device file of the broken hard drive by first listing the existing drives:
    ls /dev/disk/by-id/ -la
    Then insert the broken hard drive and repeat the command to see what new drive has shown up. Mine is /dev/sdc
  • Install GNU ddrescue:
    sudo apt-get install gddrescue
  • Make an image of the drive using ddrescue (hdImage), with 3 retries for each error (-r 3), and noting the error locations in a log file (hdImageLog):
    sudo ddrescue -r 3 /dev/sdc /media/Elements/hdImage /media/Elements/hdImageLog
  • For me this process took about 20 hours. I left it over night the first day and got 250GBs. I then had to use the computer so killed ddrescue with Ctrl+C. Because of the log file I could easily restart the imaging process later with the same command:
    # sudo ddrescue -r 3 /dev/sdc /media/Elements/hdImage /media/Elements/hdImageLog

    Press Ctrl-C to interrupt
    Initial status (read from logfile)
    rescued: 289344 MB, errsize: 39337 kB, errors: 221
    Current status
    rescued: 500091 MB, errsize: 16330 kB, current rate: 0 B/s
    ipos: 22699 MB, errors: 270, average rate: 3603 kB/s
    opos: 22699 MB, time from last successful read: 34.9 m
    Splitting failed blocks...

    The second time I ran ddrescue it went back over the error blocks, tried to split them and recover more data in between. You can see that in the first run about 40MB of errors were read, but that went down to about 16MB. I left ddrescue for about 6 hours splitting failed blocks and then just killed it again because it wasn’t making any more progress. I figured about 4.6*10^-5 % errors was good!

  • Now I had an image, I tried just mounting it as is:
    sudo mount -t ntfs -o r,force,loop hdImage mnt
    to no avail.
  • I then decided to just try some data carving software to extract any files that I could, without getting the directory structure back. I tried both Photorec and Foremost on the same image but Foremost worked best in my case.
    sudo apt-get install foremost
    sudo foremost -i /media/Elements/hdImage -o /media/Elements/recovered/

    This takes a few hours and in the end I recovered about 40,000 files!
  • Happy that I’d got as much data as I can off the image, I tried to repair the filesystem using testdisk but didn’t have any luck in this case.
  • Resources:
    https://help.ubuntu.com/community/DataRecovery
    http://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html
    http://linux.die.net/man/1/foremost
    http://www.cgsecurity.org/wiki/PhotoRec_Step_By_Step