#!/bin/bash
# 
# Automount script to mount LUKS-encrypted disk image file.
# 
# This script must be executable to work (chmod 755).
# 
# Requires losetup and cryptsetup to be available in 
# one of /bin, /sbin, /usr/bin, or /usr/sbin.
#
# The LUKS key must exist as a file at /etc/<key>.key

key="$1"
los=""
cry=""
img="/cifs/lacie-2big/backup/$key.sparseimage"
mountopts="-fstype=ext3,defaults,noatime,nodiratime"

if [ ! -e "/etc/$key.key" ]; then
	exit 0
fi

if [ ! -e "$img" ]; then
	exit 0
fi

# search for losetup and cryptsetup
for P in /bin /sbin /usr/bin /usr/sbin
do
	if [ -z "$los" -a -x $P/losetup ]; then
		los=$P/losetup
	fi
	if [ -z "$cry" -a -x $P/cryptsetup ]; then
		cry=$P/cryptsetup
	fi
	if [ -n "$los" -a -n "$cry" ]; then
		break
	fi
done

# check if a loop device already attached to this image
dev=`$los -a |grep $img |cut -d: -f1`
if [ -z "$dev" ]; then
	# select any available loop device
	dev=`$los -f`
	if [ -z "$dev" ]; then
		echo "No loop device available for mounting $img" >&2
		exit 1
	fi
	
	# attach loop device
	$los $dev $img
	
	# open with LUKS
	$cry isLuks $dev 2>/dev/null
	if [ "$?" -eq "0" ]; then
		$cry --key-file /etc/$key.key luksOpen $dev luks-`$cry luksUUID $dev` >/dev/null 2>&1
	fi
fi

# print out mapping for automount
echo $mountopts / :/dev/mapper/luks-`$cry luksUUID $dev`

This script first checks if the disk image is already attached to a loop device, and if so will not attach it again. When attaching to a loop device, it uses losetup -f to find any unused loop device and attaches the disk image to that one.

These scripts can be obtained via anonymous CVS:

cvs -d :pserver:anonymous@msqr.us:/data/cvs co twobig

1. Mount the SMB share when accessing the /cifs directory
2. Attach a loop device and set up LUKS filesystem access when accessing the /encrypted directory

$ ls /encrypted/linuxbackup

and for automount to auto-magically mount the SMB share and then mount the encrypted disk image. Unfortunately I could not find a way for automount to handle this "cascading" or "recursive" mount operation. Instead I have to execute two commands:

$ ls /cifs/lacie-2big/backup
linuxbackup.sparseimage
$ ls /encrypted/linuxbackup
hda1  hda2  hda3  hda6  hda7  lost+found

Still, two simple ls commands are better than the 4 hard-to-remember mount, losetup, cryptsetup, and mount commands needed to do this manually. And with the help of a small script along with what automount provides, I don't have to worry about the 4 hard-to-remember umount, cryptsetup, losetup, umount commands needed to unmount everything when I'm finished.

Automount CIFS mount script

I found online an example automount script based from the auto.smb one that comes in many distributions called auto.cifs that will work with SMB mounts requiring authentication. That script looks like:

#!/bin/bash
# $Id$
# This file must be executable to work! chmod 755!
key="$1"
# Note: create a cred file for each windows/Samba-Server in your network
#       which requires password authentification.  The file should contain
#       exactly two lines:
#          username=user
#          password=*****
#       Please don't use blank spaces to separate the equal sign from the
#       user account name or password.
credfile="/etc/auto.smb.$key"
# Note: Use cifs instead of smbfs:
mountopts="-fstype=cifs,file_mode=0644,dir_mode=0755,uid=root,gid=wheel"
smbclientopts=""
for P in /bin /sbin /usr/bin /usr/sbin
do
        if [ -x $P/smbclient ]
        then
                SMBCLIENT=$P/smbclient
                break
        fi
done
#echo $SMBCLIENT >&2
[ -x $SMBCLIENT ] || exit 1
if [ -e "$credfile" ]
then
        mountopts=$mountopts",credentials=$credfile"
        smbclientopts="-A "$credfile
else
        smbclientopts="-N"
fi
#echo $smbclientopts -gL $key >&2
$SMBCLIENT $smbclientopts -gL $key 2>/dev/null \
   | awk -v key="$key" -v opts="$mountopts" -F'|' -- '
        BEGIN   { ORS=""; first=1 }
	/Disk/  { if (first) { print opts; first=0 };
		  gsub(/ /, "\\ ", $2);
		  sub(/\$/, "\\$", $2);
		  print " \\\n\t /" $2, "://" key "/" $2 }
        END     { if (!first) print "\n"; else exit 1 }
        '

I created that as /etc/auto.cifs and made it executable. Then in /etc/auto.master I added this line:

/cifs /etc/auto.cifs --timeout=60

The credentials for mounting the SMB share are then stored in a /etc/auto.smb.lacie-2big file, which is how I set things up originally in my last post. Now, after reloading the autofs service, I am able to:

$ ls /cifs/lacie-2big/backup
linuxbackup.sparseimage

Automount LUKS script

Then I created another automount script /etc/auto.luks.loop.0:

#!/bin/bash
# This file must be executable to work! chmod 755!
# Make links to this file with the last digit replaced
# with other numbers for corresponding loop devices,
# e.g. if this script is executed as auto.luks.loop.1 the 
# /dev/loop1 device will be used.
#
# The LUKS key must exist as a file at /etc/.key

key="$1"
los=""
cry=""
name=`basename $0`
l=${name##*.}
img="/cifs/lacie-2big/backup/$key.sparseimage"
mountopts="-fstype=ext3,defaults,noatime,nodiratime"

if [ ! -e "/etc/$key.key" ]; then
	exit 1
fi

# search for losetup and cryptsetup
for P in /bin /sbin /usr/bin /usr/sbin
do
	if [ -z "$los" -a -x $P/losetup ]; then
		los=$P/losetup
	fi
	if [ -z "$cry" -a -x $P/cryptsetup ]; then
		cry=$P/cryptsetup
	fi
	if [ -n "$los" -a -n "$cry" ]; then
		break
	fi
done

# check if loop device already attached, if not then attach it
chk=`$los -a |grep /dev/loop$l`
if [ -z "$chk" ]; then
	if [ ! -e $img ]; then
		echo "Image file $img not found." >&2
		exit 1
	fi
	$los /dev/loop$l $img
	$cry --key-file /etc/$key.key luksOpen /dev/loop$l \
		luks-`$cry luksUUID /dev/loop$l` >/dev/null 2>&1
fi

echo $mountopts / :/dev/mapper/luks-`$cry luksUUID /dev/loop$l`

This script is hard-coded to look for disk image files in /cifs/lacie-2big/backup named imagefilename.sparseimage where filename will be the automount directory name. The script uses the last number of the script name to determine which loop device to use, although it could be easily adapted to use any freely-available device (by way of losetup -f).

Then it uses cryptsetup to open the LUKS device, using a key file located at /etc/imagefilename.key. Then it configures automount to mount the LUKS filesystem. In short, the script basically does:

$ losetup /dev/loop0 /cifs/lacie-2big/backup/linuxbackup.sparseimage
$ cryptsetup --key-file /etc/linuxbackup.key luksOpen /dev/loop0 \
	luks-`cryptsetup luksUUID /dev/loop0`
$ echo -fstype=ext3,defaults,noatime,nodiratime \
	/ :/dev/mapper/luks-`cryptsetup luksUUID /dev/loop0`

That last line is what is returned to automount and causes it to mount the image file as a directory named filename.

Finally in /etc/auto.master I added this line:

/encrypted /etc/auto.luks.loop.0 --timeout=600

Afer having the autofs reload this configuration, I am then able to do this:

ls /encrypted/linuxbackup
hda1  hda2  hda3  hda6  hda7  lost+found

With the caveat that the /cifs/lacie-2big/linuxbackup directory is already mounted. This is where I couldn't find a way for automount to mount both file systems in one call.

Auto-umount

One final piece remains to be automated, however: completely unmounting both the encrypted and SMB file systems. automount will take care unmounting the LUKS encrypted filesystem after the configured period of inactivity. However, it won't be able to unmount the SMB filesystem because the auto.luks.loop.0 script has attached the /dev/loop0 device to the disk image file on that share. In effect, the SMB share is still in use.

It would be very nice if automount provided a way to execute scripts when it unmounted a filesystem. But it does not. My solution was to write a small script that runs every so often (via cron) that looks to see if any loop devices are attached to filesystems that are "not in use" and if found, detach them. Here's the script, which I have stored at /etc/auto.luks.loop.umount:

#!/bin/bash

los=""
cry=""

# search for losetup and cryptsetup
for P in /bin /sbin /usr/bin /usr/sbin
do
	if [ -z "$los" -a -x $P/losetup ]; then
		los=$P/losetup
	fi
	if [ -z "$cry" -a -x $P/cryptsetup ]; then
		cry=$P/cryptsetup
	fi
	if [ -n "$los" -a -n "$cry" ]; then
		break
	fi
done

for dev in `$los -a|cut -d: -f1`; do
	file=`$los -a|grep /dev/loop |sed 's/.*(\(.*\)).*/\1/'`
	dir=${file%/*}
	# check if the only open file on loopback file's filesystem 
	# is from automount, if so close loopback device so automount
	# can un-mount filesystem for us later
	match=`lsof |grep $dir|cut -d" " -f1 |grep -v automount`
	if [ -n "$match" ]; then
		echo "Loop device $dev attched to in-use filesystem $dir" >&2
	else
		echo "Loop device $dev on unused filesystem $dir"
		$cry isLuks $dev 2>/dev/null
		if [ "$?" -eq "0" ]; then
			echo "Closing LUKS on $dev"
			$cry luksClose luks-`$cry luksUUID $dev`
		fi
		echo "Unattaching loop device $dev"
		$los -d $dev
	fi
done

It's not particularly clever. I'm sure it would not work on systems using loop devices for other things that what I'm using them for on my system and it's making assumptions on the output of losetup. But it does the job nicely for what I need. I have this run every so often via cron. After it runs, and if it found any loop devices it could detach, then automount will eventually unmount the SMB share for us and we're done.

These scripts can be obtained via anonymous CVS:

cvs -d :pserver:anonymous@msqr.us:/data/cvs co twobig

The LaCie NAS only supports AFP, SMB, HTTP, and FTP protocols, however. The most logical one to use with Linux would be SMB, as the Samba project provides the necessary tools for mounting and using SMB network shares. The trouble is that a SMB share will not preserve the Linux filesystem attributes I need to preserve.

I came across a useful web page, titled Optimal remote backups with rsync over Samba, that provided me with an excellent starting point for implementing my backup process. The idea is to use a sparse disk image file stored on the NAS and mounted as a loop-back device in Linux. I just needed to add encryption, which turned out to be quite easy with the LUKS support available in RHEL 5 (the distribution the server runs).

Setup Tasks

1) Create the SMB share for the backup, with a username and password, on the NAS

This was done using the NAS's web-based administration console. I created a user for the Linux server to connect as, and created a share for that user to use.

2) Mount the SMB share on Linux server

The Linux CIFS filesystem support provided by Samba allows us to mount the NAS network share. I know that later on I'll want to configure an automount for this share, and there are automount scripts available that can make use of a credentials file with the username/password for mounting the share with. So the first step is to create a credentials file with the username and password in it, named /etc/auto.smb.server. Here server represents the IP address or DNS name of the NAS. The file should only be readable by root, and the format looks like:

username=linuxbackup
password=mypass

Here linuxbackup is the user I created on the NAS for accessing the backup share. Note there shouldn't be any spaces around the equal signs.

Now create a mount point directory, and mount the SMB share:

$ mkdir /mnt/nas-backup-share
$ mount -t cifs //server/share /mnt/nas-backup-share \
	-o lfs,credentials=/etc/auto.smb.server,uid=root,gid=wheel,\
	file_mode=0660,dir_mode=0770,rw

3) Create sparse disk image to hold backup filesystem

$ dd if=/dev/zero of=/mnt/nas-backup-share/linuxbackup.sparseimage \
    bs=1M count=1 seek=150000

4) Setup loop device

$ losetup /dev/loop0 /mnt/nas-backup-share/linuxbackup.sparseimage

5) Setup LUKS disk encryption

RHEL 5 comes with the LUKS package for whole-disk encryption. It can be used with loop-back mounted devices. First I initialize the LUKS device with luksFormat and then open it with luksOpen:

$ cryptsetup luksFormat /dev/loop0

WARNING!
========
This will overwrite data on /dev/loop0 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase: 
Verify passphrase: 
Command successful.

$ cryptsetup luksOpen /dev/loop0 luks-`cryptsetup luksUUID /dev/loop0`
Enter LUKS passphrase: 
key slot 0 unlocked.
Command successful.

Now the LUKS device is accessible like any normal block device at /dev/mapper/luks-UUID where UUID is the value returned by cryptsetup luksUUID /dev/loop0. Using this naming convention may not be friendly for using now, but later on I'll setup automount so I can refer to things with friendlier names that I can remember.

6) Create filesystem for backup image

The encrypted device is now ready to have a filesystem created on it. I created an ext3 filesystem, but any one Linux supports could be used:

mkfs.ext3 -L BACKUP /dev/mapper/luks-`cryptsetup luksUUID /dev/loop0`
mke2fs 1.39 (29-May-2006)
Filesystem label=BACKUP
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
19202048 inodes, 38400127 blocks
1920006 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
1172 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

7) Create strong random password for LUKS volume

I can increase the strength of the password used to unlock the LUKS volume by creating a random one and removing the original one I setup. To create the random password, I can use dd again:

$ dd if=/dev/urandom of=/etc/linuxbackup.key bs=32 count=1
1+0 records in
1+0 records out
32 bytes (32 B) copied, 0.00018131 seconds, 176 kB/s
$ chmod 600 /etc/linuxbackup.key

This creates a 256-bit key stored in the /etc/linuxbackup.key file that can be used to open the LUKS volume. Now I can remove our original key:

$ cryptsetup luksDelKey /dev/loop0 0
Command successful.

8) Mount filesystem

Finally I can mount the LUKS encrypted device like a normal filesystem:

$ mkdir /mnt/linuxbackup
$ mount -t ext3 /dev/mapper/luks-`cryptsetup luksUUID /dev/loop0` \
    /mnt/linuxbackup -o defaults,noatime,nodiratime

I can use df to verify that the filesystem is alive and well:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/luks-259b10cd-d2f1-41a4-bf95-a533ac0c927c
                      145G  188M  137G   1% /mnt/linuxbackup

Ready for backup

Now that the encrypted device is mounted, I can use it for backups, using my normal rsync-based backup script.

Unmounting the encrypted filesystem

In order to completely umount the encrypted filesystem, I have to unmount it, close it with LUKS, detach the loop-back device, and finally unmount the NAS SMB share:

$ umount /mnt/linuxbackup
$ cryptsetup luksClose luks-`cryptsetup luksUUID /dev/loop0`
$ losetup -d /dev/loop0
$ umount /mnt/nas-backup-share

Automation tasks

This was a good first step. Now the next goals are to automate the mounting of the encrypted filesystem with automount and to get the NAS to "sleep" when not in use to conserve power. I'll details those in a future post.

It turned out not to be that difficult to do. I dusted off my (very disused) Perl skills and created a Perl program that can log into the NAS and boss it around, as well as send the wake-on-lan request to it to bring it back from hibernation.

To make the NAS sleep, I just execute

twobig --host lacie-2big --credentials /etc/twobig.cred sleep

To make the NAS wake up, I just execute

twobig --host lacie-2big --credentials /etc/twobig.cred wake

The script even checks if there are any connected users (as reported by the NAS) when the "sleep" command runs, and won't shut the NAS down if there are.

I can also use the script to ping the NAS, with

twobig --host lacie-2big --credentials /etc/twobig.cred --timeout 90 ping

This can be combined with the "wake" command, so the script does not return until the NAS has actually finished booting up, which can take nearly 90 seconds in my experience:

twobig --host lacie-2big --credentials /etc/twobig.cred --timeout 90 wake ping

I plan to integrate this into an automount script so the NAS can be woken up automatically when needed.

The script can be obtained via anonymous CVS:

cvs -d :pserver:anonymous@msqr.us:/data/cvs co twobig

"Up, up and away!" reads the "For Sale" sign in front of our house. And after 44 days, we got a mini Christmas miracle by way of a sale! It was a painfully difficult process, so it's a huge relief to have a happy outcome on this whole house-selling hooha.

I dug out my little patch of carrots today. I guess I didn't thin them out enough to grow past "baby carrot" size. It was a monumental cleaning job, attending to all those little carrots, but now the job is done and soon they will be cooked with some mustard and honey... ummm.

Lillian knows exactly what music she wants to listen to these days, and it isn't Mozart, Chopin, or any other "baby classical" you find in so many places. She asks for it, dare I say demands it, by name.

That name is David Bowie.

One of the albums we got was David Bowie's Hunky Dory. Minette and I had on several occasions mentioned that we wished we had some more Bowie in our music collection, since we literally had only Space Oddity and the Under Pressure single. We didn't have any particular album in mind, but listened to many different tracks on many different albums and decided to go with Hunky Dory.

The next day, March 2nd, 2008, I was obviously excited to listen to the new album. I put it on the stereo, cranked the volume to a comforably loud (but quite reasonably considerate) level and had a listen.

And it was good.

I must have explained to Lillian a few times that this new music was "David Bowie." Perhaps I said it very enthusiastically. Perhaps it just struck a chord (no pun intended) with her. Perhaps after playing the album 5 or 6 times over the next few days it got permanently etched in her impressionable synapsis.

Whatever the reason, she now asks to "hear David Bowie!" once, some times 28, times on any given day. She really can be quite tenacious about it, too. She might start intoning "ch-ch-ch-ch-changes!" (Changes, Hunky Dory) or she might chant "yeah girl!" ((Don't Sit Down), Space Oddity) or she might exclaim "Oh yeah!" (Queen Bitch, Hunky Dory).

Sometimes she might shout "here comes.... Bowie!" and we might cave in and turn it on (for the Nth time) only for her to dance a few groovy dance moves and then shout "Turn off!" until we turn it off. Then she might repeat this cycle as many times as we can bear.

After we'd listened to Hunky Dory about 1 million times, we put on Space Oddity once when she asked for Bowie. I did say, "OK, here's Bowie." quietly, and she listened intently and knew it was so and was happy. It's as if she actually recognizes Bowie's voice, not just the repetition of the songs from Hunky Dory. I was very impressed!

Perhaps she really does just like Bowie. Well good on ya, Lily! A fine choice, I have to agree.

It's happening... again!

There are still little missing pieces and things might not all be working... but I will just have to fix those things as I stumble on them.