Booting Linux off of a USB drive.

Last modified: November 30, 2003.

Update on February 10, 2004. Using USB boot and systemimager.

I just figured out how to use a small USB pen drive as the primary hard drive for Linux. This information floats on the net in bits and pieces, so I figured I'd put it together. You can also do this with a kernel patch listed here (look for boot-usb.patch). However, the initrd way, while longer to set up initially, is easier to reuse when you upgrade the kernel.

You have to first install Linux (I used Redhat 9.0, with which you might need to go into the "linux expert" mode) on your hard drive. Then, when it won't boot, use the rescue CD.

0. Make sure your motherboard supports booting off of USB. You might need to enable USB in the BIOS. Try all USB slots (one out of two did not work for me).

1. If your rescue CD would not mount the partitions (for some reason, mine did not), you have to mount them manually and then change root to the hard disk's filesystem. In my case, 

mount /dev/sda3 /mnt/source
mount /dev/sda1 /mnt/source/boot
chroot /mnt/source

2. You'll need to create a custom initrd image. Take your current image from /boot (see /etc/lilo.conf or /etc/grub.conf for the exact name). I'll call it initrd.img. (Use mkinitrd to create an initrd image if you don't have one.) 

cp /boot/initrd.img /tmp/initrd.gz

3. 

gunzip /tmp/initrd.gz

4. 

mkdir /tmp/a

5. 

mount -o loop /tmp/initrd /tmp/a

If this step fails, there is no loopback device support in your kernel. You have to compile it in.

6. The file /tmp/a/linuxrc is the script that runs before the root partition is mounted. We need to make sure it can mount a USB partition. If you use Redhat, replace /tmp/a/linuxrc with this (edit your script otherwise) : 

#!/bin/nash

echo "Loading jbd.o module"
insmod /lib/jbd.o
echo "Loading ext3.o module"
insmod /lib/ext3.o
echo Mounting /proc filesystem
mount -t proc /proc /proc
echo "Loading usb modules"
insmod /lib/usbcore.o
insmod /lib/scsi_mod.o
insmod /lib/sd_mod.o
insmod /lib/usb-storage.o
insmod /lib/usb-uhci.o
/bin/sleep 5
echo Creating block devices
mkdevices /dev
echo Creating root device
mkrootdev /dev/root
/bin/sfdisk -R /dev/sda
echo 0x0100 > /proc/sys/kernel/real-root-dev
echo Mounting root filesystem
mount -o defaults --ro -t ext3 /dev/root /sysroot
pivot_root /sysroot /sysroot/initrd
umount /initrd/proc

The lines before "Loading usb modules" and after "sfdisk", as well as mkdevices an mkrootdev are probably already there, so you just need to insert the extra stuff. First the script loads all the necessary kernel modules, then waits five seconds in order for the USB code to recognize the disk, then runs sfdisk to reread the partition table.

Note: Alan Deehr reported that usb-uhci.o did not see his USB 2.0 external disk. He used usb/host/ehci-hcd.o instead.

7. Copy kernel modules into the initrd image. For example, if you are running kernel 2.4.22 (uname -r to find out), 

cd /lib/modules/2.4.22/kernel/drivers
cp  usb/usbcore.o /tmp/a/lib
cp  usb/usb-uhci.o /tmp/a/lib
cp  usb/storage/usb-storage.o /tmp/a/lib
cp  scsi/scsi_mod.o /tmp/a/lib
cp  scsi/sd_mod.o /tmp/a/lib
If you have a monolithic kernel without the modules, you have to prepare a kernel that has them first.

8. Find out what libraries sleep and sfdisk use from the output of ldd: 

ldd /bin/sleep
ldd /sbin/sfdisk

Copy the libraries into /tmp/a/lib. In my case it was: 

mkdir /tmp/a/lib/tls
cp /lib/tls/libm.so.6 /tmp/a/lib/tls
cp /lib/tls/libpthread.so.0 /tmp/a/lib/tls
cp /lib/tls/libc.so.6 /tmp/a/lib/tls
cp /lib/librt.so.1 /tmp/a/lib
cp /lib/ld-linux.so.2 /tmp/a/lib

9. Copy the executables over: 

cp /bin/sleep /tmp/a/bin
cp /sbin/sfdisk /tmp/a/bin

10. 

umount /tmp/a

11. 

gzip /tmp/initrd

12. 

cp /tmp/initrd.gz /boot/usbinird.gz

13. Update your lilo.conf or grub.conf to use the new usbinitrd.gz with the same kernel that you grabbed the modules from. Run lilo or grub-install.

Simon Ilyushchenko
simonf@simonf.com

P.S. Presumably you can also patch the kernel to add a delay before a USB device is mounted at boot time.

If you want to use Systemimager to auto-install your nodes (which you should - it's a great tool), beware of some differences. Systemimager works by taking an image of a working node, booting into its own temporary kernel on another node and copying all the files over. Obviously, we want to make sure that the temporary kernel can see the USB drive, or there will not be a place to copy files to.

The differences with the process above:

  1. The initrd file is included into the RPM package systemimager-i386boot-standard. On my system its files are in /usr/share/systemimager/boot/i386/standard.
  2. The initrd file is cramfs, not loopback filesystem. This means that you can't edit a mounted file, you have to 
    cp -r
    the whole mounted directory (let's call the new directory mydir), change its contents and then 
    mkcramfs mydir initrd
gzip -f initrd
cp -f initrd.gz /tftpboot/initrd.img
  3. As you can see from the previous step, initrd.img is usually placed into /tftpboot. That's where the boot files for the systemimager kernel live.
  4. The same RPM package systemimager-i386boot-standard includes the necessary USB kernel modules. Untar boel_binaries.tar.gz and copy the USB modules into mydir/lib.
  5. As luck would have it, one of the modules (usb-storage.o) was missing. I had to install the systemimager source RPM, change the kernel configuration file (CONFIG_USB_STORAGE=m) and "make modules".
  6. The script that is run by initrd is /etc/init.d/rcS. It's more complicated than the usual initrd scripts. Put the section that loads modules in the beginning of the control flow (before switch_root_to_tmpfs) and the sfdisk call after get_boel_binaries_tarball.
  7. Note that the temporary systemimager kernel uses devfs, not the regular dev filesystem. If everything goes well, you don't need to know that, but it can make debugging interesting.