Striving for the ultimate bootable USB

Many people are working on this, many with a deeper understanding than me, but here is my take.

The requirement as I see it is for a USB key from which an arbitrary USB-bootable machine can be booted which can then load any one of a number of images. The most frequent requirement I have seen is to adjust the partitioning on a PC and the System Rescue CD works well for this but there are many other possible solutions. For each solution there will be an ISO image available so this leads the first requirement:

Requirement 1: Multiple bootable ISO images are easily added to boot menu

This approach is well-implemented by the folk at Pendrivelinux  who provide a program that will setup a USB stick that boots and displays a GRUB menu. The GRUB4DOS menu language is extensive and powerful enough to support booting from many different ISO images which simply need to be copied to the key. As a last resort (because certain images get confused about drives during booting) the image can be expanded onto the key in which case it is almost bound to work.

There is one major problem in this approach. There are two ways that a USB drive can be formatted:

1. As if it were a hard drive 
   The USB drive has a MBR (Master Boot Record) with a partition table and the usually single partition is marked as active (bootable) and has a Boot Sector that launches whatever OS is installed on that drive.
   This is the default way in which USB keys end up being formatted in Linux and Windows.
   From the Linux point of view this partition is referred to as /dev/sd[some letter]1 (number one!). To discover what the letter is run the Linux command dmesg just after you insert the stick - there will be references to sd[some letter] in the messages displayed. So for example:
   - dmesg shows that the device is sdd
   - The whole USB stick is then referred to as /dev/sdd
   - The first (usually only) partition is referred to as /dev/sdd1. GRUB menus will refer to this location using the GRUB terminology (hd0,0)
2. As if it were a floppy drive
   This is often described as a superfloppy. There is no MBR and partition table - the OS on the drive is booted through an appropriate VBR (Volume Boot Record) or boot sector on the drive.
   Formatting in this way is harder - in Linux you need to use the command:
   mkfs -I /dev/sd[some letter] for a Linux format or
   mkfs.vfat -I /dev/sd[some letter] for a VFAT format
   The -I option forces the raw device to be formatted
   GRUB menus will refer to this location using the GRUB terminology (fd0)

The ability of a particular machine BIOS to boot from a USB stick may depend on whether the stick is formatted as a hard disk or floppy. This is beyond my ability to test broadly but so far 2 out of 3 have required superfloppy formatted USB sticks to boot. I will keep trying other machines as I come across them and updating this post.  I have been told by Lance at pendrivelinux that it will be older BIOS's that impose the superfloppy restriction but I have come across it on an eeePC 901.

This leads to the second requirement:

Requirement 2: USB format (superfloppy/hard disk) should support as many machines as possible

This leads to a problem with the default Pendrivelinux approach. A Pendrive MULTIBOOT USB boots Syslinux which is very minimal Linux which can then launch other programs. In MULTIBOOT Syslinux is configured to run GRUB.EXE which then displays the menu of operating systems. This works just fine on a hard disk formatted USB stick. GRUB.EXE defaults to looking for a menu on (hd0,0) and when this is the stick all is well.

Booting from a superfloppy formatted USB stick fails to find the menu at (hd0,0)  and has to be told manually (with the GRUB command root (fd0) ) where the menu is. In theory this can be corrected through an entry in the syslinux.cfg file:

kernel grub.exe

append --config-file (fd0)/menu.lst

but in the MULTIBOOT environment this directive is not passed through to GRUB.EXE

This posting really concerns a solution to that problem (there is an experimental download on the pendrivelinux site that takes this approach) :

1. Make MULTIBOOT directly run GRUB.EXE
   The problem described above is a Syslinux one and the following changes can make the USB stick directly boot GRUB.EXE:
1. Use BOOTICE.EXE to install a GRUB.EXE bootsector on ynloadour superfloppy formatted USB stick. 
2. This bootsector looks for the file GRLDR on the USB stick
3. GRLDR runs GRUB.EXE but the version (0.4.4) from MULTIBOOT has problems locating ISO files with the find --set-root command so replace it with a newer GRUB.EXE
4. All reference to Syslinux can then be removed from the USB stick - they are no longer needed
2. Modify the boot menu for the (fd0) environmentThe boot menu is contained in the file menu.lst and contains many references to (hd0,0) - these should all be replaced with (fd0). It then remains to try out, and possibly adapt the entries provided. I place all the entries which I have not yet fixed in a separate file menu.extra  and then reference these as a sub-menu through and entry in menu.lst:
   
3. title Uninstalled & untested options
4. configfile (fd0)/menu.extra