Hi Mark, it sounds like some more basic Linux information is necessary here before diving into bootloaders.
The root filesystem (rootfs) is indeed what you see when you type “ls /” from a working system. It is the head of the device’s filesystem - all other filesystems attached are mounted underneath /. For example, home directories for users are usually set up under /home, and it is possible to set up /home on a different partition and mount it, so that a hardware failure on / won’t take out your personal files. /home still shows up in the same place, even though it is actually a pointer to a different location.
On Linux, as opposed to many other OSes, the rootfs can be separate from the kernel, and thus it can be present in any of several different forms. The most common methods are (1) kernel and rootfs in one partition/filesystem, as with most other operating systems; (2) rootfs in a separate partition, as it is with Angstrom in the system I described below; or (3) rootfs in a compressed file that is expanded into active RAM rather than residing on a physical medium, known as a RAMdisk. The beagle can boot with any of these, but #2 and #3 are most common.
Some systems do away with x-loader by creating a boot ROM that provides the functions X-loader performs, namely initializing memory. Having separate, writable firmware that handles such low-level functions makes the system incredibly flexible, but it does mean an extra step for developers and an extra level of confusion for those new to the system.
It is certainly possible to make changes to the rootfs, write them to the SD card, and reboot. It might be more useful to set up networking on the target itself, though, and use opkg to install software from a repository, as you would with a desktop system. The person who had difficulty with their changes disappearing was probably either using a RAMdisk (which is not saved after the reboot) or was having some other difficulty with installation.
Note that in a production environment, the rootfs is very often NFS-mounted over a network so that the developer can make changes to the running system without swapping cards. This is not a trivial setup, but it can be useful if you are making a lot of changes to the rootfs during production.
I hope this helps! I can recommend a few places to read more about how Linux filesystems work, but the best is Chris Hallinan’s book “Embedded Linux Primer”.