how to get read-only files system these days?

I used to do this:

    perl -pi -e 's/^ExecStart/\#ExecStart/'
/lib/systemd/system/remount-rootfs.service

And then add these line to /etc/fstab:

    tmpfs /tmp tmpfs defaults
        0 0
    tmpfs /var/run tmpfs defaults
        0 0

    # Add fstab lines that specify tmpfs for a couple dirs
    test `grep tmpfs /etc/fstab | wc -l` -eq 2' || ( \
          scp tmpfs_lines $(BBL):/tmp && \
          ssh $(BBL) 'cat /tmp/tmpfs_lines >>/etc/fstab' )

But it seems that /lib/systemd/system/remount-rootfs.service doesn't
exist in the current image, and it wasn't obvious what has replaced
it.

Any help modifying the above recipe or pointer to how to do it these
days would be appreciated.

I just declare the root file system as read-only in fstab and create some ramdisks for stuff that needs to be writable.

LABEL=rootfs / ext4 ro,noatime,errors=remount-ro 0 1
debugfs /sys/kernel/debug debugfs defaults 0 0
tmpfs /var/log tmpfs defaults,noatime,nosuid,mode=0777,size=128M 0 0

tmpfs /var/tmp tmpfs defaults,noatime,nosuid,size=16M 0 0
tmpfs /tmp tmpfs defaults,noatime,nosuid,size=128M 0 0

Service generic-board-startup (a.k.a. /opt/scripts/boot/generic-startup.sh) must run once with a writable root to set up a bunch of stuff (on later boots it complains, but it doesn’t seem to matter). After that you’re stuck hacking all the software which expects to be able to write to disk. For a working network, “/etc/resolv.conf” needs symlinking to a writable location. Many services expect to persistently store their stuff under “/var/lib”.

I’ve used tmpfiles.d to create required symlinks, e.g.

$ cat /etc/tmpfiles.d/10-ro-symlinks.conf

L+ /etc/resolv.conf - - - - /var/run/symlinks/etc/resolv.conf
L+ /etc/ppp/resolv.conf - - - - /var/run/symlinks/etc/ppp/resolv.conf

Interesting. The question is why the way I used to do it ever worked.
It did seem to though: could change files around and next boot they
would be back to their original state. I never had to do anything
special for /etc/resolv.conf or anything. Perhaps it was only ending
up applying to /root or something.

What I wish is that there was a way to tell the kernel "look, never
actually sync this file because I don't care and I don't trust the
underlying media, go ahead and gradually fill memory and die I don't
care about that either because I'm going to hard boot you every 24
hours anyway".

Britton

Hmm having tried this I don't get it. This fragment seem to be assuming
that:

  * the dirs /var/run/symlinks/etc etc. are created somehow, maybe
with a d line in the same file?

  * the link target files /var/run/symlinks/etc/resolv.conf exist,
I guess they can just be empty as connection manager is going to fix
them up for me anyway?

Thanks,
Britton

Right. I wanted to emphasize only the important lines, but prerequisites are important too :slight_smile:

Here’s the full file

#Type Path Mode UID GID Age Argument
d /var/run/symlinks 0755 - - -

d /var/run/symlinks/etc 0755 - - -
d /var/run/symlinks/etc/ppp 0755 - - -
f /var/run/symlinks/etc/resolv.conf 0755 - - -
f /var/run/symlinks/etc/ppp/resolv.conf 0755 - - -

NB! Connman might override this by linking “/usr/lib/tmpfiles.d/connman_resolvconf.conf” to “/run/connman/resolv.conf”

L+ /etc/resolv.conf - - - - /var/run/symlinks/etc/resolv.conf
L+ /etc/ppp/resolv.conf - - - - /var/run/symlinks/etc/ppp/resolv.conf

Right. I wanted to emphasize only the important lines, but prerequisites are important too :slight_smile:

Here’s the full file

#Type Path Mode UID GID Age Argument
d /var/run/symlinks 0755 - - -

d /var/run/symlinks/etc 0755 - - -
d /var/run/symlinks/etc/ppp 0755 - - -
f /var/run/symlinks/etc/resolv.conf 0755 - - -
f /var/run/symlinks/etc/ppp/resolv.conf 0755 - - -

NB! Connman might override this by linking “/usr/lib/tmpfiles.d/connman_resolvconf.conf” to “/run/connman/resolv.conf”

L+ /etc/resolv.conf - - - - /var/run/symlinks/etc/resolv.conf
L+ /etc/ppp/resolv.conf - - - - /var/run/symlinks/etc/ppp/resolv.conf

>>
>> Any help modifying the above recipe or pointer to how to do it these
>> days would be appreciated.
>
>
> I just declare the root file system as read-only in fstab and create
> some
> ramdisks for stuff that needs to be writable.
>
> LABEL=rootfs / ext4 ro,noatime,errors=remount-ro 0 1
> debugfs /sys/kernel/debug debugfs defaults 0 0
> tmpfs /var/log tmpfs defaults,noatime,nosuid,mode=0777,size=128M 0
> 0
> tmpfs /var/tmp tmpfs defaults,noatime,nosuid,size=16M 0 0
> tmpfs /tmp tmpfs defaults,noatime,nosuid,size=128M 0 0
>
> Service generic-board-startup (a.k.a.
> /opt/scripts/boot/generic-startup.sh)
> must run once with a writable root to set up a bunch of stuff (on later
> boots it complains, but it doesn't seem to matter). After that you're
> stuck
> hacking all the software which expects to be able to write to disk. For
> a
> working network, "/etc/resolv.conf" needs symlinking to a writable
> location.
> Many services expect to persistently store their stuff under "/var/lib".
>
> I've used tmpfiles.d to create required symlinks, e.g.
>
> $ cat /etc/tmpfiles.d/10-ro-symlinks.conf
> L+ /etc/resolv.conf - - - - /var/run/symlinks/etc/resolv.conf
> L+ /etc/ppp/resolv.conf - - - - /var/run/symlinks/etc/ppp/resolv.conf

Hmm having tried this I don't get it. This fragment seem to be assuming
that:

  * the dirs /var/run/symlinks/etc etc. are created somehow, maybe
with a d line in the same file?

  * the link target files /var/run/symlinks/etc/resolv.conf exist,
I guess they can just be empty as connection manager is going to fix
them up for me anyway?

Thanks,
Britton

Right. I wanted to emphasize only the important lines, but prerequisites are
important too :slight_smile:

Here's the full file

#Type Path Mode UID GID Age Argument
d /var/run/symlinks 0755 - - -
d /var/run/symlinks/etc 0755 - - -
d /var/run/symlinks/etc/ppp 0755 - - -
f /var/run/symlinks/etc/resolv.conf 0755 - - -
f /var/run/symlinks/etc/ppp/resolv.conf 0755 - - -

# NB! Connman might override this by linking
"/usr/lib/tmpfiles.d/connman_resolvconf.conf" to "/run/connman/resolv.conf"
L+ /etc/resolv.conf - - - - /var/run/symlinks/etc/resolv.conf
L+ /etc/ppp/resolv.conf - - - - /var/run/symlinks/etc/ppp/resolv.conf

Ok this work, thanks so much. There are still a couple things I don't
get though:

* I notice in your original replay you don't have a tmpfs for /run
  (which is sym linked to from /var/run). So I wonder how the dirs
  and files you create from /etc/tmpfiles.d/10-ro-symlinks.conf are
  getting created, since I would think they would be on the (read-only)
  root file system. Is there something missing from the fstab you
  originally showed? I included this line as well:

     tmpfs /run tmpfs defaults,noatime,nosuid,size=16M 0 0

  I haven't gone back and tested without it, so maybe it works as you
  show and I just don't understand how.

* I end up with a rw tmpfs for /root (root home dir), as shown by
  cat /etc/mtab (last line):

    root@bbone217slave:/etc# cat /etc/mtab
    sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
    proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
    udev /dev devtmpfs rw,relatime,size=10240k,nr_inodes=55575,mode=755 0 0
    devpts /dev/pts devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
    tmpfs /run tmpfs rw,nosuid,noatime,size=16384k,mode=755 0 0
    /dev/mmcblk1p1 / ext4 ro,noatime,errors=remount-ro,data=ordered 0 0
    securityfs /sys/kernel/security securityfs
rw,nosuid,nodev,noexec,relatime 0 0
    tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
    tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
    tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
    cgroup /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
0 0
    cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /sys/fs/cgroup/net_cls,net_prio cgroup
rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /sys/fs/cgroup/devices cgroup
rw,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /sys/fs/cgroup/cpuset cgroup
rw,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /sys/fs/cgroup/perf_event cgroup
rw,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
    cgroup /sys/fs/cgroup/freezer cgroup
rw,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /sys/fs/cgroup/memory cgroup
rw,nosuid,nodev,noexec,relatime,memory 0 0
    systemd-1 /proc/sys/fs/binfmt_misc autofs
rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
    mqueue /dev/mqueue mqueue rw,relatime 0 0
    debugfs /sys/kernel/debug debugfs rw,relatime 0 0
    fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
    configfs /sys/kernel/config configfs rw,relatime 0 0
    tmpfs /var/log tmpfs rw,nosuid,noatime,size=131072k,mode=777 0 0
    tmpfs /var/tmp tmpfs rw,nosuid,noatime,size=16384k 0 0
    tmpfs /tmp tmpfs rw,nosuid,noatime,size=131072k 0 0
    tmpfs /root tmpfs rw,relatime 0 0

  I don't know what mechanism causes this? It presumably is why my old
  way of doing this once worked.

Britton

Any help modifying the above recipe or pointer to how to do it these
days would be appreciated.

I just declare the root file system as read-only in fstab and create
some
ramdisks for stuff that needs to be writable.

LABEL=rootfs / ext4 ro,noatime,errors=remount-ro 0 1
debugfs /sys/kernel/debug debugfs defaults 0 0
tmpfs /var/log tmpfs defaults,noatime,nosuid,mode=0777,size=128M 0
0
tmpfs /var/tmp tmpfs defaults,noatime,nosuid,size=16M 0 0
tmpfs /tmp tmpfs defaults,noatime,nosuid,size=128M 0 0

Service generic-board-startup (a.k.a.
/opt/scripts/boot/generic-startup.sh)
must run once with a writable root to set up a bunch of stuff (on later
boots it complains, but it doesn’t seem to matter). After that you’re
stuck
hacking all the software which expects to be able to write to disk. For
a
working network, “/etc/resolv.conf” needs symlinking to a writable
location.
Many services expect to persistently store their stuff under “/var/lib”.

I’ve used tmpfiles.d to create required symlinks, e.g.

$ cat /etc/tmpfiles.d/10-ro-symlinks.conf
L+ /etc/resolv.conf - - - - /var/run/symlinks/etc/resolv.conf
L+ /etc/ppp/resolv.conf - - - - /var/run/symlinks/etc/ppp/resolv.conf

Hmm having tried this I don’t get it. This fragment seem to be assuming
that:

  • the dirs /var/run/symlinks/etc etc. are created somehow, maybe
    with a d line in the same file?

  • the link target files /var/run/symlinks/etc/resolv.conf exist,
    I guess they can just be empty as connection manager is going to fix
    them up for me anyway?

Thanks,
Britton

Right. I wanted to emphasize only the important lines, but prerequisites are
important too :slight_smile:

Here’s the full file

#Type Path Mode UID GID Age Argument
d /var/run/symlinks 0755 - - -
d /var/run/symlinks/etc 0755 - - -
d /var/run/symlinks/etc/ppp 0755 - - -
f /var/run/symlinks/etc/resolv.conf 0755 - - -
f /var/run/symlinks/etc/ppp/resolv.conf 0755 - - -

NB! Connman might override this by linking

“/usr/lib/tmpfiles.d/connman_resolvconf.conf” to “/run/connman/resolv.conf”
L+ /etc/resolv.conf - - - - /var/run/symlinks/etc/resolv.conf
L+ /etc/ppp/resolv.conf - - - - /var/run/symlinks/etc/ppp/resolv.conf

Ok this work, thanks so much. There are still a couple things I don’t
get though:

  • I notice in your original replay you don’t have a tmpfs for /run
    (which is sym linked to from /var/run). So I wonder how the dirs
    and files you create from /etc/tmpfiles.d/10-ro-symlinks.conf are
    getting created, since I would think they would be on the (read-only)
    root file system. Is there something missing from the fstab you
    originally showed? I included this line as well:

tmpfs /run tmpfs defaults,noatime,nosuid,size=16M 0 0

I haven’t gone back and tested without it, so maybe it works as you
show and I just don’t understand how.

The Debian images from Robert already have that set up. “/run” is a tmpfs and “/var/run” is symlink pointing to it.

  • I end up with a rw tmpfs for /root (root home dir), as shown by
    cat /etc/mtab (last line):

root@bbone217slave:/etc# cat /etc/mtab
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=10240k,nr_inodes=55575,mode=755 0 0
devpts /dev/pts devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noatime,size=16384k,mode=755 0 0
/dev/mmcblk1p1 / ext4 ro,noatime,errors=remount-ro,data=ordered 0 0
securityfs /sys/kernel/security securityfs
rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup
rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/devices cgroup
rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/cpuset cgroup
rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/perf_event cgroup
rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/freezer cgroup
rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/memory cgroup
rw,nosuid,nodev,noexec,relatime,memory 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs
rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
tmpfs /var/log tmpfs rw,nosuid,noatime,size=131072k,mode=777 0 0
tmpfs /var/tmp tmpfs rw,nosuid,noatime,size=16384k 0 0
tmpfs /tmp tmpfs rw,nosuid,noatime,size=131072k 0 0
tmpfs /root tmpfs rw,relatime 0 0

I don’t know what mechanism causes this? It presumably is why my old
way of doing this once worked.

No idea. I don’t have that last entry for /root.

>>
>> >>
>> >> Any help modifying the above recipe or pointer to how to do it these
>> >>
>> >> days would be appreciated.
>> >
>> >
>> > I just declare the root file system as read-only in fstab and create
>> >
>> > some
>> > ramdisks for stuff that needs to be writable.
>> >
>> > LABEL=rootfs / ext4 ro,noatime,errors=remount-ro 0 1
>> > debugfs /sys/kernel/debug debugfs defaults 0 0
>> > tmpfs /var/log tmpfs defaults,noatime,nosuid,mode=0777,size=128M
0
>> > 0
>> > tmpfs /var/tmp tmpfs defaults,noatime,nosuid,size=16M 0 0
>> > tmpfs /tmp tmpfs defaults,noatime,nosuid,size=128M 0 0
>> >
>> > Service generic-board-startup (a.k.a.
>> > /opt/scripts/boot/generic-startup.sh)
>> > must run once with a writable root to set up a bunch of stuff (on
later
>> > boots it complains, but it doesn't seem to matter). After that you're
>> >
>> > stuck
>> > hacking all the software which expects to be able to write to disk.
For
>> > a
>> > working network, "/etc/resolv.conf" needs symlinking to a writable
>> > location.
>> > Many services expect to persistently store their stuff under
"/var/lib".
>> >
>> > I've used tmpfiles.d to create required symlinks, e.g.
>> >
>> > $ cat /etc/tmpfiles.d/10-ro-symlinks.conf
>> > L+ /etc/resolv.conf - - - - /var/run/symlinks/etc/resolv.conf
>> > L+ /etc/ppp/resolv.conf - - - - /var/run/symlinks/etc/ppp/resolv.conf
>> >
>>
>> Hmm having tried this I don't get it. This fragment seem to be
assuming
>> that:
>>
>> * the dirs /var/run/symlinks/etc etc. are created somehow, maybe
>> with a d line in the same file?
>>
>> * the link target files /var/run/symlinks/etc/resolv.conf exist,
>> I guess they can just be empty as connection manager is going to fix
>> them up for me anyway?
>>
>> Thanks,
>> Britton
>
>
> Right. I wanted to emphasize only the important lines, but prerequisites
>
are
> important too :slight_smile:
>
> Here's the full file
>
> #Type Path Mode UID GID Age Argument
> d /var/run/symlinks 0755 - - -
> d /var/run/symlinks/etc 0755 - - -
> d /var/run/symlinks/etc/ppp 0755 - - -
> f /var/run/symlinks/etc/resolv.conf 0755 - - -
> f /var/run/symlinks/etc/ppp/resolv.conf 0755 - - -
>
> # NB! Connman might override this by linking
> "/usr/lib/tmpfiles.d/connman_resolvconf.conf" to
"/run/connman/resolv.conf"
> L+ /etc/resolv.conf - - - - /var/run/symlinks/etc/resolv.conf
> L+ /etc/ppp/resolv.conf - - - - /var/run/symlinks/etc/ppp/resolv.conf

Ok this work, thanks so much. There are still a couple things I don't
get though:

* I notice in your original replay you don't have a tmpfs for /run
  (which is sym linked to from /var/run). So I wonder how the dirs
  and files you create from /etc/tmpfiles.d/10-ro-symlinks.conf are
  getting created, since I would think they would be on the (read-only)
  root file system. Is there something missing from the fstab you
  originally showed? I included this line as well:

     tmpfs /run tmpfs defaults,noatime,nosuid,size=16M 0 0

  I haven't gone back and tested without it, so maybe it works as you
  show and I just don't understand how.

The Debian images from Robert already have that set up. "/run" is a tmpfs
and "/var/run" is symlink pointing to it.

Ok. For the record it looks like this happens due to configuration in
/etc/default/tmpfs, which configures temp file systems that have to be
available early in boot. There's some (disabled) stuff in there to configure
/tmp as tmpfs as well, though doing it from fstab as you originally describe
seems to work fine.

* I end up with a rw tmpfs for /root (root home dir), as shown by
  cat /etc/mtab (last line):

    root@bbone217slave:/etc# cat /etc/mtab
    sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
    proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
    udev /dev devtmpfs rw,relatime,size=10240k,nr_inodes=55575,mode=755 0

0
    devpts /dev/pts devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
    tmpfs /run tmpfs rw,nosuid,noatime,size=16384k,mode=755 0 0
    /dev/mmcblk1p1 / ext4 ro,noatime,errors=remount-ro,data=ordered 0 0
    securityfs /sys/kernel/security securityfs
rw,nosuid,nodev,noexec,relatime 0 0
    tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
    tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
    tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
    cgroup /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd

0 0
    cgroup /sys/fs/cgroup/blkio cgroup
rw,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /sys/fs/cgroup/net_cls,net_prio cgroup
rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /sys/fs/cgroup/devices cgroup
rw,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /sys/fs/cgroup/cpuset cgroup
rw,nosuid,nodev,noexec,relatime,cpuset 0 0
    cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /sys/fs/cgroup/perf_event cgroup
rw,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids

0 0
    cgroup /sys/fs/cgroup/freezer cgroup
rw,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /sys/fs/cgroup/memory cgroup
rw,nosuid,nodev,noexec,relatime,memory 0 0
    systemd-1 /proc/sys/fs/binfmt_misc autofs
rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
    mqueue /dev/mqueue mqueue rw,relatime 0 0
    debugfs /sys/kernel/debug debugfs rw,relatime 0 0
    fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
    configfs /sys/kernel/config configfs rw,relatime 0 0
    tmpfs /var/log tmpfs rw,nosuid,noatime,size=131072k,mode=777 0 0
    tmpfs /var/tmp tmpfs rw,nosuid,noatime,size=16384k 0 0
    tmpfs /tmp tmpfs rw,nosuid,noatime,size=131072k 0 0
    tmpfs /root tmpfs rw,relatime 0 0

  I don't know what mechanism causes this? It presumably is why my old
  way of doing this once worked.

No idea. I don't have that last entry for /root.

Ah my bad, I have a service that explicitly does it. It's been so long
since I touched this thing.

Britton