We found a significant boot-time issue in the Ubuntu images used on our BeagleBone Black and PocketBeagle systems, together with a simple fix. On our systems it reduced total boot time substantially.
We have only confirmed this on an older BeagleBoard Ubuntu image, so this may already be corrected in newer releases. Posting it here in case it helps others running affected images.
Short version
Two udev rules from bb-customizations are applied to every device added during boot instead of only to the I2C and IIO subsystems they are intended for. Because copies of these rules are also included in the initramfs, they delay both the initial root-filesystem mount and normal kernel/device initialisation after switch_root.
The affected files are:
/etc/udev/rules.d/80-i2c-noroot.rules
/etc/udev/rules.d/87-iio-noroot.rules
Both contain a rule using:
DRIVER=="*", ACTION=="add"
The commands executed by these rules recursively change group ownership and permissions on the device’s sysfs tree:
chgrp -R gpio ...
chmod -R g=u ...
At boot there are hundreds of device and driver events. On the single-core AM335x this causes the udev queue to become heavily backed up.
Fix
Restrict each rule to the subsystem it is intended for.
In:
/etc/udev/rules.d/80-i2c-noroot.rules
change:
DRIVER=="*"
to:
SUBSYSTEM=="i2c"
In:
/etc/udev/rules.d/87-iio-noroot.rules
change:
DRIVER=="*"
to:
SUBSYSTEM=="iio"
Then rebuild the initramfs:
sudo update-initramfs -u -k $(uname -r)
Rebuilding the initramfs is essential. The initramfs contains its own copies of the udev rules, and those execute before the real root filesystem is mounted. Editing only /etc/udev/rules.d fixes the second stage of boot but leaves the initial delay intact.
The change does not remove the intended functionality: the gpio group retains non-root access to the I2C and IIO devices. It simply prevents unrelated devices from executing these rules.
Quick check
To see whether an installed image contains these broad rules:
grep -n 'DRIVER=="\*"' /etc/udev/rules.d/*.rules
To check whether the rules are also present in the initramfs:
lsinitramfs /boot/initrd.img-$(uname -r) | grep noroot
We confirmed the problem and fix on both a BeagleBone Black and a PocketBeagle. Our tested image was the rcn-ee.net Ubuntu Console Image 2020-03-12, Ubuntu 18.04.6 with kernel 4.19.94-ti-r36. We have not yet confirmed whether current BeagleBoard Ubuntu images or newer versions of bb-customizations still contain these rules.