How to prevent emmc filesystem corruption after power loss

We have an HMI product based on the BeagleBone Black running:

  • uBoot: V2019.04
  • Kernel: 4.19.106-bone49
  • Debian 10.2
  • ext4 filesystem

Several customers have had issues where the device fails to boot after power loss. We have had several of these devices returned to our factory and in connecting to the serial port, see error messages during the check of the root file system (see below for partial boot log). The device can be recovered by reflashing the eMMC from a uSD card.

We do not have the option of assuming a clean shutdown and would prefer not to make hardware modifications to resolve this issue. Any suggestions for steps to take to minimize the risk of filesystem corruption?

[    2.786989]   No soundcards found.
[    2.796516] Freeing unused kernel memory: 2048K
[    2.801567] Run /init as init process
Loading, please wait...
Starting version 241
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... Scanning for Btrfs filesystems
done.
Begin: Will now check root file system ... fsck from util-linux 2.33.1
[/sbin/fsck.ext4 (1) -- /dev/mmcblk1p1] fsck.ext4 -a -C0 /dev/mmcblk1p1
[    9.273137] mmc1: cache flush error -110
[    9.277128] print_req_error: I/O error, dev mmcblk1, sector 0
rootfs: recovering journal
[   14.136598] omap_hsmmc 481d8000.mmc: Card stuck in wrong state! card_busy_detect status: 0xe00
[   14.242612] mmc1: cache flush error -110
[   15.544958] mmc1: tried to HW reset card, got error -110
[   15.550330] mmcblk1: recovery failed!
[   15.554059] print_req_error: I/O error, dev mmcblk1, sector 8192
[   15.560190] Buffer I/O error on dev mmcblk1p1, logical block 0, lost async page write
[   15.568106] Buffer I/O error on dev mmcblk1p1, logical block 1, lost async page write
[   15.581101] omap_hsmmc 481d8000.mmc: error -110 requesting status
[   15.587246] mmcblk1: recovery failed!
[   15.590971] print_req_error: I/O error, dev mmcblk1, sector 11992
[   15.597174] Buffer I/O error on dev mmcblk1p1, logical block 475, lost async page write
[   15.610270] omap_hsmmc 481d8000.mmc: error -110 requesting status
[   15.616408] mmcblk1: recovery failed!
[   15.620121] print_req_error: I/O error, dev mmcblk1, sector 12112
[   15.626327] Buffer I/O error on dev mmcblk1p1, logical block 490, lost async page write
[   15.639397] omap_hsmmc 481d8000.mmc: error -110 requesting status
[   15.645536] mmcblk1: recovery failed!
[   15.649246] print_req_error: I/O error, dev mmcblk1, sector 12152
[   15.655447] Buffer I/O error on dev mmcblk1p1, logical block 495, lost async page write
[   15.663513] Buffer I/O error on dev mmcblk1p1, logical block 496, lost async page write
[   15.676579] omap_hsmmc 481d8000.mmc: error -110 requesting status
[   15.682716] mmcblk1: recovery failed!
[   15.686426] print_req_error: I/O error, dev mmcblk1, sector 20664
[   15.692621] Buffer I/O error on dev mmcblk1p1, logical block 1559, lost async page write
[   15.705775] omap_hsmmc 481d8000.mmc: error -110 requesting status
[   15.711918] mmcblk1: recovery failed!
[   14.136598] omap_hsmmc 481d8000.mmc: Card stuck in wrong state! card_busy_detect status: 0xe00
[   14.242612] mmc1: cache flush error -110
[   15.544958] mmc1: tried to HW reset card, got error -110
[   15.550330] mmcblk1: recovery failed!

Without any more information, looks like you wore out the eMMC…

Regards,

Yep, I second what Robert said… this isn’t merely filesystem corruption, the eMMC is dead, almost certainly due to wear-out.

To investigate how much data is being written to eMMC, here’s a little perl script:

#!/usr/bin/perl

use v5.20;
use warnings qw( FATAL all );

sub slurp {
	my( $path ) = @_;
	open my $fh, '<', $path  or die;
	split ' ', <$fh>
}

my $t = (slurp '/proc/uptime')[0] / 86400;
if( $t >= 0.95 ) {
	$t = sprintf "%.1fd", $t;
} elsif( ($t *= 24) >= 0.95 ) {
	$t = sprintf "%.1fh", $t;
} else {
	$t = sprintf "%.1fm", $t * 60;
}

my $n = (slurp '/sys/block/mmcblk1/stat')[6] / 2;
my $u = 'KiB';
for(qw( MiB GiB TiB )) {
	last  if $n < 999.5;
	$n /= 1024;
	$u = $_;
}
if( $n < 9.95 ) {
	printf "%.2f %s  in  %s\n", $n, $u, $t;
} elsif( $n < 99.5 ) {
	printf "%.1f %s  in  %s\n", $n, $u, $t;
} else {
	printf "%.0f %s  in  %s\n", $n, $u, $t;
}

To help figuring out who’s doing it, replace the following line in /etc/systemd/system.conf:

#DefaultBlockIOAccounting=no

by:

DefaultBlockIOAccounting=yes

and reboot, then use this perl script to show a tree of much how much data has been accounted to which systemd unit:

#!/usr/bin/perl

use v5.20;
use warnings qw( FATAL all );
use File::Basename qw( basename );

chdir '/sys/fs/cgroup/blkio'  or die;

my %dev;

sub check {
	my( $path ) = @_;
	if( open my $fh, '<', "./$path/blkio.io_service_bytes_recursive" ) {
		while( <$fh> ) {
			/^(\d+:\d+) Write (\d+)/ or next;
			my $n = int( $2 / 1024**2 * 100 + 0.5 ) / 100  or next;
			my $dev = $dev{$1} //= basename readlink "/sys/dev/block/$1";
			$dev eq 'mmcblk1'  or next;
			printf "%7.2f MiB  /%s\n", $n, $path;
			last;
		}
	}
	check( $_ )  for grep -d, glob( $path ? "$path/*" : "*" );
}

check '';

It’s not perfect, it seems the kernel can’t always correctly account writes (at least on the kernel versions I’ve tested), and this tree also isn’t too helpful to track how much is being written by short-lived services, but it’s at least a start.

There’s also a way to reconfigure the eMMC into a more reliable mode which should have much greater write-endurance (as well as more robustness to power interruptions), at the cost of sacrificing 50% of the storage capacity. This can be configured using mmc-utils, though the sequence of steps is a bit tricky and needs to be done with great care since it involves one-time-programmable configuration (i.e. no room for mistakes). To ensure this is done correctly, since we’re doing this to our beaglebones in production, I’ve added a custom command to mmc-utils that performs some sanity checks and then the steps required to reconfigure the eMMC: add reliable-slc-configuration command · dutchanddutch/mmc-utils@d550d3e · GitHub
Like the commit message says, beware that using this command causes the eMMC to be erased and permanently lose 50% of its capacity, and you must power-cycle (not merely reboot) the beaglebone after using this command.

1 Like

Thanks for your replies. I will give the perl script a try and see what that tells me. Since the devices are recovered by reflashing - I was thinking this was not an eMMC issue. Could reflashing potentially fix the condition? Also is there a way to keep track of the number of times the journal is used when rebooting?

Sorry for the late reply.

Ehh, that makes no sense, the errors you’re showing in your log indicate the kernel was completely unable to access the eMMC at all, which would also preclude reflashing

By the way, I wonder if the current BBB SW uses the Power-off notification feature of eMMC v.5.1 specification.

According to TI, the AM335x MMC controller supports up to v.4.3 but many features of newer versions are not dependent on the controller HW. They only require a v5.1 eMMC part (such as Kingston EMMC04G-M627) so the question

Seems to be enabled:

debian@BeagleBone:~$ sudo mmc extcsd read /dev/mmcblk1 | grep POWER_OFF
Power off notification [POWER_OFF_LONG_TIME: 0xff]
Power Off Notification [POWER_OFF_NOTIFICATION]: 0x01
debian@BeagleBone:~$ uname -r
5.10.131-ti-r49
debian@BeagleBone:~$ sudo beagle-version | grep eeprom
eeprom:[A335BNLTEIA02119SBB13318]

Looks like this random BBB board was built in 2019…

Regards,

Unless you can have a battery backup solution, a sudden power loss on a standard install will always run the risk of bricking the system. It is not just the files you may be writing to at the time of power loss, but any other parts of the flash the embedded controller in the eMMC may be writing to. It will just depend on how lucky you are.

Assuming you can’t have a battery backup solution, your best bet is to only mount the eMMC read only, and use ram disks where you need to write data. This of course requires some work and you will lose any data on power loss.

If you need to write data, I would still mount the eMMC read only and use an SD card for data. This can be checked at power up and if needed, repaired or formatted, but at least the system should boot.

I would like to say that mounting the system read only is 100% safe from failure, but the eMMC is probably doing a little writing every time it is powered up. There will always be the potential for power loss to cause problems if it happens at just the wrong moment if the firmware in the eMMC can’t cope with this.

Thanks for the clarification, Robert. Is it documented anywhere (besides the source code)?

That’s a very good point to mind. Especially when the eMMC background ops (which is now a mandatory feature in the spec) are enabled. BTW I have battery backup solution implemented in all my boards. Anyway, it’s essential responsibility of the platform software to prepare all the HW for power interruption as appropriate. A standard service from a good system. At least when user has pressed the shutdown button. As for read-only mounting, this adds safety but it’s not convenient when your application occasionally wants to write something.