fsck fails to start on writable partition at boot/startup

GoogleGroups
1

Hi,

I’m hoping that someone has come across this problem before and can point me in the right direction.

I’m trying to troubleshoot a BeagleBone Black with Debian 8 that appears to have a filesystem corruption. The system has two partitions, a read-only rootfs partition and a writable partition for essentially everything else. When the system boots, U-Boot completes and hands control to the kernel, which runs an fsck on the rootfs successfully, but then fails to run an fsck on the writable parition. At that point the startup process appears to simply hang. I cannot seem to break to a console prompt (or get to a login prompt, obviously).

The best hypothesis I have so far is that some sort of power failure caused a corruption, but I’d like to see if I can examine the “footprint” at all. I’ve never seen a corruption where fsck can’t be run at all. Usually fsck can be run and the corruption can be examined and hopefully repaired.

My question is, why can’t fsck be run on the partition at all? Can I somehow break to the console prompt when the startup process hangs up? Earlier in the process, I can interrupt U-Boot and run “mmc”, “part”, and “ls” types of U-Boot commands to look at the partition in question - at that level things appear to look OK. But obviously it can’t tell why fsck won’t run on the partition.

Could there be anything else going on that I’m not thinking of?

Below is a log of the boot/startup process. Any light anyone can shed would be very helpful.

Thanks for any help,
Dave

2

Hi Dave,

Have you tried booting using an initial ramdisk and using the fsck? I am guessing that your filesystem is in the eMMC.

Thanks,
Gautam.

3

Hi Gautam,

Thanks for the reply! I actually had the same idea last night, and did manage to boot the board using an image on an SD card, and was able to run the fsck from there against the bad partition on the eMMC, and saw the details of the corruption. I was able to repair the partition and ultimately reflashed the board. So thanks for the reply and the confirmation!

But, what I’m still baffled about is: Why the fsck couldn’t run as part of the kernel startup when the system was booted normally? I assume the partition hasn’t been mounted at that point yet, so why would the fsck fail to start? It’s really just more of an understanding-type of question.

Anyway, we 're now looking at ways to prevent sudden or unexpected power downs from potentially effecting such behavior/corruptions. Found this reference which looks pretty helpful: https://www.embeddedarm.com/about/resource/preventing-filesystem-corruption-in-embedded-linux … and the BBB has power down signalling (section 5.10 of the ref manual) that we take advantage of as well.

Thanks again!

Dave

4

Hi Dave,

Although I have seen a few corrupt file systems, I haven’t stumbled on a case where fsck fails to run. With vanilla config fsck usually is started on boot and, in case of errors, patiently waits for you to manually confirm each fix by pressing “y”. This can be worked around by adding to the kernel command line “fsck.mode=force fsck.repair=yes”.

What’s in your /etc/fstab and /boot/uEnv.txt?

5

Hi Dave,

Could you please let me know what the corruption was once you ran fsck from the SD card? I am interested to the see what the corruption was.

In case of fsck failure on startup did you lose your tty terminal once the fsck started? Due to this was it waiting for some confirmation or had it just started and you lost the output messages?

Is this for something which will go into an actual product and is the requirement to have a robust FS during power failure?

Thanks,
Gautam.

6

Hi Tarmo,

Thanks for the reply! And we are considering doing something like you’re suggesting - adding fsck options to the kernel to do automatic repairs during startup when possible. Thanks for that.

/etc/fstab just has a few entries. Two partitions and then directories bound within the second writable partition (/var):

/etc/fstab: static file system information.

7

Hi Gautam,

As you requested, here’s the output from running fsck on the writable partition after booting/running from the SD card. Nothing that truly unusual I don’t think?

root@bbb-dev:~# fsck -f -v /dev/mmcblk1p2
fsck from util-linux 2.25.2
e2fsck 1.43 (17-May-2016)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Unattached inode 45
Connect to /lost+found? no
Unattached inode 81
Connect to /lost+found? no
Pass 5: Checking group summary information
Block bitmap differences: +(96768–97021)
Fix? no
Free blocks count wrong for group #2 (1174, counted=1428).
Fix? no
Free blocks count wrong (1091090, counted=1091344).
Fix? no

rootfs_var: ********** WARNING: Filesystem still has errors **********

5413 inodes used (1.43%, out of 378256)
20 non-contiguous files (0.4%)
2 non-contiguous directories (0.0%)

of inodes with ind/dind/tind blocks: 0/0/0

Extent depth histogram: 5389/10
419310 blocks used (27.76%, out of 1510400)
0 bad blocks
1 large file

3987 regular files
1408 directories
0 character device files
0 block device files
0 fifos
0 links
8 symbolic links (5 fast symbolic links)
1 socket

8

Hi Dave,

As you said not unusual but I am not sure. As in the earlier post it is better not to have fsck ask for confirmation. I feel there is some problem in the scripts which might have failed and not in the fsck. What is the filesystem type here?

A battery backup maybe in the form of a super capacitor would be something. Just curious as what sort of power event caused so much of problems? Is it just a normal abrupt power failure or is it some kind of intermittent power failure?

Thanks,
Gautam.

9

Hi Gautam,

The filesystem is ext4; the O/S is debian 8/jessie, the image is the “officially released” BBB image from last spring (bone-debian-8.7-iot-armhf-2017-03-19-4gb.img.xz). To date we have not modified any of the kernel startup scripts, save for setting a serial port baud rate in /etc/rc.local.

I wish I could tell you the power event that caused this to happen - it was a unit in field test and folks there have been pretty much just killing the power to it at various times. We had discussed putting the circuitry in to allow a more graceful shutdown, but are only able to get to it now. That said, power has been shutdown abruptly (power simply removed) literally thousands of times on dozens of BBBs, but we’ve only seen a couple corruptions. (Separately, we’ve also seen “bad block” corruptions occasionally, presumably from wear (many, many writes over time).)

Dave

10

Hi Dave,

I too have a similar problem but not have worked directly on the problem. This is in medical devices where the instruments power is pulled off abruptly without shutting down. Also the wear and tear of the eMMC and detection of errors etc. The problems become much more serious and I cannot boot the instrument if the complete system is not working at 100%. Also the problem of run time monitoring to detect any problems in the system and alert the user or shutdown the system.

I have been thinking of evaluating datalight filesystem solutions for this problem. If your problem is mission critical I am interested in knowing your solutions.

Thanks,
Gautam.

11

On Thu, 7 Dec 2017 19:17:09 -0800 (PST),
mindentropy@gmail.com declaimed the
following:

I too have a similar problem but not have worked directly on the problem.
This is in medical devices where the instruments power is pulled off
abruptly without shutting down. Also the wear and tear of the eMMC and
detection of errors etc. The problems become much more serious and I cannot

  Somehow I don't see a medical device being approved for usage if it is
sensitive to the above.

  I'd suspect the device will need to have an ancillary power management
circuit (super-cap or such to carry through a shutdown cycle with safety
margin) with an intermediate processor monitoring the main power. On
detection of loss of main power, the intermediate would trigger the safe
shutdown of the BBx. The eMMC would only contain the operating firmware
(ideally, it would be a read-only system), any dynamic data (logs, etc.)
should be placed on a field serviceable SD card.

  You'll probably also need a few processes running that just performs
checksums/CRCs of the critical firmware, and in extreme cases, periodic
tests of RAM (tricky to do when the OS can put a program anywhere in RAM).

12

Can’t say it does. Your /var/ file system is writable and gets corrupted - this is expected. fsck failing to run is unexpected.

Looks like systemd has taken over the responsibility of boot-time fsck-ing. A look at the service which fails (/lib/systemd/system/systemd-fsck@.service) tells me that there is some scant documentation available in “man systemd-fsck”. However, it looks like you might have to dig into systemd source to analyze how exactly boot-time fsck-s are triggered and what could go wrong there.

Alternatively, perhaps you’d want to have a quick glance at changelog for systemd, to see if they’ve fixed any bugs in this area since your installed version. My BBB with Debian 8 is running systemd version “230-7~bpo8+2” (dpkg -l systemd) and they’ve made 5 releases since.

13

On Thu, 7 Dec 2017 19:17:09 -0800 (PST),
minde...@gmail.com declaimed the
following:

I too have a similar problem but not have worked directly on the problem.
This is in medical devices where the instruments power is pulled off
abruptly without shutting down. Also the wear and tear of the eMMC and
detection of errors etc. The problems become much more serious and I cannot

Somehow I don’t see a medical device being approved for usage if it is
sensitive to the above.

Yes it will not get approved if it is sensitive to all of these. Most of the regulated industries will not approve.

I’d suspect the device will need to have an ancillary power management
circuit (super-cap or such to carry through a shutdown cycle with safety
margin) with an intermediate processor monitoring the main power. On
detection of loss of main power, the intermediate would trigger the safe
shutdown of the BBx. The eMMC would only contain the operating firmware
(ideally, it would be a read-only system), any dynamic data (logs, etc.)
should be placed on a field serviceable SD card.

I have seen this majority in microcontrollers. Being usually low power the super cap is ideal. There is also good voltage supervisors present to monitor the supply voltage.

You’ll probably also need a few processes running that just performs
checksums/CRCs of the critical firmware, and in extreme cases, periodic
tests of RAM (tricky to do when the OS can put a program anywhere in RAM).

I have seen this being done in implantable medical hardware. Have you done something in similar to this. Would really like to hear your story.

Thanks,
Gautam.

14

On Fri, 8 Dec 2017 10:02:33 -0800 (PST),
mindentropy@gmail.com declaimed the
following:

  <SNIP>

I have seen this majority in microcontrollers. Being usually low power the

super cap is ideal. There is also good voltage supervisors present to
monitor the supply voltage.

  <SNIP>

I have seen this being done in implantable medical hardware. Have you done
something in similar to this. Would really like to hear your story.

  Prior to a lay-off, my last four years were avionics -- fortunately
mostly maintenance work for existing software (since my prior 30 years
experience qualified as "mainframe number crunching", so the entire
embedded/real-time world was a new experience).

  No experience with use of an OS as Linux in such applications -- from
what I could tell, general-purpose OS with real file-systems was only
approved for things like monitoring/logging systems wherein the loss of the
system would have no effect on the operation of the aircraft itself.

  The rest tended to follow two concepts:

A) operation out of Flash memory (similar to what one would find on an
Arduino, TIVA C, and similar boards).

B) operation out of RAM with firmware loaded from Flash memory (closer to
BBx/R-Pi, except the Flash is not a general file system and is read-only)

  Both likely used some sort of RTOS providing the core of tasking and
IPC (and the highest level having fixed time-slice assigned to tasks with
watch-dogs for overrun detection). In both, analysts created memory maps
defining what code/data is placed where in memory.

(A) tended to have just one copy of the firmware/data in the layout;
(B) tended to have two copies allowing for swap-over if the boot-up checks
detected corruption (and reducing the odds of "bricking" when loading
updates into one copy)

  In both, CRC checks were run during booting before passing control to
the real RTOS/application. Data regions tended to have CRC checks performed
periodically during operation (one assignment involved a board with
expanded Flash memory -- where the person responsible for the CRC found he
had to run it in phases as the new data region was too large to be CRCd in
the allowed time-slot). In (B) the CRC was also run before loading the code
into RAM.

  A CRC failure would result in the code going into a "halt" state
(usually implemented as an endless loop) or an attempt to restart the
processor -- while control of the craft went to the second flight computer.

  Power-glitches (ever watch all the lights flicker on a passenger jet
while they swap over from ground generator to engine generators?) trigger
state save, and later checks of RAM contents (the restart logic would check
a timer -- capacitor bleed down -- to determine if it needed to do full
reboot or take a fast start path), if reading all the RAM does not trigger
a CRC error, control is passed to the application at the position saved.
While a glitch was considered ~10 seconds with no power, it seems some
boards could keep RAM stable for hours after loss of power.

15

Thanks a lot for the insight. This is very interesting and something which I have seen in medical implants too. There were a lot of checks with the scheduler and also their queues which I don’t seem to recall. Also as you have said I too have seen/worked mostly on microcontrollers with some type of qualified RTOS for such regulated devices. Also as you have noticed usually they relegated the Application processor with Linux for monitoring.

Although I have not explored it, I feel that the design can be made much better now with the availability of Cortex M4 in AM57x and the i.MX7 series. With this you have the Application processor and the microcontroller in the same die thereby shrinking the board to a huge extent and also making development much more compact and also simplifying the implementation of safety to the system.

The safety checks can be implemented during boot-up can be done using the POST framework in the boot loader. I have yet to explore this deeply and the possibilities are good.

-Gautam.