Beaglebone Black dies occasionally

Hello guys,

I have a very weird behaviour on some of my beaglebone blacks. Heres what I am doing:
I have a Beaglebone Black with an LCD Display (gen4-4DCAPE-50T), a custom Cape board with Can-Transceiver and some other simple IOs. I communicate with some CAN-Devices and interact with my QT application with the device. I built the image from the Beaglebone Debian 10 IoT flasher.

The devices operated fine for a while but at some point, some devices started “dieing”. Meaning they freeze during bootup, the Heartbeat LED doesn’t even blink anymore and I have no way of finding out what is causing the problem. Other devices worked fine for a long time…

If a flash the devices again, they start working for some time, but they eventually die at some point again.

Do you have any ideas on how to investigate in this issue? How could I figure out what is causing this behaviour if I cannot connect to it via USB or SSH. How could I prevent the devices from dieing?

Any help or suggestions will be very much appreciated.

Thank you!

I would monitor the debug port and see what is going on.

  1. Might be a massive memory leak. If that is the case fix your code.
    2.Thermal issue with the board, toss it out and buy a new one. Boards from G&H are doing very well.
  2. Not sure what graphics package you are using but make sure you are cleaning things up properly.
  3. Get a good lab grade power supply

you shouldn’t have to ‘reflash’ to get it working, normally a simple reboot should bring it back up… i know with a lcd cape, the j1 header is hard to get at, can you bend and run the usb-serial adapter connector under the lcd cape to watch what is happening over serial?

i’ve also used un-soldered headers to lift the lcd another 1/4 inch to get more space for the debug header…

Regards,

Hi Robert,

Thanks for the reply.
I have bought an FTDI cable and this is the output on the Debug Pins:

    ```sh
    U-Boot SPL 2019.04-g1da4564 (Aug 31 2021 - 14:32:40 +0000)
    Trying to boot from MMC2
    Loading Environment from EXT4... Card did not respond to voltage select!
    
    
    U-Boot 2019.04-g1da4564 (Aug 31 2021 - 14:32:40 +0000)
    
    CPU  : AM335X-GP rev 2.1
    I2C:   ready
    DRAM:  512 MiB
    No match for driver 'omap_hsmmc'
    No match for driver 'omap_hsmmc'
    Some drivers were not found
    Reset Source: Power-on reset has occurred.
    RTC 32KCLK Source: External.
    MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
    Loading Environment from EXT4... Card did not respond to voltage select!
    Board: BeagleBone Black
    <ethaddr> not set. Validating first E-fuse MAC
    BeagleBone Black:
    BeagleBone Cape EEPROM: no EEPROM at address: 0x54
    BeagleBone Cape EEPROM: no EEPROM at address: 0x55
    BeagleBone Cape EEPROM: no EEPROM at address: 0x56
    BeagleBone Cape EEPROM: no EEPROM at address: 0x57
    Net:   eth0: MII MODE
    cpsw, usb_ether
    Press SPACE to abort autoboot in 0 seconds
    board_name=[A335BNLT] ...
    board_rev=[00C0] ...
    Card did not respond to voltage select!
    Card did not respond to voltage select!
    Card did not respond to voltage select!
    gpio: pin 56 (gpio 56) value is 0
    gpio: pin 55 (gpio 55) value is 0
    gpio: pin 54 (gpio 54) value is 0
    gpio: pin 53 (gpio 53) value is 1
    Card did not respond to voltage select!
    Card did not respond to voltage select!
    switch to partitions #0, OK
    mmc1(part 0) is current device
    Scanning mmc 1:1...
    gpio: pin 56 (gpio 56) value is 0
    gpio: pin 55 (gpio 55) value is 0
    gpio: pin 54 (gpio 54) value is 0
    gpio: pin 53 (gpio 53) value is 1
    switch to partitions #0, OK
    mmc1(part 0) is current device
    gpio: pin 54 (gpio 54) value is 1
    Checking for: /uEnv.txt ...
    Checking for: /boot.scr ...
    Checking for: /boot/boot.scr ...
    Checking for: /boot/uEnv.txt ...
    gpio: pin 55 (gpio 55) value is 1
    2432 bytes read in 12 ms (197.3 KiB/s)
    Loaded environment from /boot/uEnv.txt
    Checking if uname_r is set in /boot/uEnv.txt...
    gpio: pin 56 (gpio 56) value is 1
    Running uname_boot ...
    loading /boot/vmlinuz-4.19.94-ti-r68 ...
    10183168 bytes read in 656 ms (14.8 MiB/s)
    debug: [enable_uboot_overlays=1] ...
    debug: [enable_uboot_cape_universal=1] ...
    debug: [uboot_base_dtb_univ=am335x-boneblack-uboot-univ.dtb] ...
    uboot_overlays: [uboot_base_dtb=am335x-boneblack-uboot-univ.dtb] ...
    uboot_overlays: Switching too: dtb=am335x-boneblack-uboot-univ.dtb ...
    loading /boot/dtbs/4.19.94-ti-r68/am335x-boneblack-uboot-univ.dtb ...
    167492 bytes read in 43 ms (3.7 MiB/s)
    uboot_overlays: [fdt_buffer=0x60000] ...
    uboot_overlays: loading /boot/dtbs/4.19.94-ti-r68/overlays/BB-ADC-00A0.dtbo ...
    645 bytes read in 23 ms (27.3 KiB/s)
    uboot_overlays: loading /lib/firmware/BB-BONE-4D7R-01-00A1.dtbo ...
    5648 bytes read in 251 ms (21.5 KiB/s)
    uboot_overlays: loading /lib/firmware/BB-CAN0-00A0.dtbo ...
    1013 bytes read in 251 ms (3.9 KiB/s)
    uboot_overlays: loading /boot/dtbs/4.19.94-ti-r68/overlays/BB-BONE-eMMC1-01-00A0.dtbo ...
    1605 bytes read in 30 ms (51.8 KiB/s)
    uboot_overlays: loading /boot/dtbs/4.19.94-ti-r68/overlays/BB-HDMI-TDA998x-00A0.dtbo ...
    5321 bytes read in 34 ms (152.3 KiB/s)
    uboot_overlays: loading /boot/dtbs/4.19.94-ti-r68/overlays/AM335X-PRU-RPROC-4-19-TI-00A0.dtbo ...
    3627 bytes read in 23 ms (153.3 KiB/s)
    debug: [console=ttyS0,115200n8 bone_capemgr.uboot_capemgr_enabled=1 root=/dev/mmcblk1p1 ro rootfstype=ext4 rootwait coherent_pool=1M net.ifnames=0 lpj=1990656 rng_core.default_quality=100 quiet] ...
    debug: [bootz 0x82000000 - 88000000] ...
    ## Flattened Device Tree blob at 88000000
       Booting using the fdt blob at 0x88000000
       Loading Device Tree to 8ff72000, end 8fffffff ... OK
    
    Starting kernel ...
    
    [    0.002264] timer_probe: no matching timers found
    [    0.200754] l4_wkup_cm:clk:0010:0: failed to disable
    [    0.665265] OF: graph: no port node found in /fb
    [    0.670491] OF: graph: no port node found in /fb
    [    0.676130] OF: graph: no port node found in /fb
    [    0.988209] omap_voltage_late_init: Voltage driver support not added
    [    1.019970] pinctrl-single 44e10800.pinmux: pin PIN40 already requested by panel; cannot claim for 0-0070
    [    1.029695] pinctrl-single 44e10800.pinmux: pin-40 (0-0070) status -22
    [    1.036300] pinctrl-single 44e10800.pinmux: could not request pin 40 (PIN40) from group nxp_hdmi_bonelt_pins  on device pinctrl-single
    [    1.048475] tda998x 0-0070: Error applying setting, reverse things back
    [    1.133014] EXT4-fs (mmcblk1p1): VFS: Found ext4 filesystem with invalid superblock checksum.  Run e2fsck?
    [    1.143183] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,1)
    [    1.151671] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.94-ti-r68 #1xross
    [    1.158748] Hardware name: Generic AM33XX (Flattened Device Tree)
    [    1.164912] [<c0113e18>] (unwind_backtrace) from [<c010e388>] (show_stack+0x20/0x24)
    [    1.172701] [<c010e388>] (show_stack) from [<c0d28a3c>] (dump_stack+0x8c/0xa0)
    [    1.179965] [<c0d28a3c>] (dump_stack) from [<c013c710>] (panic+0x10c/0x278)
    [    1.186970] [<c013c710>] (panic) from [<c1401910>] (mount_block_root+0x1e4/0x290)
    [    1.194491] [<c1401910>] (mount_block_root) from [<c1401ae8>] (mount_root+0x12c/0x148)
    [    1.202447] [<c1401ae8>] (mount_root) from [<c1401c64>] (prepare_namespace+0x160/0x1a8)
    [    1.210490] [<c1401c64>] (prepare_namespace) from [<c140142c>] (kernel_init_freeable+0x3f8/0x40c)
    [    1.219412] [<c140142c>] (kernel_init_freeable) from [<c0d3df64>] (kernel_init+0x18/0x128)
    [    1.227719] [<c0d3df64>] (kernel_init) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
    [    1.235321] Exception stack(0xdc149fb0 to 0xdc149ff8)
    [    1.240397] 9fa0:                                     00000000 00000000 00000000 00000000
    [    1.248614] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [    1.256829] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
    [    1.263496] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,1) ]---
    ```

Have you ever seen this behaviour and would you know how we can prevent this from happening?

Thank you!

The root file system is corrupt. As it suggests you need to run a check.
Did the board have it’s power removed while running ?

if you stick a live image (not a flasher one) onto an sd card and boot that, once booted you will be able to run e2fsck on the bad partition. If badly corrupted then you will need to reflash.

Thanks for the quick reply.

I was not there when it happened but it is likely that the device was disconnected from power while running.
Would you know any recommendations in order to avoid or decrease the risk of corruption of the file system when power is disconnected?

Thank you!

There is no quick fix for that unfortunately.

Run the board from a small UPS if there is one.

Failing that the best way is to run from a read only system, but that is not always practical and it takes some setting up. If you never write to the media then it should not get corrupted.

You can use tmpfs to create some writable directories that are obviously lost when powered down, but you are limited in RAM. If running from eMMC, you can make that read only and then use an sd card for non volatile storage. But if doing that, you will obviously need to check that it mounts correctly and if not run a disk check on it.

I see you loading a lcd screen along with the hdmi, which are you using? LCD or hdmi, if LCD, i’d really disable the hdmi… https://elinux.org/Beagleboard:BeagleBoneBlack_Debian#U-Boot_Disable_on-board_devices

Regards,

Thanks for the suggestion. I am using an LCD screen. I will disable HDMI.

How much free ram do you have in your system? Depending on that, i usually recommend users look at ‘overlayroot’ it’s usually pre-installed in our default images… It sets the main partition as ro, and does an overlayfs on top for systemd/xorg/etc to stay happy… However overlayroot runs in ram, so…

To configure it, just add overlayroot in the cmdline/bootarg option at the end of /boot/uEnv.txt

Regards,

Thanks for the sugesstions,

I am running from eMMC so I think I will go for the read only FS with an SD card for storing settings and user data.