Hi,
I’m examining a beaglebone black failure.
Brief background:
- This beaglebone black (rev A6A) has been running for roughly two months
- The beaglebone was powered by a 5V/1A power supply through the barrel connector
- CPU usage was around 20% usually throughout the time period
- Ethernet was connected and sending data periodically
Yesterday I started investigating intermittent ethernet failures on this board (failed to respond to ping; may be unrelated to this problem). I could ping the board, but SSH would fail with a “connection reset by peer” error, immediately after I tried to initiate an SSH session. I tried to power cycle it once. After the power cycle I was not able to ping it at all; not via ethernet over the network, not via ethernet directly connected, and not via the ftdi USB-ethernet. In fact, the USB connection did not even enumerate on my laptop.
The heartbeat LED has always functioned throughout this failure. I see some dim flickering on the activity light.
To diagnose, I first tried connecting the micro-hdmi port to a monitor; nothing showed up on the screen (seems like there was a signal, just filled with black).
Next, I made a serial cable and watched the output on boot:
parallels@ubuntu:~$ sudo picocom /dev/ttyUSB0 -b 115200 -l
picocom v1.7
port is : /dev/ttyUSB0
flowcontrol : none
baudrate is : 115200
parity is : none
databits are : 8
escape is : C-a
local echo is : no
noinit is : no
noreset is : no
nolock is : yes
send_cmd is : sz -vv
receive_cmd is : rz -vv
imap is :
omap is :
emap is : crcrlf,delbs,
Terminal ready
ot SPL 2015.1-g15e7184 (J - 09:01:57)
2015.07-rc2-0e7184 (Jun 0901:57 -050nkins-github_Bootloader-Builder-163
Watchdog enabled
I2y
DRAM: 512 MiBMMCOMAP SD/MMC: 0/MMC: 1
Using default e Validating fiMAC
psw
0 gpvalue is 1
Card did not respondelect!
Card did not respond to vol0
gpio: pin 55 (gpio 5n 54 (gpio s 0
mmc1(part 0) i device
gpio: pin 5h for: /uEnChe: /boot.scrCer Cng for: /boot…
go: pin 55 (gpio 55) va5 bytes read in 28 .5 KiB/sd environment from /boot/uEnv.tx if uname_r is boot/uEnv.txt…
o: pin 56 (gpie is 1
Runninboot …
loaboot/vmlinuz-3-r67 …
7184 bytes read (16.2 MiB/s)loa/boot/dtbs/3.17/am335x-bone …
8161 bytes read in 94 ms (905.3 KiB/loa /boot/init43-ti-r67 …3634s (15.4 MiB/sdebug: [console=ttyO0,1151a87-5c24-40070ab2cfd4d e=ext4 rootcoherent_po …
debug: [bootz 0x820000000:3774e9 0x…
Kernel image @ 0x82000000 [ 0x000000 - 0x6cd030 ]Flattened Device Tree blob at 88000000
ob at 0x8800 Loading Ramdi88000, end 8O
Lo Tre 8fc87478
[ 0.000420] WARNING: Your ‘console=ttyO0’ has been replaced by ‘ttyS0’
[ 0.000428] This ensures that you still see kernel messages. Please
[ 0.000436] update your kernel commandline.
[ 2.815620] musb-hdrc musb-hdrc.0.auto: Failed to request rx1.
[ 2.821584] musb-hdrc musb-hdrc.0.auto: musb_init_controller failed with status -517
[ 2.895506] musb-hdrc musb-hdrc.1.auto: Failed to request rx1.
[ 2.901456] musb-hdrc musb-hdrc.1.auto: musb_init_controller failed with status -517
[ 3.052411] cpu cpu0: cpu0 clock notifier not ready, retry
[ 3.061824] Error: Driver ‘tfp410’ is already registered, aborting…
[ 3.070128] slave hdmi: could not get i2c
[ 3.472926] musb-hdrc musb-hdrc.1.auto: otg: usb_otg_kick_fsm: invalid host/gadget device
[ 3.739126] usb 1-1: device v413c p2003 is not supported
[ 7.718918] mmcblk0: timed out sending r/w cmd command, card status 0xe00
[ 7.725799] end_request: I/O error, dev mmcblk0, sector 421752
[ 7.731667] Buffer I/O error on device mmcblk0p1, logical block 52463
[ 7.738232] EXT4-fs (mmcblk0p1): error loading journal
mount: mounting /dev/disk/by-uuid/31351a87-5c24-4001-a9b2-5570ab2cfd4d on /root failed: Invalid argument
mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn’t have requested /sbin/init.
No init found. Try passing init= bootarg.BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash)
Enter ‘help’ for a list of built-in commands.
(initramfs) help
Built-in commands:. : [ alias break cd chdir command continue echo eval exec exit
export false getopts hash help let local printf pwd read readonly
return set shift test times trap true type ulimit umask unalias
unset wait [ [[ ash awk basename blockdev cat chmod chroot chvt
clear cmp cp cut deallocvt df dnsdomainname du dumpkmap echo
egrep env expr false fbset fdflush fgrep find grep gunzip gzip
hostname ifconfig ip kill ln loadfont loadkmap ls mkdir mkfifo
mknod mkswap mktemp modinfo more mount mv openvt pidof printf
ps pwd readlink reset rm rmdir sed seq setkeycodes sh sleep sort
stat static-sh stty switch_root sync tail tee test touch tr true
tty umount uname uniq wc wget which yes zcat
(initramfs) ls
dev run etc sbin lib sys tmp
root init bin scripts conf proc var
(initramfs) pwd
/
(initramfs) ls /root
(initramfs) ls /dev/disk/by-uuid
31351a87-5c24-4001-a9b2-5570ab2cfd4d
(initramfs) mount /dev/disk/by-uuid/31351a87-5c24-4001-a9b2-5570ab2cfd4d /root
[ 249.406929] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.439949] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.472807] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.531187] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.589584] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.648051] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.655167] end_request: I/O error, dev mmcblk0, sector 2050
[ 249.660882] EXT4-fs (mmcblk0p1): unable to read superblock
[ 249.667745] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.726297] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.784802] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.843252] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.901664] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.960077] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 249.967186] end_request: I/O error, dev mmcblk0, sector 2050
[ 249.972911] EXT4-fs (mmcblk0p1): unable to read superblock
[ 250.005117] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.037979] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.096794] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.129690] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.188119] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.246548] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.253660] end_request: I/O error, dev mmcblk0, sector 2050
[ 250.259377] EXT4-fs (mmcblk0p1): unable to read superblock
[ 250.266014] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.324543] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.357346] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.390194] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.448686] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.507087] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.514193] end_request: I/O error, dev mmcblk0, sector 2048
[ 250.519911] FAT-fs (mmcblk0p1): unable to read boot sector
[ 250.526431] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.559374] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.617865] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.676351] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.709187] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.716361] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.723468] end_request: I/O error, dev mmcblk0, sector 2048
[ 250.729181] FAT-fs (mmcblk0p1): unable to read boot sector
[ 250.736579] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.769539] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.828103] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.860963] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.893832] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.952220] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 250.959327] end_request: I/O error, dev mmcblk0, sector 2048
[ 251.017583] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.076086] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.134653] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.193064] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.251450] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.309842] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.316950] end_request: I/O error, dev mmcblk0, sector 2048
[ 251.322667] F2FS-fs (mmcblk0p1): Unable to read 1th superblock
[ 251.380094] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.412914] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.471302] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.529697] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.588116] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.646510] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.653617] end_request: I/O error, dev mmcblk0, sector 2056
[ 251.659323] F2FS-fs (mmcblk0p1): Unable to read 2th superblock
[ 251.666173] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.724705] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.783163] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.841623] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.900053] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.958477] mmcblk0: timed out sending r/w cmd command, card status 0x400e00
[ 251.965581] end_request: I/O error, dev mmcblk0, sector 2176
[ 251.971279] Buffer I/O error on device mmcblk0p1, logical block 16
mount: mounting /dev/disk/by-uuid/31351a87-5c24-4001-a9b2-5570ab2cfd4d on /root failed: Invalid argument
(initramfs)
Do these mmc error indicate a mmc hardware failure or is it simply a corrupt partition? I have not tried yet re-flashing it in order to leave it in its current state.
Thanks, anyone who has insight to this question,
Chris C