Black kernel panic with production image

Hi All,
Got my Bone Black today and all seemed well. It booted as expected and I was able to log in, create a new user and so on. I was unable to SSH and so I thought I would make sure that I had the latest production code before going any further with testing the board.

Downloaded BBB-eMMC-flasher-2013.04.13-DDR3-400MHZ.img.xz from Circuitco's website. I don't have access to a Windows machine, so I use xz -dkc BBB....img.xz > /dev/sdc to decompress and save the image to my uSD. That seemed to go well as I can mount on my Ubuntu machine and see all the files without any problem.

I inserted the card on the black, held the boot button down and applied power. It booted normally all the way to the user prompt, and I think started the eMMC flash program. A few seconds later I got a kernel panic message. The bizzare part is now it is not reliable at all when I try to reboot. It would simply hang at different points in the boot process. If I leave the board un-powered for several minutes, it will at least boot to the prompt and start eMMC flash. Sometimes I won't get any kind of message; it will hang with LEDs on.

I tried version 2013.04.12 as well with the same result.

It is beginning to smell like a hardware issue with the board. Anyone else seen this behavior?

The boot sequence can be found here Beagleboard black kernel panic - Pastebin.com
Here is a snippet of the boot sequence:
U-Boot 2013.01.01-00018-gff666a3 (Mar 18 2013 - 09:16:52)

I2C: ready
DRAM: 512 MiB
WARNING: Caches not enabled
NAND: No NAND device found!!!
0 MiB
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
*** Warning - readenv() failed, using default environment

...
systemd-fsck[83]: BBB-eMMC-flasher: clean, 12695/218592 files, 185058/873534 blocks

.---O---.

      > .-. o o
  > >-----.-----.-----.| | .----..-----.-----.
      > > __ | ---'| '--.| .-'| | |
  > > > > >--- || --'| | | ' | | | |

'---'---'--'--'--. |-----''----''--' '-----'-'-'-'
                 -' |
                 '---'

The Angstrom Distribution beaglebone ttyO0

Angstrom v2012.12 - Kernel 3.8.3

beaglebone login: [ 16.408241] libphy: PHY 4a101000.mdio:01 not found
[ 16.413328] net eth0: phy 4a101000.mdio:01 not found on slave 1
...
root@beaglebone:~# [ 38.623132] Unable to handle kernel NULL pointer dereference at virtual address 00000114
[ 38.631617] pgd = c0004000
[ 38.634444] [00000114] *pgd=00000000
[ 38.638192] Internal error: Oops: 17 [#1] SMP ARM
[ 38.643109] Modules linked in: ip_tables x_tables rfcomm ircomm_tty ircomm irda ipv6 hidp bluetooth rfkill autofs4
[ 38.653999] CPU: 0 Not tainted (3.8.3 #1)

Thanks
Mark

The image you tried to flash was already on the board.

Sounds to me like the process was not completed. Make sure you have a DC supply and try the flash process again. It takes 45 minutes to complete the process. If the process is successful, You should boot from eMMC and see a Kernel 3.9.8. From the printout you seem to be running the kernel form the SD card. You can remove the SD card and power cycle the board. If it does not boot, then the eMMC was not flashed correctly.

Gerald

Gerald,
Sorry I wasn’t more specific. Versions (2013.04.13 and 2013.04.12) booted from the SD card, but both hung up with either a kernel panic or no message at all. The eMMC was cleared. The LED’s flashed (heartbeat) but only briefly and would be solid on or off after each attempt.

I am using a good 5V supply and I did not notice any voltage dips or noise using my trusty scope during boot.

I did see an earlier post speculating that using an old SD card was shutting down the power management, although that is not what I am seeing as I still have some LED’s on. It is strange that it does seem related somehow to the SD.

I guess I will try a version that does not attempt to do the flash.

Thanks
Mark

The image you tried to flash was already on the board.

Sounds to me like the process was not completed. Make sure you have a DC supply and try the flash process again. It takes 45 minutes to complete the process. If the process is successful, You should boot from eMMC and see a Kernel 3.9.8.

Is that a typo, or where is the 3.9.x kernel? I just did ./oebb.sh update and ended up with 3.8.10 update from 3.8.8?

Not sure about 3.9. We are actually discussing moving on to 3.11 or 3.12 to try and catch up to the crowd…

Gerald

FWIW, Robert Nelson’s 2013-04-26 release boots just fine. (Nice work Robert!)

I did try a better grade SD card with the Circuitco’s production code (2013.04.13) and I am still seeing the same issue with hanging at various points in the boot process.

Mark

A little more info…
I tried the 2013.05.02 version from Koen and it has the same problem on my board. I do see the LED behaving like I would expect when flashing the nand, but it always either stops with a kernel panic or does not display any message at all. The bizarre part is that I cannot get a good boot after that unless I leave power off for a short while (5 min or so). I tried a little freeze spray and it does allow the board to boot back up again, but stops shortly thereafter. I wondering if there is something marginal on my board so that during the nand flash it causes Linux to crash.

I did build Angstrom version 3.8.11 that runs from the SD card just fine.

Any thoughts?

Thanks
Mark

We believe the DDR timings may be are wrong in the flasher image. We are trying to get that verified. Another possibility is that you can plug in a serial cable and see if that helps. There is a mod that was done on the board to fix an issue we had. If the mod is not done correctly, then it could also cause a similar result.

It is not a freeze spray issue…

Gerald

Thanks Gerald.
I do have a FTDI serial cable and it did not seem to make a difference whether it was attached or not. Is there a way I can verify if the mod was applied correctly?

Mark

Not really. It is invisible. If it were this issue, plugging in the FTDI cable would have fixed it.

This is unofficial and experimental, but give this a try

http://dominion.thruhere.net/koen/angstrom/beaglebone/BBB-eMMC-flasher-2013.05.03.img.xz

We are in the process of testing it for a similar issue.

Let us know if it works!

Gerald

Sadly no. However it is different from what I was seeing before. I am not getting a consistent kernel panic in cape manager (or just after), and it seemed to have fixed the issue of hanging when I try to reboot; the issue what I suspected what a thermal issue. In other words when I restart, it does continue to boot until the kernel panic.
Here the serial dump

U-Boot SPL 2013.04-dirty (Apr 26 2013 - 07:32:17)
musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn)
musb-hdrc: MHDRC RTL version 2.0
musb-hdrc: setup fifo_mode 4
musb-hdrc: 28/31 max ep, 16384/16384 memory
USB Peripheral mode controller at 47401000 using PIO, IRQ 0
musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn)
musb-hdrc: MHDRC RTL version 2.0
musb-hdrc: setup fifo_mode 4
musb-hdrc: 28/31 max ep, 16384/16384 memory
USB Host mode controller at 47401800 using PIO, IRQ 0
OMAP SD/MMC: 0
reading u-boot.img
reading u-boot.img

U-Boot 2013.04-dirty (Apr 26 2013 - 07:32:17)

I2C: ready
DRAM: 512 MiB
WARNING: Caches not enabled
NAND: No NAND device found!!!
0 MiB
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
*** Warning - readenv() failed, using default environment

musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn)
musb-hdrc: MHDRC RTL version 2.0
musb-hdrc: setup fifo_mode 4
musb-hdrc: 28/31 max ep, 16384/16384 memory
USB Peripheral mode controller at 47401000 using PIO, IRQ 0
musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn)
musb-hdrc: MHDRC RTL version 2.0
musb-hdrc: setup fifo_mode 4
musb-hdrc: 28/31 max ep, 16384/16384 memory
USB Host mode controller at 47401800 using PIO, IRQ 0
Net: not set. Validating first E-fuse MAC
cpsw, usb_ether
Hit any key to stop autoboot: 0
gpio: pin 53 (gpio 53) value is 1
mmc0 is current device
micro SD card found
mmc0 is current device
gpio: pin 54 (gpio 54) value is 1
SD/MMC found on device 0
reading uEnv.txt
14 bytes read in 3 ms (3.9 KiB/s)
Loaded environment from uEnv.txt
Importing environment from mmc …
gpio: pin 55 (gpio 55) value is 1
4484848 bytes read in 802 ms (5.3 MiB/s)
gpio: pin 56 (gpio 56) value is 1
22492 bytes read in 54 ms (406.3 KiB/s)
Booting from mmc …

Booting kernel from Legacy Image at 80007fc0 …

Image Name: Angstrom/3.8.3/beaglebone
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 4484784 Bytes = 4.3 MiB
Load Address: 80008000
Entry Point: 80008000
Verifying Checksum … OK

Flattened Device Tree blob at 80f80000

Booting using the fdt blob at 0x80f80000
XIP Kernel Image … OK
OK
Using Device Tree in place at 80f80000, end 80f887db

Starting kernel …

Uncompressing Linux… done, booting the kernel.
[ 0.106027] pinctrl-single 44e10800.pinmux: prop pinctrl-0 index 0 invalid phandle
[ 0.222224] omap2_mbox_probe: platform not supported
[ 0.471625] tps65217-bl tps65217-bl: no platform data provided
[ 0.555488] bone-capemgr bone_capemgr.8: slot #0: No cape found
[ 0.592594] bone-capemgr bone_capemgr.8: slot #1: No cape found
[ 0.629703] bone-capemgr bone_capemgr.8: slot #2: No cape found
[ 0.666812] bone-capemgr bone_capemgr.8: slot #3: No cape found
[ 0.751306] pinctrl-single 44e10800.pinmux: pin 44e1080c already requested by mmc.4; cannot claim for gpio-leds.7
[ 0.762116] pinctrl-single 44e10800.pinmux: pin-3 (gpio-leds.7) status -22
[ 0.769330] pinctrl-single 44e10800.pinmux: could not request pin 3 on device pinctrl-single
[ 1.018922] EXT4-fs (mmcblk0p2): error loading journal
[ 1.024874] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2)
[ 1.033735] [] (unwind_backtrace+0x0/0xe0) from [] (panic+0x84/0x1e0)
[ 1.042298] [] (panic+0x84/0x1e0) from [] (mount_block_root+0x1e0/0x220)
[ 1.051126] [] (mount_block_root+0x1e0/0x220) from [] (mount_root+0xec/0x10c)
[ 1.060406] [] (mount_root+0xec/0x10c) from [] (prepare_namespace+0x15c/0x1bc)
[ 1.069787] [] (prepare_namespace+0x15c/0x1bc) from [] (kernel_init+0x8/0xe4)
[ 1.079075] [] (kernel_init+0x8/0xe4) from [] (ret_from_fork+0x14/0x3c)
[ 1.087810] drm_kms_helper: panic occurred, switching back to text console

Mine booted fine and flashed the eMMC. After flashing, I rebooted fine. I had no desktop image, but it did boot.

Gerald

The Angstrom Distribution beaglebone ttyO0

Angstrom v2012.12 - Kernel 3.8.11

beaglebone login: [ 20.060479] libphy: PHY 4a101000.mdio:01 not found
[ 20.065607] net eth0: phy 4a101000.mdio:01 not found on slave 1
[ 20.814927] tilcdc 4830e000.fb: timeout waiting for framedone
[ 119.031737] hub 1-1:1.0: cannot reset port 1 (err = -110)
[ 120.039549] hub 1-1:1.0: cannot reset port 1 (err = -110)
[ 121.047371] hub 1-1:1.0: cannot reset port 1 (err = -110)

U-Boot SPL 2013.04-dirty (Apr 26 2013 - 07:32:17)
musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn)
musb-hdrc: MHDRC RTL version 2.0
musb-hdrc: setup fifo_mode 4
musb-hdrc: 28/31 max ep, 16384/16384 memory
USB Peripheral mode controller at 47401000 using PIO, IRQ 0
musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn)
musb-hdrc: MHDRC RTL version 2.0
musb-hdrc: setup fifo_mode 4
musb-hdrc: 28/31 max ep, 16384/16384 memory
USB Host mode controller at 47401800 using PIO, IRQ 0
OMAP SD/MMC: 0
mmc_send_cmd : timeout: No status update
reading u-boot.img
reading u-boot.img

U-Boot 2013.04-dirty (Apr 26 2013 - 07:32:17)

I2C: ready
DRAM: 512 MiB
WARNING: Caches not enabled
NAND: No NAND device found!!!
0 MiB
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
*** Warning - readenv() failed, using default environment

musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn)
musb-hdrc: MHDRC RTL version 2.0
musb-hdrc: setup fifo_mode 4
musb-hdrc: 28/31 max ep, 16384/16384 memory
USB Peripheral mode controller at 47401000 using PIO, IRQ 0
musb-hdrc: ConfigData=0xde (UTMI-8, dyn FIFOs, HB-ISO Rx, HB-ISO Tx, SoftConn)
musb-hdrc: MHDRC RTL version 2.0
musb-hdrc: setup fifo_mode 4
musb-hdrc: 28/31 max ep, 16384/16384 memory
USB Host mode controller at 47401800 using PIO, IRQ 0
Net: not set. Validating first E-fuse MAC
cpsw, usb_ether
Hit any key to stop autoboot: 0
gpio: pin 53 (gpio 53) value is 1
Card did not respond to voltage select!
mmc0(part 0) is current device
mmc_send_cmd : timeout: No status update
Card did not respond to voltage select!
No micro SD card found, setting mmcdev to 1
mmc1(part 0) is current device
mmc_send_cmd : timeout: No status update
gpio: pin 54 (gpio 54) value is 1
SD/MMC found on device 1
reading uEnv.txt
14 bytes read in 3 ms (3.9 KiB/s)
Loaded environment from uEnv.txt
Importing environment from mmc …
gpio: pin 55 (gpio 55) value is 1
4218832 bytes read in 775 ms (5.2 MiB/s)
gpio: pin 56 (gpio 56) value is 1
23469 bytes read in 50 ms (458 KiB/s)
Booting from mmc …

Booting kernel from Legacy Image at 80007fc0 …

Image Name: Angstrom/3.8.11/beaglebone
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 4218768 Bytes = 4 MiB
Load Address: 80008000
Entry Point: 80008000
Verifying Checksum … OK

Flattened Device Tree blob at 80f80000

Booting using the fdt blob at 0x80f80000
XIP Kernel Image … OK
OK
Using Device Tree in place at 80f80000, end 80f88bac

Starting kernel …

Uncompressing Linux… done, booting the kernel.
[ 0.192660] omap2_mbox_probe: platform not supported
[ 0.203144] tps65217-bl tps65217-bl: no platform data provided
[ 0.279682] bone-capemgr bone_capemgr.9: slot #0: No cape found
[ 0.316791] bone-capemgr bone_capemgr.9: slot #1: No cape found
[ 0.353897] bone-capemgr bone_capemgr.9: slot #2: No cape found
[ 0.391006] bone-capemgr bone_capemgr.9: slot #3: No cape found
[ 0.421385] omap_hsmmc mmc.4: of_parse_phandle_with_args of ‘reset’ failed
[ 0.485323] pinctrl-single 44e10800.pinmux: pin 44e10854 already requested by 44e10800.pinmux; cannot claim for gpio-leds.8
[ 0.497065] pinctrl-single 44e10800.pinmux: pin-21 (gpio-leds.8) status -22
[ 0.504377] pinctrl-single 44e10800.pinmux: could not request pin 21 on device pinctrl-single
[ 0.902714] tilcdc 4830e000.fb: timeout waiting for framedone
[ 9.780321] libphy: PHY 4a101000.mdio:01 not found
[ 9.785424] net eth0: phy 4a101000.mdio:01 not found on slave 1
[ 10.674286] tilcdc 4830e000.fb: timeout waiting for framedone
systemd-fsck[83]: Angstrom: clean, 50417/112672 files, 283888/449820 blocks

.—O—.

.-. o o

-----.-----.-----.| | .----…-----.-----.
__ | —'| ‘–.| .-’| | |

— || --‘| | | ’ | | | |
‘—’—’–‘–’–. |-----‘’----‘’–’ ‘-----’-‘-’-’
-’ |
‘—’

The Angstrom Distribution beaglebone ttyO0

Angstrom v2012.12 - Kernel 3.8.11

beaglebone login:

Are you letting it flash the eMMC, removing the SD card once it has flashed, and power cycling?

Gerald

It never got that far. However I am now using a new SD card since the boot log was whining about the SD filesystem. It has gotten past the kernel panic and looks to be flashing the nand. It is looking pretty good now considering I wasn’t able to get even close to this point with the earlier versions.

Mark

Great! Keep us posted. If all goes well, when you boot this image it will default to the best CEA resolution, meaning that it will support the audio playback. Assuming that you monitor can support audio,

Gerald

Bingo! It is working.
Gerald, thanks to you and all the wizards behind the curtains.

Mark

Thank you for the feedback. We are working to get this worked in as one of the many latest images to come! We will also be working it in as the next production image.

Gerald