Lockups on BBGW with latest bb-wl18xx-firmware

I’m getting the following errors with the latest firmware 1.20221129.7-0~bullseye+20221129 on BBGW, hardware version 1.1. Is this still a work in progress?

[   45.545676] Bluetooth: hci0: command 0xfd06 tx timeout
[   53.737782] Bluetooth: hci0: send command failed
[   53.742536] Bluetooth: hci0: download firmware failed, retrying...
[   80.057358] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
[  108.057350] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
...

I’ve been working on:

today…

debian@beaglebone:~$ dmesg | grep wlcore | grep firmware
[   68.440215] wlcore: PHY firmware version: Rev 8.2.0.0.242
[   68.488248] wlcore: firmware booted (Rev 8.9.0.0.79)
debian@beaglebone:~$ tree /lib/firmware/ti-connectivity/git/
/lib/firmware/ti-connectivity/git/
├── wl18xx-fw-4-8.9.0.0.79.bin
├── wl18xx-fw-4-8.9.0.0.81.bin
├── wl18xx-fw-4-8.9.0.0.83.bin
├── wl18xx-fw-4-8.9.0.0.84.bin
├── wl18xx-fw-4-8.9.0.0.85.bin
├── wl18xx-fw-4-8.9.0.0.86.bin
└── wl18xx-fw-4-8.9.0.0.88.bin

0 directories, 7 files

So related, but for you it’s still bluetooth side of wl18xx, hopefully work on that tomorrow…

Regards,

# apt install --only-upgrade bb-wl18xx-firmware
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
bb-wl18xx-firmware is already the newest version (1.20221129.7-0~bullseye+20221129).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Looks like I’ve already got the latest. As you suggest, the problem must be with the bluetooth side. If I log in and make it busy with commands immediately after boot it seems to do better. Timing issue?

FYI - I get the same results as you did with the dmesg/grep and tree commands. Thanks for taking a look at the bluetooth side!

@ridgelift please run

sudo beagle-version

I’m trying to trigger:

[   45.545676] Bluetooth: hci0: command 0xfd06 tx timeout
[   53.737782] Bluetooth: hci0: send command failed
[   53.742536] Bluetooth: hci0: download firmware failed, retrying...

Regards,

Hi Robert - sorry for the delay in getting back to you. Here’s the output:

eeprom:[A335BNLTGW1ABBGW21114685]
model:[TI_AM335x_BeagleBone_Green_Wireless]
dogtag:[BeagleBoard.org Debian Buster IoT Image 2020-05-18]
bootloader:[eMMC-(default)]:[/dev/mmcblk1]:[U-Boot SPL 2022.04-gd0771d53 (Jul 21 2022 - 02:13:23 +0000)]:[location: dd MBR]
UBOOT: Booted Device-Tree:[am335x-boneblack-uboot.dts]
UBOOT: Loaded Overlay:[BB-BBGW-WL1835-00A0.kernel]
UBOOT: Loaded Overlay:[BB-BONE-eMMC1-01-00A0.kernel]
UBOOT: Loaded Overlay:[uio-pruss-5.4]
kernel:[5.10.145-ti-r54]
/boot/uEnv.txt Settings:
uboot_overlay_options:[enable_uboot_overlays=1]
uboot_overlay_options:[uboot_overlay_addr4=/lib/firmware/CT-INPUT-KEYS-00A0.dtbo]
uboot_overlay_options:[uboot_overlay_addr5=/lib/firmware/CT-PWM-P8_34-00A0.dtbo]
uboot_overlay_options:[uboot_overlay_addr6=/lib/firmware/CT-EGPIO-00A0.dtbo]
uboot_overlay_options:[disable_uboot_overlay_video=1]
uboot_overlay_options:[disable_uboot_overlay_audio=1]
uboot_overlay_options:[disable_uboot_overlay_adc=1]
uboot_overlay_options:[uboot_overlay_pru=/lib/firmware/uio-pruss-5.4.dtbo]
pkg check: to individually upgrade run: [sudo apt install --only-upgrade <pkg>]
pkg:[bb-cape-overlays]:[4.14.20210821.0-0~bullseye+20210821]
pkg:[bb-customizations]:[1.20221108.0-0~bullseye+20221108]
pkg:[bb-usb-gadgets]:[1.20220816.0-0~bullseye+20220816]
pkg:[bb-wl18xx-firmware]:[1.20221129.7-0~bullseye+20221129]
pkg:[kmod]:[28-1]
WARNING:pkg:[librobotcontrol]:[NOT_INSTALLED]
cmdline:[console=ttyS0,115200n8 root=/dev/mmcblk1p1 ro rootfstype=ext4 rootwait coherent_pool=1M net.ifnames=0 lpj=1990656 rng_core.default_quality=100 quiet]
dmesg | grep remote
[    3.848281] remoteproc remoteproc0: wkup_m3 is available
[   28.549329] remoteproc remoteproc0: powering up wkup_m3
[   28.549368] remoteproc remoteproc0: Booting fw image am335x-pm-firmware.elf, size 217148
[   28.549673] remoteproc remoteproc0: remote processor wkup_m3 is now up
[   41.015848] Bluetooth: hci0: change remote baud rate command in firmware
dmesg | grep pru
dmesg | grep pinctrl-single
[    3.708818] pinctrl-single 44e10800.pinmux: 142 pins, size 568
dmesg | grep gpio-of-helper
lsusb
Bus 001 Device 002: ID 0424:2514 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
END

Here’s the serial log. As I mentioned earlier, if I log in and run a few commands immediately after boot then I don’t get the failures.

U-Boot SPL 2022.04-gd0771d53 (Jul 21 2022 - 02:13:23 +0000)
Trying to boot from MMC2


U-Boot 2022.04-gd0771d53 (Jul 21 2022 - 02:13:23 +0000)

CPU  : AM335X-GP rev 2.1
Model: TI AM335x BeagleBone Black
DRAM:  512 MiB
Reset Source: Global external warm reset has occurred.
Reset Source: Global warm SW reset has occurred.
Reset Source: Power-on reset has occurred.
RTC 32KCLK Source: External.
Core:  150 devices, 14 uclasses, devicetree: separate
WDT:   Started wdt@44e35000 with servicing (60s timeout)
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
Loading Environment from EXT4... Board: BeagleBone Black
<ethaddr> not set. Validating first E-fuse MAC
BeagleBone Black:
Model: SeeedStudio BeagleBone Green Wireless:
BeagleBone Cape EEPROM: no EEPROM at address: 0x54
BeagleBone Cape EEPROM: no EEPROM at address: 0x55
BeagleBone Cape EEPROM: no EEPROM at address: 0x56
BeagleBone Cape EEPROM: no EEPROM at address: 0x57
Net:   Could not get PHY for ethernet@4a100000: addr 0
eth2: ethernet@4a100000, eth3: usb_ether
Press SPACE to abort autoboot in 0 seconds
board_name=[A335BNLT] ...
board_rev=[GW1A] ...
gpio: pin 56 (gpio 56) value is 0
gpio: pin 55 (gpio 55) value is 0
gpio: pin 54 (gpio 54) value is 0
gpio: pin 53 (gpio 53) value is 1
switch to partitions #0, OK
mmc1(part 0) is current device
Scanning mmc 1:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Scanning disk mmc@48060000.blk...
Disk mmc@48060000.blk not ready
Scanning disk mmc@481d8000.blk...
Found 2 disks
No EFI system partition
BootOrder not defined
EFI boot manager: Cannot load any image
gpio: pin 56 (gpio 56) value is 0
gpio: pin 55 (gpio 55) value is 0
gpio: pin 54 (gpio 54) value is 0
gpio: pin 53 (gpio 53) value is 1
switch to partitions #0, OK
mmc1(part 0) is current device
gpio: pin 54 (gpio 54) value is 1
Checking for: /uEnv.txt ...
Checking for: /boot/uEnv.txt ...
gpio: pin 55 (gpio 55) value is 1
2268 bytes read in 3 ms (738.3 KiB/s)
Loaded environment from /boot/uEnv.txt
Checking if uname_r is set in /boot/uEnv.txt...
gpio: pin 56 (gpio 56) value is 1
Running uname_boot ...
loading /boot/vmlinuz-5.10.145-ti-r54 ...
10514944 bytes read in 663 ms (15.1 MiB/s)
debug: [enable_uboot_overlays=1] ...
debug: [enable_uboot_cape_universal=] ...
debug: [uboot_base_dtb_univ=am335x-boneblack-uboot.dtb] ...
uboot_overlays: [uboot_base_dtb=am335x-boneblack-uboot.dtb] ...
uboot_overlays: Switching too: dtb=am335x-boneblack-uboot.dtb ...
loading /boot/dtbs/5.10.145-ti-r54/am335x-boneblack-uboot.dtb ...
91807 bytes read in 10 ms (8.8 MiB/s)
Found 0 extension board(s).
uboot_overlays: [fdt_buffer=0x60000] ...
uboot_overlays: uboot loading of [BB-ADC-00A0.dtbo] disabled by /boot/uEnv.txt [disable_uboot_overlay_adc=1]...
uboot_overlays: loading /lib/firmware/CT-INPUT-KEYS-00A0.dtbo ...
1247 bytes read in 6 ms (202.1 KiB/s)
uboot_overlays: loading /lib/firmware/CT-PWM-P8_34-00A0.dtbo ...
1275 bytes read in 7 ms (177.7 KiB/s)
uboot_overlays: loading /lib/firmware/CT-EGPIO-00A0.dtbo ...
4603 bytes read in 8 ms (561.5 KiB/s)
uboot_overlays: loading /boot/dtbs/5.10.145-ti-r54/overlays/BB-BONE-eMMC1-01-00A0.dtbo ...
1605 bytes read in 6 ms (260.7 KiB/s)
uboot_overlays: loading /boot/dtbs/5.10.145-ti-r54/overlays/BB-BBGW-WL1835-00A0.dtbo ...
3977 bytes read in 6 ms (646.5 KiB/s)
uboot_overlays: loading /lib/firmware/uio-pruss-5.4.dtbo ...
687 bytes read in 6 ms (111.3 KiB/s)
loading /boot/initrd.img-5.10.145-ti-r54 ...
6859328 bytes read in 438 ms (14.9 MiB/s)
debug: [console=ttyS0,115200n8 root=/dev/mmcblk1p1 ro rootfstype=ext4 rootwait coherent_pool=1M net.ifnames=0 lpj=1990656 rng_core.default_quality=100 quiet] ...
debug: [bootz 0x82000000 0x88080000:68aa40 88000000] ...
Kernel image @ 0x82000000 [ 0x000000 - 0xa07200 ]
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Ramdisk to 8f975000, end 8ffffa40 ... OK
   Loading Device Tree to 8f8f9000, end 8f974fff ... OK

Starting kernel ...

[    0.115134] l3-aon-clkctrl:0000:0: failed to disable
[    3.671411] debugfs: Directory '49000000.dma' with parent 'dmaengine' already present!
[    3.939962] omap_voltage_late_init: Voltage driver support not added
rootfs: recovering journal
rootfs: clean, 60954/233392 files, 678511/932864 blocks

Kernel: 5.10.145-ti-r54 #1bullseye SMP PREEMPT Fri Nov 18 21:59:46 UTC 2022

beaglebone login: [   45.385624] Bluetooth: hci0: command 0xfd06 tx timeout
[   53.481627] Bluetooth: hci0: send command failed
[   53.486383] Bluetooth: hci0: download firmware failed, retrying...
[   80.057334] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:3:130]

I figured I would start from scratch with the last Debian iot images. This didn’t appear to help - it’s suffering from the same problem on the v1.1 BBGW hardware. hci0 errors and lockups that are sometimes mitigated by quickly logging in and doing stuff.

Switching these back to the old method:

git clone -b v5.10.x-ti-unified https://git.beagleboard.org/beagleboard/BeagleBoard-DeviceTrees.git
cd ./BeagleBoard-DeviceTrees/
make
sudo make install_arm
sudo reboot

To use Bluetooth, you’d need to enable it thru userspace…

sudo bb-wl18xx-bluetooth-hciattach

Regards,

Please test this test image… (link will auto disappear after 30 days)…

https://rcn-ee.net/rootfs/debian-armhf-iot/2022-12-07/am335x-debian-11.5-iot-armhf-2022-12-07-4gb.img.xz

Regards,

bb-wl18xx-bluetooth-hciattach definitely worked, although I did get the following error:

Bluetooth: hci0: sending frame failed (-49)

I’ll give the test image a try as well, thanks.

Hi Robert,

I’m trying to set up dependencies in my services to wait until the WiFi is fully associated before starting bluetooth. I’m trying to wait for the network-online.target. I’m wondering if this never occurs because the usb interfaces never receive an IP address.

Is this a correct approach? Regardless, I’m also wondering how to disable the SoftAp0, usb0 and usb1 interfaces. I used to to this in /etc/default/bb-boot, but now it’s unclear to me how to accomplish this.

This command appears to disable the usb0 and usb1 interfaces now:

systemctl disable bb-usb-gadgets

SoftAp0 is started by

#/etc/udev/rules.d/81-add-SoftAp0-interface.rules
SUBSYSTEM=="net", ACTION=="add", KERNEL=="wlan[0-9]", DRIVERS=="wl18xx_driver" \
        RUN+="/sbin/iw dev %k interface add SoftAp0 type managed"
SUBSYSTEM=="net", ACTION=="add", KERNEL=="wlan[0-9]", DRIVERS=="brcmfmac" \
        RUN+="/sbin/iw dev %k interface add SoftAp0 type __ap"

Then hostapd on SoftAp0 is started by:

#/etc/udev/rules.d/82-SoftAp0-start-hostpad.rules
SUBSYSTEM=="net", KERNEL=="SoftAp0", DRIVERS=="wl18xx_driver", TAG+="systemd", ENV{SYSTEMD_WANTS}="hostapd.service"
SUBSYSTEM=="net", KERNEL=="SoftAp0", DRIVERS=="brcmfmac", TAG+="systemd", ENV{SYSTEMD_WANTS}="hostapd.service"

and configured thru:

#/etc/hostapd/hostapd.conf
interface=SoftAp0

# a simply means 5GHz
# g simply means 2.4GHz band
hw_mode=g

# the channel to use
channel=1

# limit the frequencies used to those allowed in the country
#ieee80211d=1
# the country code
#country_code=US

# 802.11n support
ieee80211n=1
# QoS support, also required for full speed on 802.11n/ac/ax
wmm_enabled=1

ssid=BeagleBone-F366

# 1=wpa, 2=wep, 3=both
auth_algs=1
wpa=2
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP
wpa_passphrase=BeagleBone

macaddr_acl=0
ignore_broadcast_ssid=0

logger_syslog=-1
logger_syslog_level=2

USB0 / USB1

is started by bb-usb-gadgets and configured via:

#/etc/systemd/network/usb0.network
[Match]
Name=usb0

[Link]
RequiredForOnline=no

[Network]
DHCP=no
Address=192.168.7.2/24
DHCPServer=on

[DHCPServer]
EmitDNS=no
EmitRouter=no
EmitTimezone=no
MaxLeaseTimeSec=20min
PoolSize=1

and

#/etc/systemd/network/usb1.network
[Match]
Name=usb1

[Link]
RequiredForOnline=no

[Network]
DHCP=no
Address=192.168.6.2/24
DHCPServer=on

[DHCPServer]
EmitDNS=no
EmitRouter=no
EmitTimezone=no
MaxLeaseTimeSec=20min
PoolSize=1

So other then the bb-usb-gadgets systemd file

#/lib/systemd/system/bb-usb-gadgets.service 
[Unit]
Description=BeagleBoard.org USB gadgets
After=usb-gadget.target
ConditionFileIsExecutable=/usr/bin/bb-start-usb-gadgets

[Service]
Type=forking
ExecStartPre=/sbin/modprobe libcomposite
ExecStart=/usr/bin/bb-start-usb-gadgets

[Install]
WantedBy=multi-user.target
#/usr/bin/bb-start-usb-gadgets 
#!/bin/bash

log="bb-usb-gadgets"

#Examples
#https://github.com/linux-usb-gadgets/libusbgx/tree/master/examples

#built_in="gadget-acm-ecm"
#built_in="gadget-ffs"
#built_in="gadget-hid"
#built_in="gadget-midi"
#built_in="gadget-ms"
#built_in="gadget-printer"
#built_in="gadget-rndis-os-desc"
#built_in="gadget-uac2"
#built_in="gadget-uvc"

#/usr/bin/gadget-export
#/usr/bin/gadget-import
#/usr/bin/gadget-vid-pid-remove
#/usr/bin/show-gadgets
#/usr/bin/show-udcs

#echo "${log} /usr/bin/$build_in"
#/usr/bin/$built_in

#Someday... (RNDIS os-desc broken...)
#echo "${log}: gt load --off bbb-acm-ncm-rndis.scheme g_multi"
#gt load --off bbb-acm-ncm-rndis.scheme g_multi
##update mac addresss...
#echo "${log}: gt enable g_multi"
#gt enable g_multi

#Working old..
#TODO: generate unique mac address...
echo "${log}: bb-start-acm-ncm-rndis-old-gadget"
/usr/bin/bb-start-acm-ncm-rndis-old-gadget

Regards,

That’s super helpful, thanks!