vision_apps / EdgeAI on BBAI-64 with kernel 6.12: remote cores hang, “Exceeded max object descriptors”
Follow-up to my earlier post — first of all, thanks for the overlay files
(k3-j721e-edgeai-apps.dtso / k3-j721e-rtos-memory-map.dtsi). They were very
helpful as a reference, although the memory map itself turned out NOT to be the
remaining problem in my setup. I’m posting the full root cause and a step-by-step
fix here, because this will hit anyone running TI vision_apps (PSDK-Analytics
11.02.x) on the BeagleBone AI-64 with a mainline-based kernel (bb.org 6.12 in my
case), and the failure mode is extremely confusing to debug.
Symptom
vx_app_arm_mem.out (and every other vision_apps demo) fails on the A72 with:
ownContextCreateCmdObj: context object descriptor [0] allocation failed / Exceeded max
All 5 remote cores show running in remoteproc, no errors in dmesg. But the
shared APP_LOG (phys 0xAC000000) reveals that only mcu2_0, mcu2_1 and C6x_2
reach the vision_apps boot sync barrier (“APP: Syncing with 5 CPUs”), while
C6x_1 and C7x_1 never even print their first line — they hang in the very
earliest RTOS startup. Since the barrier never completes, no RTOS core ever runs
tivxPlatformResetObjDescTableInfo(), the object descriptor table at
0xAC040000 stays uninitialized, and the A72 finds no free (0xFFFF) slot →
“Exceeded max” at descriptor [0]. The error message is purely a downstream
symptom.
Root cause: Linux steals the firmware’s DMTimers
Each vision_apps RTOS core uses a fixed main-domain DMTimer as its OS tick:
| Core | Timer |
|---|---|
| C66x_1 | main_timer0 |
| C66x_2 | main_timer1 |
| C7x_1 | main_timer2 |
| R5F | main_timer12–15 |
TI’s vendor kernel marks exactly these timers status = "reserved" in
k3-j721e-main.dtsi (see ti-linux-6.6.y: main_timer0, 1, 2, 12, 13, 14, 15).
Mainline — and therefore the bb.org kernel — reserves none of them. So Linux
probes all 20 main timers with TI_SCI_PD_EXCLUSIVE power domains, and at boot
you get a race between the Linux timer driver and the auto-booting RTOS cores
over the TI-SCI device claims. In my boots, C6x_2 happened to win TIMER1 while
C6x_1 and C7x_1 lost TIMER0/TIMER2 — k3conf dump devices showed exactly those
two as DEVICE_STATE_OFF while everything else was ON. A core whose tick timer
it cannot claim hangs before its first log line. That’s why two identical C66
cores behaved differently: it’s a boot lottery, not a configuration difference.
This explains why nobody sees this on the TI SK-EVM with the TI kernel — the
reservations are already in TI’s SoC dtsi there.
Secondary finding: C66 dma-region pairing differs between BBAI-64 and SK
While digging, I also found that k3-j721e-beagleboneai64.dts pairs the C66
reserved-memory nodes straight (c66_0 → dma@a6000000 + mem@a6100000),
whereas k3-j721e-som-p0.dtsi (SK) pairs them crossed. If you port the
vision_apps memory map by overriding reg on the existing nodes (instead of
using the overlay), the correct assignment per app_mem_map.h is:
- node
c66-dma-memory@a6000000(dsp@4d80800000 / C66x_1) →0xa9000000 - node
c66-dma-memory@a7000000(dsp@4d81800000 / C66x_2) →0xa8000000
Note this was NOT the cause of the hang: readelf -x .resource_table shows the
C66 firmwares request their vrings with da=0xFFFFFFFF (ADDR_ANY), so the
kernel allocates freely and the firmware reads the address back. It still needs
to be correct for a clean setup. Also, the C7x’s
remoteproc remoteproc2: unsupported resource 65538 in dmesg is harmless — it’s
TI’s vendor trace-buffer resource entry, which mainline simply skips.
The fix
Add this to your board DT (I include a .dtsi into the board dts via a Yocto
bbappend; the same content works in an overlay):
/* DMTimers used by vision_apps RTOS firmware — keep Linux off them.
* Parity with TI vendor kernel (ti-linux-6.6.y k3-j721e-main.dtsi). */
&main_timer0 { status = "reserved"; };
&main_timer1 { status = "reserved"; };
&main_timer2 { status = "reserved"; };
&main_timer12 { status = "reserved"; };
&main_timer13 { status = "reserved"; };
&main_timer14 { status = "reserved"; };
&main_timer15 { status = "reserved"; };
plus the corrected C66 dma reg values mentioned above if you use the
reg-override approach.
Rebuild the DTB, deploy, reboot. Result on my board: all 5 RTOS cores boot and
complete the sync barrier (APP_LOG shows “Syncing with 5 CPUs” twice per core),
the obj_desc table gets reset (entries start 0000ffff — the 0xFFFF INVALID
marker is the type field at offset 2, not offset 0), and vx_app_arm_mem.out
runs clean end-to-end, including after a plain reboot with kernel auto-boot of
all cores.
How to verify (no JTAG needed)
# Timer states — none of the firmware timers should be OFF / Linux-bound:
k3conf dump devices | grep TIMER
# APP_LOG + obj_desc check via /dev/mem:
python3 - <<'PYEOF'
import mmap,os
f=os.open('/dev/mem',os.O_RDONLY|os.O_SYNC)
d=mmap.mmap(f,0x40000,prot=mmap.PROT_READ,offset=0xAC000000)[:0x40000]
print('Syncing:',d.count(b'Syncing with'),'| C6x_1:',d.count(b'C6x_1'),'C7x_1:',d.count(b'C7x_1'))
o=mmap.mmap(os.open('/dev/mem',os.O_RDONLY|os.O_SYNC),4096,prot=mmap.PROT_READ,offset=0xAC040000)
print('obj_desc[0..8]:', o[:8].hex())
PYEOF
# Goal: Syncing=10 (5 cores x 2 lines), C6x_1>=1, C7x_1>=1, obj_desc 0000ffff...
One warning
Do NOT try to fix this at runtime by unbinding the Linux timer driver and
re-enabling the devices with k3conf enable device … while restarting cores. I
managed to hard-freeze the entire SoC that way (heartbeat included) — most
likely an interconnect stall or a k3conf-vs-kernel race on the secure proxy
(k3conf talks TI-SCI via /dev/mem behind the kernel’s back). Put the
reservations in the device tree, where they belong.
Environment for reference
- BeagleBone AI-64 (J721E SR1.1), Yocto
tisdk-edgeai-image - Kernel 6.12.43-ti (linux-bb.org), extlinux boot (kernel loaded low so all
vision_apps reserved-memory regions can be placed) - PSDK-Analytics 11.02.03 (
ti-vision-apps11.02.03,ti-edgeai-firmware
REL.PSDK.ANALYTICS.11.02.00.10) - Other prerequisites from earlier in the saga (kernel 6.12 specifics): carveout
DT node for thedma-heap-carveoutshared region, R5F split-mode
(ti,cluster-mode = <0>), ti-rpmsg-char chrdev-lookup fix, mcu3 removed from
the HLOS IPC list, DMA-heap name symlinks
(vision_apps_shared-memories→carveout_…) at runtime.
Hope this saves someone the weeks it cost me. Happy to share the full .dtsi /
bbappend if anyone wants it.