TIDL Support on BeagleY-AI and BeagleBone AI-64
Both boards are based on TI’s Jacinto 7 SoCs (BeagleY-AI: AM67A (J722S); BeagleBone AI‑64: TDA4VM (J721E)). In TI’s Processor-SDK Edge AI, both processors support the TI Deep Learning (TIDL) library and tools github.com. For example, a user reported that on the AM67A (BeagleY-AI) board using Processor-SDK Linux v10.00.00.08, the included TIDL version is 10.00.04.00 e2e.ti.com. (The TDA4VM board similarly runs SDK 10.x, which includes a comparable TIDL release.) In short, TIDL 10.x is the supported toolchain on both boards, enabling the TI DL runtime to leverage the C7x DSP and MMA accelerators when properly enabled github.come2e.ti.com.
Transformer-Layer Support in TIDL
TI’s TIDL library has limited support for transformer operations, primarily oriented toward vision models. As TI’s documentation explains, starting in SDK 9.1 TIDL added support for basic Vision Transformer (ViT) models and layers: it supports multi-head attention, LayerNorm, SoftMax and similar blocks used in image-classification transformers e2e.ti.com. Release notes for newer TIDL versions (v10.1.4) explicitly highlight improvements for “transformer-based architectures”: e.g. fixed-point acceleration of LayerNorm, SoftMax, Concat and Add, and enhanced accuracy for vision transformer backbones like Swin, DeiT and LeViT github.com. However, this support is aimed at vision transformers (image tasks). TIDL does not provide a general-purpose transformer accelerator for large language models; it does not, for example, implement all the attention/embedding/gather operations typical of an LLM. In practice, complex LLM architectures like Qwen2.5‑Coder (a large code-generation transformer) contain operators beyond TIDL’s current supported set. TI’s own forums note that only “basic transformer (classification) ViT” and “partial support of SwinT” are supported e2e.ti.com, implying that a full LLM is outside the optimized use case.
Documentation and Community Notes on Compatibility
TI’s official notes and the BeagleBoard community confirm these limitations. The TI forums explicitly state that TIDL’s transformer support is restricted to certain vision tasks e2e.ti.com, and urge use of TIDL’s model-import tools with specific workarounds (e.g. ONNX opset downgrades) for ViT/DeiT networks. Moreover, BeagleBoard users have reported practical issues enabling TIDL on these boards. For instance, BeagleY-AI’s default device tree disables the C7x DSP (necessary for TIDL acceleration) in the stock image forum.beagleboard.org. Enabling the DSP requires custom overlays or modified firmware – workarounds that are nontrivial. As one community post summarized: “You will have to create a device tree overlay because the C7x is disabled in the k3-j722s base device tree. I don’t see any of that in the beagley‑ai base tree, yet.” forum.beagleboard.org. By mid-2025 no straightforward solution was reported. In short, there are no official TI/BeagleBoard statements claiming full transformer support on these boards; all evidence suggests that (a) TIDL is only partly transformer-compatible and (b) on BeagleY‑AI/AI‑64 the DSP accelerator isn’t enabled by default, so TIDL cannot transparently speed up a general LLM.
Implications: Why CPU Inference Is Used on Ollama
Because the TIDL toolchain on these boards cannot fully accelerate a Qwen2.5‑Coder model, the only reliable execution path is CPU-only. Ollama (the LLM runtime) falls back to running the model on the ARM cores. This is “most compatible” because it does not rely on any specialized hardware or unsupported operators. In contrast, attempting to offload Qwen2.5‑Coder to TIDL would hit unsupported layers or fail to run entirely. Running on CPU avoids all these compatibility issues, at the expense of throughput. In summary: TIDL on BeagleY-AI and AI‑64 is limited to vision-optimized models and often isn’t fully enabled on the boards, so CPU inference remains the practical choice for Ollama deployments e2e.ti.com forum.beagleboard.org.
Sources: TI Processor-SDK documentation and forums e2e.ti.com e2e.ti.com; Texas Instruments edgeai-tidl-tools release notes github.com; BeagleBoard community forums forum.beagleboard.org (see above citations).