Why dont you simply use the image with Kernel: 5.10.109-ti-r45 from here? Debian 11.x (Bullseye) - Monthly Snapshot - 2023-10-07
This would solve your issue because kernel 5.10 is above 5.5.
If you want again the IoT-version of the image then take the Debian 11.x (Bullseye) IOT image.