Debian 9 with desktop image issues.

I’ve been working on porting my build Retro Arena to the Beagle Bone Black, and I can successfully compile emulationstation and retroarch, they run, but I’ve noticed everything is running insanely slow, to the point NES lags. The build uses sdl/gles2
Any idea why the board is so slow? It’s like this even sitting in Linux, I’ve watched htop. Something is putting the CPU use up high when nothing is happening, but when I do away with the desktop things no longer want to display, almost like the drivers I’m using only function inside a desktop. Is there perhaps an image where I can use gles and sdl from console without issue? The desired results is the board boots directly into emulationstation and not the desktop. One of the users in my community sent me the board, to port the OS because the current one which I think is called bsnes isn’t working. Version 0.8 results in failed run on the device. Also it’s on trusty. I’ve got a Ubuntu 20 server image running as well, but without desktop it doesn’t want to display things.