Reminds me of compiling gcc on my armv5, that took a couple weeks to compiling! What you can do to speed things up is use a virtual machine, like qemu. Then you can keep the beagle enviroment, but have the speed of the host processor, especially if it simulates direct in the kernel.
Pocketbeagle2 is in meta-ti, a yocto image is the best way to go. Only install what you need, if you are working in python just install pip in the yocto image so you can load stuff you might have not included in the first build.
for now i will stick to the fastest way i found : use the version contained in the official-package-repos..
no more optimizing at that point in time.
later i will try the optimized image..