I have an A4 with the same issue running Ubuntu 12.04 and 3.2.21-psp15 kernel.
I thought I had a broken Beaglebone, then on one occasion I powered off the PC that was running the serial console (using gtkterm) and without doing anything else it booted from the same hung task situation.
The situation now - leave that USB unconnected and the Beaglebone will boot to a full LXDE desktop which suggests a problem with the serial console code.
This happens on every PC it is connected to.
There is no problem with the Pandaboard serial interface.
root@panda:~# uname -a
Linux panda 3.2.7-x5 #1 SMP Sun Feb 26 04:34:36 UTC 2012 armv7l armv7l armv7l GNU/Linux
root@panda:~# cat /etc/lsb-release
DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"
Try leaving the USB unconnected, boot up and check you can ssh in.