This seems to be a kernel issue with the ubuntu/debian images, based
on you just happen to be powering the board... For the time being use
a 5volt dc power connector and plugin the usb/serial adapter later (or
was it the reverse, of that, i don't remember off hand.... Or use
angstrom's kernel on top of ubuntu's userspace..
Regards,