What is recommended setup for XServer?

Current setup:
BBB with minimalist Ubuntu build ( 12.04.02 )
gcc installed using
sudo apt-get install build-essential
qt installed

sudo apt-get install qt4-dev-tools

minimal QT application is written, compiled with no errors
had to add:

sudo apt-get install libicu48

after that application “tries” to run

“qte: cannot connect to X server”

Makes sense, actually… So to the question: what is recommended settings for the x-server on BBB? At the end of the day the board suppose to be part of an embedded system that runs ONE application with pretty simple GUI, so full desktop environment isn’t required, but actually would be detrimental to the performance . QT GUI suppose to be able to run without full desktop, correct?
So back to the question: what would recommended x-server setup would be in such situation?

Since you say you don't want X, is there some reason you're trying to
install it and not just running QT directly on the frame buffer?