I started to read OMAP3 TRM to get an idea how clock and DPLL configuration works. I still try to get display resolution 1280 x 1024 >= 50Hz to work. For this, we need pixel clock ~80MHz. With DPLL4 configuration used at the moment this is not possible. Result are patches in attachment. Unfortunately, they don't seem to work. But maybe they are helpfult for others 
Idea is to reconfigure periperhal DPLL4 so that we get DSS1_ALWON_FCLK (max: 173MHz) @ 162MHz. With this, it should be possible to get 81MHz pixel clock. Yes, this will break PRM_96M_ALWON_CLK != 96MHz. But for the moment it is only to get an idea how things work.
I tried both changes, in U-Boot and Linux kernel, but calc_ck_div() in dispc.c still reports 432MHz fck instead of expected 324MHz what would be necessary for 162MHz. Seems that something is still missing 
If interested, please find patches in attachment.
Dirk
change_dpll4_clock.txt (2.97 KB)
1280_1024_50_patch.txt (1.5 KB)