It took me two (long) days to discover why my LCD ran perfectly with a 3.2 kernel and did not with the 3.8 kernel.
Using an original Beaglebone and the LVDS LCD cape, I got a nice stable image on my LCD running kernel 3.2.
But a BBB with the same cape would not give a stable display, most of the time it was completely distorted. If it was ok for a moment, it seemed to worsen with CPU load.
As a first step I reduced the problem and ran both kernels with a BeagleBone (I eliminated the changes caused by the BeagleBone Black).
This did not improve matters, so this had to be a software only problem.
After making sure that the generated signals for both situations were exactly the same (timing, edges, polarity), I started to suspect the clock signal.
Tracking the clock sources for the lcd showed that the lcd clock generation has changed between kernels 3.2 and 3.8.
With the 3.8 kernel, the 45 MHz lcd clock was derived from a 24 MHz base clock. I'm not familiar with the details of the AM335x internal PLLs, but to get the requested 45 MHz clock an intermediate 360 MHz (24*15) is needed to get 45 MHz (360/8).
Luckily the LCD panel spec gave a clock range from 39 to 52 MHz, with 45 MHz being the typical value. That range includes 48 MHz, being 2 times 24 MHz. A much easier task for the PLL and no strange (non-24MHz) harmonics in the generated signal.
A simple change, and voila: After changing the display timing to 48 MHz the generated LCD image was completely stable!
To try this theory, I also tested with 40 MHz instead of 48 MHz. This would need an intermediate 120 MHz (mult 5, div 3). This output was not as bad as with 48 MHz. Stable if idle, but any CPU activity would make the display unreadable.
So anybody implementing LCD or HDMI timing must keep in mind that to get the best (jitter-free) clock, the PLL multiplier and divisor have to be choosen carefully!
-- Bas