[beagleboard-gsoc] [GSoC 2010] XBMC Rendering improvements

Hi!

First I would like to excuse if this post will contain lots of text but
I want to get down all my assumptions so my implementation idea makes
more sense, especially since my sandbox implementation is still in the
sketching :slight_smile:

On my proposal I have stated that I wish to get event based rendering
and draw regions (so XBMC does not have to render the full screen on
updates.). I fear that this is the biggest hurdle and these makes sense
to tackle in one swoop.

I have made a few assumptions which I think will always hold true which
I have designed around.

A control can be valid, or invalid. By valid it is processed and has not
changed. An invalid control is in need to be both processed and is in
need to be redrawn.

A control can be redrawn without it being invalid, this happens when
another control underneath have been invalidated and forces the entire
tree to be redrawn. This makes it imperative that all calculations on a
control needs to be split from the actual drawing as much as possible to
save precious CPU cycles.

A control can only become invalidated by the following reasons. a) An
outside force have changed making it invalid. This can happen for
example when an input event occur (keyboard, mouse etc.) or data the
control depends on changes (text gets updated). b) It is in animation. A
control can be static and become animated, i.e. static but an animation
will start in X ms. This makes me think we need a notion of scheduling
an invalidation so it can be processed and redrawn.

A control is one or a group of 2D planes in a 3D world.

Ok, so with these assumptions I have a proposition that I think will
solve both generating draw regions and event based rendering.
First, any calculation that may change the control needs to be moved
into a processing method. The processing method will be able to schedule
a new invalidation while it is calculating the animation, this step
should make it possible for XBMC to go down to a complete idle state.

If a control can know its final transform when being processed, it would
be possible to create a view port that control will change when being
redrawn. Since controls live in a tree structure the control only needs
to know its fathers final transform to know its final transform. Since a
control only can be changed in the processing stage it should be safe
for a control to pass down its final transform to its children so an
invalid view port could be calculated.

For the header I have attached. CDrawRegion is basically a 2D
axis-aligned bounding box. This one should be calculated by the current
camera based on a transform of a quad.

The important part is really CGUIControl::Update. This one will (if
invalid) Animate, update its visibility and if any of these have
changed, it will calculate the proper CDrawRegion needed to be drawn for
the change. Worth to note is that in the attached GUIControl.cpp an
origin should be able to change without a control being invalid, this is
something I haven't fixed in my sandbox so please disregard that one :slight_smile:

CGUIControl::Animate is a stub and meant for the different controls to
implement.

CGUIControl::Draw will first check if its inside the given region to be
drawn. If its to be drawn it will setup its needed view port and draw.
When its drawn it will update the current draw region to only reflect
the newly calculated drawn region and will clear the old draw region
(since its possible for a control to move or become invisible the
current and old draw region may differ).

This is more or less the biggest refactor I have currently, next big one
is font rendering since its very wasteful but I'll save that one for a
separate mail (Also I am not really sure how to do that one yet so I
need to think more :slight_smile: ).

Cheers,
Tobias

ControlCleanups.h (2.55 KB)

GUIControl.cpp (1.77 KB)

Hi Tobias,

Hi!

A control can be redrawn without it being invalid, this happens when
another control underneath have been invalidated and forces the entire
tree to be redrawn. This makes it imperative that all calculations on a
control needs to be split from the actual drawing as much as possible to
save precious CPU cycles.

A control can only become invalidated by the following reasons. a) An
outside force have changed making it invalid. This can happen for
example when an input event occur (keyboard, mouse etc.) or data the
control depends on changes (text gets updated). b) It is in animation. A
control can be static and become animated, i.e. static but an animation
will start in X ms. This makes me think we need a notion of scheduling
an invalidation so it can be processed and redrawn.

This is how most toolkits do it. You don't re-draw immediately but
queue the updates up, so they can be coalesced or other optimisations
applied. I would guess though that rather than a fully event driven
model you need some hybrid to sync with the video, although putting
the video in a separate layer also means the widget rendering doesn't
necessarily need to sync with it.

Another (simple?) notion is that a widget can indicate whether or not
it is opaque, in which case you could always ignore the rectangular
region underneath it. That might require theme support/require a
simplified theme that just uses flat and opaqute overlays if there
just isn't enough pixel power left.

A control is one or a group of 2D planes in a 3D world.

Ok, so with these assumptions I have a proposition that I think will
solve both generating draw regions and event based rendering.
First, any calculation that may change the control needs to be moved
into a processing method. The processing method will be able to schedule
a new invalidation while it is calculating the animation, this step
should make it possible for XBMC to go down to a complete idle state.

If a control can know its final transform when being processed, it would
be possible to create a view port that control will change when being
redrawn. Since controls live in a tree structure the control only needs
to know its fathers final transform to know its final transform. Since a
control only can be changed in the processing stage it should be safe
for a control to pass down its final transform to its children so an
invalid view port could be calculated.

For the header I have attached. CDrawRegion is basically a 2D
axis-aligned bounding box. This one should be calculated by the current
camera based on a transform of a quad.

The important part is really CGUIControl::Update. This one will (if
invalid) Animate, update its visibility and if any of these have
changed, it will calculate the proper CDrawRegion needed to be drawn for
the change. Worth to note is that in the attached GUIControl.cpp an
origin should be able to change without a control being invalid, this is
something I haven't fixed in my sandbox so please disregard that one :slight_smile:

Well unless you're rendering to a separate full-sized off-screen
raster, why wouldn't a move invalidate it? Or is that your plan? I
guess this is another option if it wasn't, although it is pretty heavy
on memory use, and can be slower anyway with animations if you're
updating areas that are not visible. As below, it might be an option
for large areas of text though.

CGUIControl::Animate is a stub and meant for the different controls to
implement.

CGUIControl::Draw will first check if its inside the given region to be
drawn. If its to be drawn it will setup its needed view port and draw.
When its drawn it will update the current draw region to only reflect
the newly calculated drawn region and will clear the old draw region
(since its possible for a control to move or become invisible the
current and old draw region may differ).

Hmmm.

Why are you using a float for time? The toolkit uses an unsigned int
doesn't it (unless i'm looking at the wrong codebase)?

How does this stuff fit within the current drawing methods, like
DoRender? This already cascades the widget transforms (I think?) so
the rendering context is ready at each level, shouldn't this just be
re-used? And all the widgets have their own transforms/etc.

There is already a SetInvalid() method - although it just invalidates
the whole widget for both rendering and geometry changes. Have you
considered just adding a dirty region to the/another setinvalid
method, which simply marks the region for redraw? When it comes to
rendering the dirty regions can be checked to see if any work is
required and then rendering performed in the right order. It's a bit
more complicated than that, but not egregiously so.

Might be worth looking at existing toolkits since they pretty much all
have to solve this problem and there's a lot of them out there - but
some of them go overboard on complexity so aren't worth looking at,
and I suspect most of the GL type ones don't optimise much. The swing
toolkit has a pretty simple invalidate/update pipeline that might be
good for some ideas (it's somewhat nicer and easier to understand than
something like gtk, and well, i've been using lately). e.g.
http://www.pushing-pixels.org/?p=68 The key is probably cascading
the dirty regions up the hierarchy without doing excess work - but
that's exactly what a tree is good at.

Obviously you want to change the existing code as little as possible -
and from what i've seen it seems to have plenty already done, but the
basic rendering will need to know about invalid/dirty regions.
Although as a first cut you might just be able to use graphics context
clipping regions without actually having to change the rendering code
itself much. And then you can just focus on making sure the correct
regions are marked dirty in the first place. Changing various widgets
to only render the changed region(s) should then be fairly
straightforward.

The textbox seems(?) to mix some animation stuff with the rendering
function, things like that will probably be the biggest gotchas since
the guarantee that the rendering function will be called every frame
will have to go.

Large areas of text might be tricky - particularly for scrolling and
so on you don't want to re-render every letter. Perhaps the whole
area could be pre-rendered, then scrolling is just a move with an
exposed region redraw, and the actually rendering is just using a
simple raster.

Any thoughts on the video overlay stuff yet? That's probably the most
important (imho, anyone's opinion differ?) for noticeable performance
improvements of the actual video.

Michael

tor 2010-05-20 klockan 14:50 +0930 skrev Michael Zucchi:

Hi Tobias,

> Hi!
>

> A control can be redrawn without it being invalid, this happens when
> another control underneath have been invalidated and forces the entire
> tree to be redrawn. This makes it imperative that all calculations on a
> control needs to be split from the actual drawing as much as possible to
> save precious CPU cycles.
>
> A control can only become invalidated by the following reasons. a) An
> outside force have changed making it invalid. This can happen for
> example when an input event occur (keyboard, mouse etc.) or data the
> control depends on changes (text gets updated). b) It is in animation. A
> control can be static and become animated, i.e. static but an animation
> will start in X ms. This makes me think we need a notion of scheduling
> an invalidation so it can be processed and redrawn.

This is how most toolkits do it. You don't re-draw immediately but
queue the updates up, so they can be coalesced or other optimisations
applied. I would guess though that rather than a fully event driven
model you need some hybrid to sync with the video, although putting
the video in a separate layer also means the widget rendering doesn't
necessarily need to sync with it.

Yeah this is something that needs to be thought through, because we
don't want the event driven model to destroy video sync against display
sync, which has been worked so hard to achieve good already.

Another (simple?) notion is that a widget can indicate whether or not
it is opaque, in which case you could always ignore the rectangular
region underneath it. That might require theme support/require a
simplified theme that just uses flat and opaqute overlays if there
just isn't enough pixel power left.

Thats a great idea, most of the time this could probably be rather
automagic since it shouldn't be impossible to extrapolate if a quad is
opaque or not.

Only problem I see is that we would need to walk front to back to know
which control is ontop of which. Although perhaps a more advanced
CDrawRegion could solve this by creating a vector of the drawregions and
we just walk backward on that vector instead?

> A control is one or a group of 2D planes in a 3D world.
>
> Ok, so with these assumptions I have a proposition that I think will
> solve both generating draw regions and event based rendering.
> First, any calculation that may change the control needs to be moved
> into a processing method. The processing method will be able to schedule
> a new invalidation while it is calculating the animation, this step
> should make it possible for XBMC to go down to a complete idle state.
>
> If a control can know its final transform when being processed, it would
> be possible to create a view port that control will change when being
> redrawn. Since controls live in a tree structure the control only needs
> to know its fathers final transform to know its final transform. Since a
> control only can be changed in the processing stage it should be safe
> for a control to pass down its final transform to its children so an
> invalid view port could be calculated.
>
> For the header I have attached. CDrawRegion is basically a 2D
> axis-aligned bounding box. This one should be calculated by the current
> camera based on a transform of a quad.
>
> The important part is really CGUIControl::Update. This one will (if
> invalid) Animate, update its visibility and if any of these have
> changed, it will calculate the proper CDrawRegion needed to be drawn for
> the change. Worth to note is that in the attached GUIControl.cpp an
> origin should be able to change without a control being invalid, this is
> something I haven't fixed in my sandbox so please disregard that one :slight_smile:

Well unless you're rendering to a separate full-sized off-screen
raster, why wouldn't a move invalidate it? Or is that your plan? I
guess this is another option if it wasn't, although it is pretty heavy
on memory use, and can be slower anyway with animations if you're
updating areas that are not visible. As below, it might be an option
for large areas of text though.

Sorry, I was probably a bit unclear here.

My thinking here is that a control is valid when its own relative
transform is correct. This means that a fathering control can change
while my transform is still valid. The final transform will be different
though.

For example, a Textured Quad could be translated 20 pixels to the right
and the fatering control at 0, this makes final transform 20. If the
fathering control is animated and moved to 20 pixels (fathering control
is invalid). The textured quads relative transform is still 20 pixels
and thus valid, however the update will tell that the origin has changed
and a new final transform need to be calculated, so while the textured
quad is valid the new final transform is 40 pixels to the screen. So it
needs to update the drawregion here. I guess this idea of valid and
invalid might be different from other toolkits though?

> CGUIControl::Animate is a stub and meant for the different controls to
> implement.
>
> CGUIControl::Draw will first check if its inside the given region to be
> drawn. If its to be drawn it will setup its needed view port and draw.
> When its drawn it will update the current draw region to only reflect
> the newly calculated drawn region and will clear the old draw region
> (since its possible for a control to move or become invisible the
> current and old draw region may differ).

Hmmm.

Why are you using a float for time? The toolkit uses an unsigned int
doesn't it (unless i'm looking at the wrong codebase)?

Ooops, was a miss :slight_smile: I wrote the sandbox without looking to hard on the
exact parameters of the original. It will be int in the final one :slight_smile:

See now I did Draw/Render mixup aswell :S

How does this stuff fit within the current drawing methods, like
DoRender? This already cascades the widget transforms (I think?) so
the rendering context is ready at each level, shouldn't this just be
re-used? And all the widgets have their own transforms/etc.

Yeah, this widget transform cascade stuff is what I wish to move to
update instead of being in the rendering actually. Since a render can
happen several times between updates theres no need to recalc the final
transform.

There is already a SetInvalid() method - although it just invalidates
the whole widget for both rendering and geometry changes. Have you
considered just adding a dirty region to the/another setinvalid
method, which simply marks the region for redraw? When it comes to
rendering the dirty regions can be checked to see if any work is
required and then rendering performed in the right order. It's a bit
more complicated than that, but not egregiously so.

Might be worth looking at existing toolkits since they pretty much all
have to solve this problem and there's a lot of them out there - but
some of them go overboard on complexity so aren't worth looking at,
and I suspect most of the GL type ones don't optimise much. The swing
toolkit has a pretty simple invalidate/update pipeline that might be
good for some ideas (it's somewhat nicer and easier to understand than
something like gtk, and well, i've been using lately). e.g.
http://www.pushing-pixels.org/?p=68 The key is probably cascading
the dirty regions up the hierarchy without doing excess work - but
that's exactly what a tree is good at.

Hmm, interesting. Need to read through that more, perhaps its easier to
use. Quite interesting to see that its calculating the dirty region on
invalidation instead.

Only problem I see with that (might be me missing how its working
though?) is that a draw can't change the region needed for the next
draw.

Obviously you want to change the existing code as little as possible -
and from what i've seen it seems to have plenty already done, but the
basic rendering will need to know about invalid/dirty regions.
Although as a first cut you might just be able to use graphics context
clipping regions without actually having to change the rendering code
itself much. And then you can just focus on making sure the correct
regions are marked dirty in the first place. Changing various widgets
to only render the changed region(s) should then be fairly
straightforward.

Yeah, I would think that would get us 90% on the way. I know jmarshall
(guilib mastermind) have expressed that he would love a way to limit
regions in 3D aswell for certain controls, but thats nothing thats
important here as a first step (probably just sane to add when controls
are able to be buffered into an FBO, then its trivial to add region in
3D since its just a texture.)

The textbox seems(?) to mix some animation stuff with the rendering
function, things like that will probably be the biggest gotchas since
the guarantee that the rendering function will be called every frame
will have to go.

Large areas of text might be tricky - particularly for scrolling and
so on you don't want to re-render every letter. Perhaps the whole
area could be pre-rendered, then scrolling is just a move with an
exposed region redraw, and the actually rendering is just using a
simple raster.

This is the plan I had with text actually. As it is now every glyph gets
rendered each frame, with no buffering what so ever. So I planned on
adding a buffering stage, where something like BufferText is called
whenever the text has changed. The DrawText could take an ID then and a
limiting factor (the exposed region for example).

I think that the pre drawn area with just a scrolling as you suggest is
the most interesting in term of performance, however it will take a bit
more memory.

I think that it would be nice to make this a tad abstracted since its
not necessarily best to pre-render. For example on the xbox were memory
was incredibly limiting and gpu horsepower was "plentiful", the role
were reversed. However I doubt that ratio will come back since memory is
so cheap nowadays :slight_smile:

tor 2010-05-20 klockan 14:50 +0930 skrev Michael Zucchi:

Another (simple?) notion is that a widget can indicate whether or not
it is opaque, in which case you could always ignore the rectangular
region underneath it. That might require theme support/require a
simplified theme that just uses flat and opaqute overlays if there
just isn't enough pixel power left.

Thats a great idea, most of the time this could probably be rather
automagic since it shouldn't be impossible to extrapolate if a quad is
opaque or not.

Only problem I see is that we would need to walk front to back to know
which control is ontop of which. Although perhaps a more advanced
CDrawRegion could solve this by creating a vector of the drawregions and
we just walk backward on that vector instead?

Well, I presume the widgets are depth ordered. You just walk up the
ancestral tree and along the way every parent is lower. For siblings
that can overlap, yeah you have to walk down the tree, but it becomes
a range search and isn't (terribly) expensive if each level fully
bounds it's children (which i presume it does?)

If they're not depth ordered/using GL for depth, then this obviously
becomes a bit more complicated - but not a lot. You still have to
find the `column' of widgets that intersect the dirty region(s), and
then just render them in the right order (and simply start at the
highest opaque one if that info is available, and ignore the rest).

Sorry, I was probably a bit unclear here.

My thinking here is that a control is valid when its own relative
transform is correct. This means that a fathering control can change

[...]

quad is valid the new final transform is 40 pixels to the screen. So it
needs to update the drawregion here. I guess this idea of valid and
invalid might be different from other toolkits though?

The last toolkit i used was wpf and I though that it meant something
different there - but to be honest it was so badly documented who
really knew.

But whatever it's called, it'll need some separation of 'needs
geometry updated' and 'needs to be redrawn' otherwise things turn into
a mess. e.g. you don't want a draw operation queuing more draw
operations if you've just spent a whole bunch of cycles working out
what optimally needs to be drawn.

In your example, moving the container geometry automatically marks
everything in that bounding box dirty (and the bounding box of where
it was before), so it doesn't have to do any further dirty
calculations for subregions within those. i.e. the 'setinvalid' would
also have to setup a few dirty regions - the union of the old and new
bounding box (i.e. where it is now, and what it just exposed).

How does this stuff fit within the current drawing methods, like
DoRender? This already cascades the widget transforms (I think?) so
the rendering context is ready at each level, shouldn't this just be
re-used? And all the widgets have their own transforms/etc.

Yeah, this widget transform cascade stuff is what I wish to move to
update instead of being in the rendering actually. Since a render can
happen several times between updates theres no need to recalc the final
transform.

I saw .transform is used for mouse lookups - so is this just
calculated when the geometry changes or dynamically? I also saw
'origin' coordinates, are they global/screen coordinates?

Might be worth looking at existing toolkits since they pretty much all
have to solve this problem and there's a lot of them out there - but
some of them go overboard on complexity so aren't worth looking at,
and I suspect most of the GL type ones don't optimise much. The swing
toolkit has a pretty simple invalidate/update pipeline that might be
good for some ideas (it's somewhat nicer and easier to understand than
something like gtk, and well, i've been using lately). e.g.
http://www.pushing-pixels.org/?p=68 The key is probably cascading
the dirty regions up the hierarchy without doing excess work - but
that's exactly what a tree is good at.

Hmm, interesting. Need to read through that more, perhaps its easier to
use. Quite interesting to see that its calculating the dirty region on
invalidation instead.

Remember 'invalidate' above means 'area that needs repainting', not
geometry calculations.

Only problem I see with that (might be me missing how its working
though?) is that a draw can't change the region needed for the next
draw.

Well that's not a problem, that's by design. 'Render' can't go around
changing what it needs to do, since it could get into a loop, and
someone's just spent a bunch of time working out what should be drawn
too.

to only render the changed region(s) should then be fairly
straightforward.

Yeah, I would think that would get us 90% on the way. I know jmarshall
(guilib mastermind) have expressed that he would love a way to limit
regions in 3D aswell for certain controls, but thats nothing thats
important here as a first step (probably just sane to add when controls
are able to be buffered into an FBO, then its trivial to add region in
3D since its just a texture.)

Hmm, is that just clipping or something more complicated? I can't
imagine there's much you can optimise out of 3d stuff outside of the
2d screen bounding boxes?

The textbox seems(?) to mix some animation stuff with the rendering
function, things like that will probably be the biggest gotchas since
the guarantee that the rendering function will be called every frame
will have to go.

Large areas of text might be tricky - particularly for scrolling and
so on you don't want to re-render every letter. Perhaps the whole
area could be pre-rendered, then scrolling is just a move with an
exposed region redraw, and the actually rendering is just using a
simple raster.

This is the plan I had with text actually. As it is now every glyph gets
rendered each frame, with no buffering what so ever. So I planned on
adding a buffering stage, where something like BufferText is called
whenever the text has changed. The DrawText could take an ID then and a
limiting factor (the exposed region for example).

I think that the pre drawn area with just a scrolling as you suggest is
the most interesting in term of performance, however it will take a bit
more memory.

No, you misread me. You just render the area basically visible as you
say above. But when you scroll within the bounds of that area you
just move the data that has moved, and then render the newly 'exposed'
regions. You probably wouldn't render all of the text to a bitmap
even if you had unlimited memory because it adds a lot of heavy
burstiness to the work load, which you particularly don't want with a
realitme system.

I think that it would be nice to make this a tad abstracted since its
not necessarily best to pre-render. For example on the xbox were memory
was incredibly limiting and gpu horsepower was "plentiful", the role
were reversed. However I doubt that ratio will come back since memory is
so cheap nowadays :slight_smile:

Should be pretty trivial to hide an abstraction like that. Just need
to keep it in mind when designing the rest - but I doubt it requires
anything special to keep in mind.