U-Boot Device Tree Overlays -- Where are we heading?

Hi all,

given that the current trend seems to be to let U-Boot handle all the DT overlay stuff, I have a few general questions about what this all means.

I agree in principle, that having DT stuff handled by the bootloader is the right thing to do. But there are quite a few downsides to this. Are they being addressed, and how?

  1. If the bootloader collects the overlays into a final DT blob, there is no way to control the order of initialization from within Linux. With the old system, I could load an overlay after doing some initialization, even after having started a program. This allowed putting the hardware into a certain state before giving control to a driver. I don’t see a way to do this with the new system anymore. Is there an alternative?

I have a concrete example: The timer driver doesn’t initialize when there’s no functional clock. If I want to clock one of the AM335x timers from an external clock fed in via TCLKIN, I now have to make sure the clock runs when Linux boots. What if my external clock source needs to be initialized first, before it produces a clock signal? How do I ensure that this initialization happens before the timer driver initializes?

(OK, I guess in this particular example, it would be a possibility to let the timer driver initialize the timer with an internal clock, and later switch to an external clock when it is available. But that raises the question how to control a clock muxer programmatically from user space, instead of having it set up via the DT.)

  1. The U-Boot output isn’t accessible from Linux after booting. The only way to capture it is to have a debug terminal connected on bootup. While I can do that in the lab, it doesn’t help me when I have to collect diagnostic information from a (remote) user. Is there any way to capture U-Boot output to be used from within Linux after booting, or is there activity to implement this? Granted, this isn’t specific for DT Overlays, but the more functionality is covered by U-Boot, the more important this point becomes.

  2. The DT source files are part of the Linux kernel source tree. If U-Boot is now responsible for the DT stuff, that seems misplaced. Do I have to expect a wholesale transplant of DT sources to the U-Boot source tree?

  3. Is there any effort to reduce the confusion that this change is causing? I mean, some kind of explanatory page that doesn’t mention it between the lines in passing, but actually has a chance to get found via Google, and tells the story with some clarity, like “Oi people, if you’re doing DT overlays, this is what happens right now, this is where we are going, this is what it means to you, here’s the information you should read, and don’t let anybody confuse you who talks about kernels 4.9 downwards.”

  4. It is not clear to me if this actually makes the DT stuff readonly by the kernel. Or are there still possibilities to modify things after boot time, i.e. by working with the /sys filesystem. Or is all of that now under driver control, and therefore out of DT territory? In general. I have found it very difficult to collect reliable information about the relationship between the DT and the stuff one can see and do in /sys. Are there any (up-to-date) places where this is properly documented?

Sorry for the length of this post, but I guess I need to get a full picture first, rather than asking about details.

Cheers
Stefan

Hi all,

given that the current trend seems to be to let U-Boot handle all the DT
overlay stuff, I have a few general questions about what this all means.

I agree in principle, that having DT stuff handled by the bootloader is the
right thing to do. But there are quite a few downsides to this. Are they
being addressed, and how?

1. If the bootloader collects the overlays into a final DT blob, there is no
way to control the order of initialization from within Linux. With the old
system, I could load an overlay after doing some initialization, even after
having started a program. This allowed putting the hardware into a certain
state before giving control to a driver. I don't see a way to do this with
the new system anymore. Is there an alternative?

I have a concrete example: The timer driver doesn't initialize when there's
no functional clock. If I want to clock one of the AM335x timers from an
external clock fed in via TCLKIN, I now have to make sure the clock runs
when Linux boots. What if my external clock source needs to be initialized
first, before it produces a clock signal? How do I ensure that this
initialization happens before the timer driver initializes?

(OK, I guess in this particular example, it would be a possibility to let
the timer driver initialize the timer with an internal clock, and later
switch to an external clock when it is available. But that raises the
question how to control a clock muxer programmatically from user space,
instead of having it set up via the DT.)

Set the clock in the overlay... Or the driver needs to be fixed..

2. The U-Boot output isn't accessible from Linux after booting. The only way
to capture it is to have a debug terminal connected on bootup. While I can
do that in the lab, it doesn't help me when I have to collect diagnostic
information from a (remote) user. Is there any way to capture U-Boot output
to be used from within Linux after booting, or is there activity to
implement this? Granted, this isn't specific for DT Overlays, but the more
functionality is covered by U-Boot, the more important this point becomes.

There's a couple ideas on this:

https://elinux.org/images/f/fd/Update_on_Shared_Logging_between_the_Kernel_and_the_Bootloader.pdf

But yes, other then parsing the output of our script
"/opt/scripts/tools/version.sh" debugging via u-boot overlays is fun..

But it's more deterministic then kernel overlays (which suffer from
race conditions..)..

3. The DT source files are part of the Linux kernel source tree. If U-Boot
is now responsible for the DT stuff, that seems misplaced. Do I have to
expect a wholesale transplant of DT sources to the U-Boot source tree?

U-Boot is currently copying DT elements from the Kernel.

If you look at our implementation, we are directly referencing the
kernel DT's along with our overlay project.

We actually aren't using U-Boot DT elements yet..

4. Is there any effort to reduce the confusion that this change is causing?
I mean, some kind of explanatory page that doesn't mention it between the
lines in passing, but actually has a chance to get found via Google, and
tells the story with some clarity, like "Oi people, if you're doing DT
overlays, this is what happens right now, this is where we are going, this
is what it means to you, here's the information you should read, and don't
let anybody confuse you who talks about kernels 4.9 downwards."

It's really v4.4.x+ should use DT u-boot overlays. There's a lot of
bugs fixed by using U-Boot Overlays, vs Kernel Overlays. (99% are due
to race conditions in the kernel when loading overlays)..

5. It is not clear to me if this actually makes the DT stuff readonly by the
kernel. Or are there still possibilities to modify things after boot time,
i.e. by working with the /sys filesystem. Or is all of that now under driver
control, and therefore out of DT territory? In general. I have found it very
difficult to collect reliable information about the relationship between the
DT and the stuff one can see and do in /sys. Are there any (up-to-date)
places where this is properly documented?

readonly? U-Boot overlay didn't change any of that.

The only thing we locked up was "slots" when u-boot overlays were
enabled, as there was no way to pass the full pin-used list..

Regards,

Thanks Robert!

Hi all,

given that the current trend seems to be to let U-Boot handle all the DT
overlay stuff, I have a few general questions about what this all means.

I agree in principle, that having DT stuff handled by the bootloader is the
right thing to do. But there are quite a few downsides to this. Are they
being addressed, and how?

  1. If the bootloader collects the overlays into a final DT blob, there is no
    way to control the order of initialization from within Linux. With the old
    system, I could load an overlay after doing some initialization, even after
    having started a program. This allowed putting the hardware into a certain
    state before giving control to a driver. I don’t see a way to do this with
    the new system anymore. Is there an alternative?

I have a concrete example: The timer driver doesn’t initialize when there’s
no functional clock. If I want to clock one of the AM335x timers from an
external clock fed in via TCLKIN, I now have to make sure the clock runs
when Linux boots. What if my external clock source needs to be initialized
first, before it produces a clock signal? How do I ensure that this
initialization happens before the timer driver initializes?

(OK, I guess in this particular example, it would be a possibility to let
the timer driver initialize the timer with an internal clock, and later
switch to an external clock when it is available. But that raises the
question how to control a clock muxer programmatically from user space,
instead of having it set up via the DT.)

Set the clock in the overlay… Or the driver needs to be fixed…

I’ve got the impression that it is the hardware that needs fixing in this example, something like a timeout on register access.

But the underlying problem is more general: It may not make sense to go ahead with initialization of some piece of hardware before some prerequisites have been provided by some other piece of software. If I can’t orchestrate this with loading device tree overlays from user space, I suppose another way needs to be devised to make sure things happen in the right order. Do I assume correctly that such a way isn’t currently available with U-Boot overlays? Are there plans or ideas for this? Perhaps I’m missing something in the DT concept that would allow proper ordering, but then forgive me for not seeing it.

I’m really concerned with the general problem. The specific timer issue can be circumvented some way or another, I’m sure. This would be simple if I could flip the clock multiplexer for the timer functional clock from user space. Do you happen to know whether and how this can be done? I know that there is a set_parent() call in the common clock framework API, but AFAIK this can only be used by drivers.

  1. The U-Boot output isn’t accessible from Linux after booting. The only way
    to capture it is to have a debug terminal connected on bootup. While I can
    do that in the lab, it doesn’t help me when I have to collect diagnostic
    information from a (remote) user. Is there any way to capture U-Boot output
    to be used from within Linux after booting, or is there activity to
    implement this? Granted, this isn’t specific for DT Overlays, but the more
    functionality is covered by U-Boot, the more important this point becomes.

There’s a couple ideas on this:

https://elinux.org/images/f/fd/Update_on_Shared_Logging_between_the_Kernel_and_the_Bootloader.pdf

But yes, other then parsing the output of our script
“/opt/scripts/tools/version.sh” debugging via u-boot overlays is fun…

But it’s more deterministic then kernel overlays (which suffer from
race conditions…)…

Thanks! So it looks like I have to wait and hope.

Don’t get me wrong, however, I generally agree that moving DT support from the Kernel to U-Boot is the right plan.

  1. The DT source files are part of the Linux kernel source tree. If U-Boot
    is now responsible for the DT stuff, that seems misplaced. Do I have to
    expect a wholesale transplant of DT sources to the U-Boot source tree?

U-Boot is currently copying DT elements from the Kernel.

If you look at our implementation, we are directly referencing the
kernel DT’s along with our overlay project.

We actually aren’t using U-Boot DT elements yet…

But I guess when the DT show moves over to U-Boot, you’ll eventually have to follow the bandwagon…

  1. Is there any effort to reduce the confusion that this change is causing?
    I mean, some kind of explanatory page that doesn’t mention it between the
    lines in passing, but actually has a chance to get found via Google, and
    tells the story with some clarity, like “Oi people, if you’re doing DT
    overlays, this is what happens right now, this is where we are going, this
    is what it means to you, here’s the information you should read, and don’t
    let anybody confuse you who talks about kernels 4.9 downwards.”

It’s really v4.4.x+ should use DT u-boot overlays. There’s a lot of
bugs fixed by using U-Boot Overlays, vs Kernel Overlays. (99% are due
to race conditions in the kernel when loading overlays)…

Yeah, whatever the version is. My point is that the docs out there are pretty confusing by now. It is hard to tell what still applies, and what is outdated. This has annoyed me for quite a while with the bone and its ecosystem, but in my impression it gets worse. If it confuses experienced people, what does it do to beginners?

  1. It is not clear to me if this actually makes the DT stuff readonly by the
    kernel. Or are there still possibilities to modify things after boot time,
    i.e. by working with the /sys filesystem. Or is all of that now under driver
    control, and therefore out of DT territory? In general. I have found it very
    difficult to collect reliable information about the relationship between the
    DT and the stuff one can see and do in /sys. Are there any (up-to-date)
    places where this is properly documented?

readonly? U-Boot overlay didn’t change any of that.

The only thing we locked up was “slots” when u-boot overlays were
enabled, as there was no way to pass the full pin-used list…

Well, what I really wanted to say is that there is no good resource (or else I haven’t found it) that describes /sys and what can be done with it, with an eye towards where the DT fits in. For example, are there any settings from the DT that can be changed dynamically by manipulating one of the many files in /sys ? How would I know or find out? What must I read to understand this?

Or, perhaps more concisely: How do I manipulate hardware settings from userspace through /sys ?

This becomes more important when I can’t load DT overlays at runtime anymore.

Cheers
Stefan

I've got the impression that it is the hardware that needs fixing in this
example, something like a timeout on register access.

But the underlying problem is more general: It may not make sense to go
ahead with initialization of some piece of hardware before some
prerequisites have been provided by some other piece of software. If I can't
orchestrate this with loading device tree overlays from user space, I
suppose another way needs to be devised to make sure things happen in the
right order. Do I assume correctly that such a way isn't currently available
with U-Boot overlays? Are there plans or ideas for this? Perhaps I'm missing
something in the DT concept that would allow proper ordering, but then
forgive me for not seeing it.

U-Boot has no concept of order, it just applies all the tree's and
jumps into the kernel.. So as far as the kernel is considered, it was
just "one" dtb blob..

I'm really concerned with the general problem. The specific timer issue can
be circumvented some way or another, I'm sure. This would be simple if I
could flip the clock multiplexer for the timer functional clock from user
space. Do you happen to know whether and how this can be done? I know that
there is a set_parent() call in the common clock framework API, but AFAIK
this can only be used by drivers.

Okay, what you describe above is something that requires kernel overlays.

"We" (me and the org) don't have the resources to fix/push that
upstream. We did what we could with what we had, then we abandoned
our kernel overlay work to just utilize U-Boot Overlays, as it solved
95% of our problems.

In the big picture, what you want is something Frank Rowand hinted as
the next steps at elc 2018:

https://youtu.be/HYdb5uimPtE

So yeah, what you need is more of the big picture, needs kernel work stuff.

> 2. The U-Boot output isn't accessible from Linux after booting. The only
> way
> to capture it is to have a debug terminal connected on bootup. While I
> can
> do that in the lab, it doesn't help me when I have to collect diagnostic
> information from a (remote) user. Is there any way to capture U-Boot
> output
> to be used from within Linux after booting, or is there activity to
> implement this? Granted, this isn't specific for DT Overlays, but the
> more
> functionality is covered by U-Boot, the more important this point
> becomes.

There's a couple ideas on this:

https://elinux.org/images/f/fd/Update_on_Shared_Logging_between_the_Kernel_and_the_Bootloader.pdf

But yes, other then parsing the output of our script
"/opt/scripts/tools/version.sh" debugging via u-boot overlays is fun..

But it's more deterministic then kernel overlays (which suffer from
race conditions..)..

Thanks! So it looks like I have to wait and hope.

Don't get me wrong, however, I generally agree that moving DT support from
the Kernel to U-Boot is the right plan.

The long term plan, has always been to move the DT"s out of the kernel
into a single "shared" repo..

It's just, that hasn't happened yet, and the KERNEL repo kinda of
became the shared repo..

> 3. The DT source files are part of the Linux kernel source tree. If
> U-Boot
> is now responsible for the DT stuff, that seems misplaced. Do I have to
> expect a wholesale transplant of DT sources to the U-Boot source tree?

U-Boot is currently copying DT elements from the Kernel.

If you look at our implementation, we are directly referencing the
kernel DT's along with our overlay project.

We actually aren't using U-Boot DT elements yet..

But I guess when the DT show moves over to U-Boot, you'll eventually have to
follow the bandwagon...

> 4. Is there any effort to reduce the confusion that this change is
> causing?
> I mean, some kind of explanatory page that doesn't mention it between
> the
> lines in passing, but actually has a chance to get found via Google, and
> tells the story with some clarity, like "Oi people, if you're doing DT
> overlays, this is what happens right now, this is where we are going,
> this
> is what it means to you, here's the information you should read, and
> don't
> let anybody confuse you who talks about kernels 4.9 downwards."

It's really v4.4.x+ should use DT u-boot overlays. There's a lot of
bugs fixed by using U-Boot Overlays, vs Kernel Overlays. (99% are due
to race conditions in the kernel when loading overlays)..

Yeah, whatever the version is. My point is that the docs out there are
pretty confusing by now. It is hard to tell what still applies, and what is
outdated. This has annoyed me for quite a while with the bone and its
ecosystem, but in my impression it gets worse. If it confuses experienced
people, what does it do to beginners?

Doc's will always be out of date. We switched to U-Boot overlays
around ELC 2017, (March 2017).. So vs's the history of the Black, it's
pretty much brand new..

The PocketBeagle was the first device that was U-Boot Overlays only..

> 5. It is not clear to me if this actually makes the DT stuff readonly by
> the
> kernel. Or are there still possibilities to modify things after boot
> time,
> i.e. by working with the /sys filesystem. Or is all of that now under
> driver
> control, and therefore out of DT territory? In general. I have found it
> very
> difficult to collect reliable information about the relationship between
> the
> DT and the stuff one can see and do in /sys. Are there any (up-to-date)
> places where this is properly documented?

readonly? U-Boot overlay didn't change any of that.

The only thing we locked up was "slots" when u-boot overlays were
enabled, as there was no way to pass the full pin-used list..

Well, what I really wanted to say is that there is no good resource (or else
I haven't found it) that describes /sys and what can be done with it, with
an eye towards where the DT fits in. For example, are there any settings
from the DT that can be changed dynamically by manipulating one of the many
files in /sys ? How would I know or find out? What must I read to understand
this?

Or, perhaps more concisely: How do I manipulate hardware settings from
userspace through /sys ?

This becomes more important when I can't load DT overlays at runtime
anymore.

Actually you can use the mainline
"/sys/kernel/config/device-tree/overlays/" path to apply overlays...

See the i2c accelerometer overlay example here:

https://github.com/e-ale/Code/tree/master/RESOURCES/pocketbeagle

sudo mkdir -p /sys/kernel/config/device-tree/overlays/accel

Regards,