RFD: Devicetree overlay code in cape's eeprom

Hello,

The recent discussion about (un)use of the eeprom on the capes
encouraged me to ask for group's opinion (and hopefully start a
discussion) about using the memory as aprovider as DT overlay code.
Currently for each cape the system needs to have a corresponding
overlay file in /lib/firmware. It may not look like a big problem, but
may be annoying for new users and is prone to mistakes. Shipping the
overlays with the system may be considered a short-term solution, but
with growing number of capes in production it will become very quickly
impossible to manage. On the other hand, each cape is required to
provide 32KB of permanent storage which is intended primarily to
contain configuration information, from which only a few bytes are
currently used by the system.

What I propose is to add to the eeprom format an option to store the
overlay code and to expand the cape manager with an ability to read
the overlays not only from the files, but also from the capes. This
would make the capes almost completely "plug&play" and relieve
distribution managers from providing overlays for all existing capes.

regards,
j.

From a Hardware production standpoint, this could be hard to maintain and control, especially at the rate at which the code changes. There would need to be a way to maintain this code separate from the production process, and therefore load it, shall we say, in the wild.

Currently these EEPROMs are write protected preventing them from being overwritten by well meaning SW and in this case, could render the board useless. That would then generate an RMA request because the board is now “broken”. There would need to be a way to remove the write protect, which there is, but that means you have to tie two points together, shorting the write protect. A function that requires some hardware knowledge on how to do that, but could be done.

All this may require redesign of these boards to ease these issues, I doubt the cape suppliers would be interesting in investing money in redesigning these boards and at the same time rendering the existing boards, incompatible. Many people have already invested in these boards.

So from a product and practicality standpoint, may I suggest that while SW can be changed every hour, hardware is not so forgiving and involves real hard cash to build, support, and maintain. Before the software goes of in these different direction to make things “easier” please keep in mind those of us who create, fund, produce, and martian this hardware. We are limited on how quickly we can change direction.

Gerald

New capes may be designed as field-upgradable, this would require
adding one jumper or a switch on the board on the WP line. Current
standard of the capes already includes a requirement for switches
which set the board address, so adding one more section will not be an
issue. Adding an ability to read the overlay code from the eeprom does
not mean that the manager must not read the overlay from files: if the
file does not exist, the manager checks the eeprom. This means that
for legacy capes and capes which will not contain the overlay code
nothing will have to be done. Soft-upgrade of the overlay code will
require only downloading the new file to /lib/firmware - exactly how
it is performed today.

I know very well that the software is currently in a very fluid form,
but it will not last forever and today is the perfect time to think
about and introduce long-term solutions that will be used in the
future. In a few months the 3.8 kernel line will stabilize, the rapid
development will move on to the newer versions and new revisions of
hardware configuration files will be necessary only when the hardware
changes or for someone who wants to live on the bleeding edge of
software development (definitely not newbies). This is where the
binding of the overlay with the HW for which it was created is most
beneficial: users will not generate RMA request because they
downloaded a wrong overlay and the board does not work anymore.
As you see I don't want to force everybody to redesign existing capes,
but to open a possibility for easier bundling of cape configuration
with the board itself.

j.

Ahh, all NEW capes. What about all the old capes? Yes i am familiar with the current standard for the capes. I wrote it.

If everyone agrees to this, then I will consider it. But I will require a LOT of convincing that nothing will happen down the road for something like Device Forest that will cause this to become an issue. I need guarantees.

The 3.8 will not stabilize. Why? Well, because we really need to move to 3.10 or 3.12 to prevent from falling behind. I wonder what changes were made in 3.10 and 3.12?

I2C is not exactly the fastest in the world. Downloading up to 32KB of data will take a little while for four capes. I hope people don’t complain about the boot time, but something tells me they will. Oh, and the virtual capes, (eMMC and HDMI) they have no EEPROM.

Everyone needs to agree to this and then everyone needs to convince me that this is the way to go and will not have to change down the road!!

Gerald

Nooo!

Being not familiar with all the 3.8 details, but having written the eeprom decoder for the 3.2 kernel, I've been wondering why the Cape EEPROM configuration scheme for the pins was abandoned so suddenly. The cape manager is not part of the kernel code, right? So we're free of kernel politics. Then why can't the cape manager decode the pin table in the eeprom and parse it as if it were a DT file? This does not give us all the nifty features of the DT system, but would probably suffice for the simple and existing capes. And new capes can choose which way to go, a DT file being present then would override the eeprom pin table.

One big annoyance I have with the BBB is backwards compatibility. Let's all use our brains better and design for compatibility !

just my 2ct.

-- Bas

Ahh, all NEW capes. What about all the old capes?

Let me quote myself:
    Adding an ability to read the overlay code from the eeprom
    does not mean that the manager must not read the overlay
    from files: if the file does not exist, the manager checks the
    eeprom. This means that for legacy capes and capes which
    will not contain the overlay code nothing will have to be done.
Old capes will not be afected at all. They will use overlays in files
in /lib/firmware.

Yes i am familiar with the current standard for the capes. I wrote it.

I studied it thoroughly and I am fully aware of its authorship :smiley:

If everyone agrees to this, then I will consider it. But I will require a
LOT of convincing that nothing will happen down the road for something like
Device Forest that will cause this to become an issue. I need guarantees.

That's why I wrote the RFD

The 3.8 will not stabilize. Why? Well, because we really need to move to
3.10 or 3.12 to prevent from falling behind. I wonder what changes were made
in 3.10 and 3.12?

That's up to the SW policy for the boards, but I think that for
inexperienced users and your own peace of mind it may be better to
have a line of stable software, which does not break things once a
week.

I2C is not exactly the fastest in the world. Downloading up to 32KB of data
will take a little while for four capes. I hope people don't complain about
the boot time, but something tells me they will. Oh, and the virtual capes,
(eMMC and HDMI) they have no EEPROM.

That's a valid argument "against". Exactly the kind I wanted to see in
this discussion.

regards,
j.

Part of my annoyance in all this is that I will do whatever everyone else agrees on as long as it makes sense from a HW/Product/Production perspective. Unfortunately agreement is something that seems to be tough to get in the free wheeling Linux world. And when someone does make the call, DT for an example, a lot of people run around with their hair on fire.

When people complain about using them as “Guinea Pigs”, that is basically what Linux is about. From a BeagleBoard standpoint, this can be fixed. We just close it off. Make releases every 6 months, and let everyone be users and do not allow any other OSs on the boards, except what we choose. Do not allow any one to choose. Do not let any other OSs onto the board. I for one, am not at all in favor of a closed system, like some other boards I can think of…

So, after reading the short paragraph above, that is the reason for the first paragraph. My charter is to protect the ability to keep the cost down on the boards, hold down the number of expensive board spins, and keep the product overall, viable. And to set the ONLY fixed entity in this whole conversation, the Hardware.

Gerald

Nooo!

Being not familiar with all the 3.8 details, but having written the eeprom
decoder for the 3.2 kernel, I've been wondering why the Cape EEPROM
configuration scheme for the pins was abandoned so suddenly. The cape
manager is not part of the kernel code, right? So we're free of kernel
politics. Then why can't the cape manager decode the pin table in the eeprom
and parse it as if it were a DT file? This does not give us all the nifty
features of the DT system, but would probably suffice for the simple and
existing capes. And new capes can choose which way to go, a DT file being
present then would override the eeprom pin table.

The pin muxing info from the cape is useless. The new kernels need
much more than that to configure peripherials. I wrote a cape contents
generator (GitHub - piranha32/beaglebone-pinmux-tool: Forget about mode numbers and setting bits for pinmuxing. Let the tool think about them.), but I
don't think it was ever useful. I don't see the fact that I wrote
software for something as an argument against moving on to something
better.

j.

OK

Gerald

Part of my annoyance in all this is that I will do whatever everyone else agrees on as long as it makes sense from a HW/Product/Production perspective. Unfortunately agreement is something that seems to be tough to get in the free wheeling Linux world. And when someone does make the call, DT for an example, a lot of people run around with their hair on fire.

We made a call and are now spending a good amount of time justifying it after the fact. It’ll be great to see the software working before making any changes to production software.

When people complain about using them as “Guinea Pigs”, that is basically what Linux is about. From a BeagleBoard standpoint, this can be fixed. We just close it off. Make releases every 6 months, and let everyone be users and do not allow any other OSs on the boards, except what we choose. Do not allow any one to choose. Do not let any other OSs onto the board. I for one, am not at all in favor of a closed system, like some other boards I can think of…

So, after reading the short paragraph above, that is the reason for the first paragraph. My charter is to protect the ability to keep the cost down on the boards, hold down the number of expensive board spins, and keep the product overall, viable. And to set the ONLY fixed entity in this whole conversation, the Hardware.

The whole idea is one that has been discussed informally for over a year now and is an idea I fully support in concept. In practicality, I think we need a reference implementation to scrutinize.

This would need to be in addition to any information currently contained within the EEPROMs and would need to be completely optional, as-in, it doesn’t mean you don’t have to create the existing pin mux/usage entries. Having any newer firmware versions in /lib/firmware would need to override what ships with the board. The idea, however, that a board developer could simply ship their board with the devicetree information in the EEPROM and enabling usage of older board software with newer cape hardware without forcing any software downloads seems a very powerful concept to me. New cape hardware could be designed to work at the moment of the first power-up without downloading any new software and without the hardware developer even needing to develop software if Linux drivers exist for the components they utilized.

This was part of the early concept and one of the reasons for the large size of the EEPROMs.

I look forward to patches and some reports about testing. Would love to see some sourcing of a cape-creator’s guide as suggested inputs to the SRM.

> Part of my annoyance in all this is that I will do whatever everyone else
> agrees on as long as it makes sense from a HW/Product/Production
> perspective. Unfortunately agreement is something that seems to be tough
> to get in the free wheeling Linux world. And when someone does make the
> call, DT for an example, a lot of people run around with their hair on
> fire.

We made a call and are now spending a good amount of time justifying it
after the fact. It'll be great to see the software working before making
any changes to production software.

> When people complain about using them as "Guinea Pigs", that
> is basically what Linux is about. From a BeagleBoard standpoint, this can
> be fixed. We just close it off. Make releases every 6 months, and
> let everyone be users and do not allow any other OSs on the boards,
> except what we choose. Do not allow any one to choose. Do not let any
> other OSs onto the board. I for one, am not at all in favor of a closed
> system, like some other boards I can think of..
>
> So, after reading the short paragraph above, that is the reason for the
> first paragraph. My charter is to protect the ability to keep the cost
> down on the boards, hold down the number of expensive board spins, and
> keep the product overall, viable. And to set the ONLY fixed entity in
> this whole conversation, the Hardware.

The whole idea is one that has been discussed informally for over a year
now and is an idea I fully support in concept. In practicality, I think we
need a reference implementation to scrutinize.

This would need to be in addition to any information currently contained
within the EEPROMs and would need to be completely optional, as-in, it
doesn't mean you don't have to create the existing pin mux/usage entries.
Having any newer firmware versions in /lib/firmware would need to override
what ships with the board. The idea, however, that a board developer could
simply ship their board with the devicetree information in the EEPROM and
enabling usage of older board software with newer cape hardware without
forcing any software downloads seems a very powerful concept to me. New
cape hardware could be designed to work at the moment of the first power-up
without downloading any new software and without the hardware developer
even needing to develop software if Linux drivers exist for the components
they utilized.

Perhaps to get around the eeprom speed problem, the cape manager could
look to see if there is a file in /lib/firmware, and if not see if there is
data on the eeprom, and if it is there copy it to /lib/firmware. That way
we only do the copy (the slow bit) once, and if a newer version is needed
then it can be placed in /lib/firmware and the eeprom version gets ignored.
For old capes without the data, if the /lib/firmware file does not exist
then the user must download it as before. In addition the eeprom never needs
to be updated.

David

Nooo!

Being not familiar with all the 3.8 details, but having written the eeprom
decoder for the 3.2 kernel, I've been wondering why the Cape EEPROM
configuration scheme for the pins was abandoned so suddenly. The cape
manager is not part of the kernel code, right? So we're free of kernel
politics. Then why can't the cape manager decode the pin table in the eeprom
and parse it as if it were a DT file? This does not give us all the nifty
features of the DT system, but would probably suffice for the simple and
existing capes. And new capes can choose which way to go, a DT file being
present then would override the eeprom pin table.

The pin muxing info from the cape is useless. The new kernels need
much more than that to configure peripherials. I wrote a cape contents

Can you elaborate a bit more? I really want to understand why it is as you say.

My cape is using the PRUSS and a lot of other peripherals like PWM, I2C, GPIO.
Has it to do with something like 'ownership' of a pin? I.e. that if one assigns a pin via the eeprom
table and sets the mux accordingly, that e.g. the pwm driver no longer can use use that signal?

Some of the signals are assigned in such a way that I can change the mux setting (on the fly)
to get the behaviour I want. With the 3.2 kernel that was no problem. I get the feeling that
with DT this isn't possible anymore, right?

generator (GitHub - piranha32/beaglebone-pinmux-tool: Forget about mode numbers and setting bits for pinmuxing. Let the tool think about them.), but I

Ah, nice tool. I'll try it with my cape. A pity I wasn't aware of its existence.

don't think it was ever useful. I don't see the fact that I wrote
software for something as an argument against moving on to something
better.

No that would make you a fool, and I guess you're far from that : ) But why are you insinuating I said something like that?

Maybe one should also consider stability and backwards compatibility as quality (your 'better') !
A problem with part of the Linux community is that they always want to make things 'better', good never being 'good enough', and thus never reaching a stable situation. Making things 'better' just for the fun of it, makes things worse for Joe Average-User. How many of the 20K+ BBBs are going to be used as cheap replacements for a BeagleBone White? But it's not! These people might want something 'good enough' right now, instead of something 'better' somewhere in the future. (Of course something better in the future is welcome, as long as there's something good enough right now ; ) !)

That's why I was putting emphasis on compatibility and stability.

-- Bas

I’d thought about of this possibility when posting the thread (relationship between cape eprom and device tree) that I suspect prompted this RFD. Here are some thoughts I’d had (that were not included in the original thread):

In general, putting info in an eprom is appropriate when the info can be
a) determined at hardware definition (compile) time, and
b) is static in nature.

  1. Eprom backwards compatibility
    The current eprom spec has the bulk of the eprom area (offset 244 for 32543 bytes) marked as “Available”. The comment in the BBB SRM says: “Available space for other non-volatile codes/data to be used as needed by the manufacturer or SW driver. Could also store presets for use by SW.”

If we alter the eprom spec to use part or all of this area for a DT file, we should consider the situation that some or all of this space has already been used by a manufacturer in an existing cape eprom. Any space dedicated to DT usage would become space that is no longer available for other uses to the manufacturer. That could make an existing eprom contents not “forward compatible” with the proposed eprom spec change.

FYI, I’ve run across at least one cape that uses some of this area for it’s own purposes.

This possibility of pre-use also implies at least some signature checking logic for a DT file in the eprom so that there is a high probability that an existing eprom will not be mistaken for one containing a DT file etc.

There are few enough capes today, many of which are manufactured by one entity, that it might be possible to quantify how many capes would be impacted by the proposed change.

This may or may not be a big consideration – I’m not passing judgment - just calling it out as one consideration.

  1. DT eprom size allocation
    How much of the eprom “Available” space would be allocated to DT storage?
    The obvious ends of the scale are 0 and “all of it”.

Future expansion proofing makes me want to back off of “all of it” (it’s always good to have some left over space in the eprom so that if nothing else you can indicate “this eprom scheme got obsolete over time go look for the expanded eprom contents”).

Can someone provide some input as to any naturally existing bounds that could be taken advantage of? For example is there a maximum DT file size?

  1. Duplicate information
    Following on from the “size” thoughts takes me back to my original thread’s topic. Ideally I’d prefer not to duplicate the eprom’s pin usage area in a more verbose format (DT source). That just seems wasteful of eprom space.

However, leveraging the existing pin map into a DT file could require a pin table to DT source translator. If we just put a DT source file in the eprom, the “translator/author” is a human. If we want to use the pin map info to create a DT file, we are lead into a software exercise that is probably harder than the storage space gain is worth. I’m uncertain as to how difficult this such a translation might be as I’ve not seen a “DT source language spec” (and thus don’t feel I’m really entitled to an opinion on that).

How hard this could be also depends on the next item –

  1. Store DT source or compiled DT file?
    If one stores a compiled file:
    Uses less space (I’m assuming that, I did not check the numbers).
    Results in a simple copy from eprom to file system.
    Since the cape manager uses compiled overlays, this avoids any need to “compile DT source on the fly” during the boot process. I initially want to shy away from on the fly compiles…. Feel to me like that could generate too much support hassle if the compile fails.

A compiled DT file is not very human friendly. But that may not be an issue as I’ve read that if you swap the input and output files to the DTC, it will “decompile a DTO into source”. I’ve not tried that – can anyone confirm that is working?

  1. DT has more than pin info
    There was a comment made in my other thread that effectively said that reading the pins from the eprom could be done, but that it was of minor value as the DT files handle more than pin configuration.

  2. HDW pin init vs device driver loading
    Being new to the DT stuff, I’ll accept 5) at face value. The additional info he mentioned as being handled in the DT file were device drivers.

Are there other things that DT is suppose to do also?
(where can one find a DT compiler input language syntax and semantics specification?)

I admit that I was thinking low level hardware pins in my orig post. I tend to think of device drivers as “the next layer up” and as part of the OS. I want my hardware to come up and be all configured for initial use (I tend to include internal eprom processing in the “hardware layer” viewpoint). A 2nd layer of abstraction is created when the OS device drivers are loaded.

With a DT source specifying both pin conifg and driver loads, it makes me wonder about what types of things need to be considered here?
I assume a DT file in an a eprom could end up referring to a device driver that no longer exists; What happens then?
Would (does?) cape manager just not load the driver or does it throw away the entire DT file as “invalid”?

Perhaps others could offer thought as to what need to be considered here.

Dave

Having just created a dts file, after wasting a day creating an EEPROM mux config, I would vote for storing the DTS file in EEPROM. But, as I posted in another discussion, this would be a voluntary move for cape designers. The change would force a rev A2 to the EEPROM contents, but version A0 and A1 would still be available as valid formats. A simple check during the boot process could determine what to do with the contents. Or why even do it at boot? Just plug in the cape, run “rebuild-cape-drivers” script, reboot, and now you have a working cape.

I don’t see any hardware changes being required, the biggest dts file size seems to be around 16 kB, so 32kB is enough space.

I like the idea of using a local dtbo file, if available, and only use the EEPROM dts file for the initial loading and dtbo creation, if necessary.

This reminds me of a USB cellular modem I had. It enumerated as two devices, one was the modem and the other was a flash drive that contained the driver. This enable me to “get on the air” and then I could update the driver over the new internet connection.

This is a very good discussion, as it is bringing up things that I did not think of initially.
Louis