Ti's RPMsg Examples Significantly Changed

Also, /dev/mem/ + mmap() are identical to the PRU in relation to having direct access to any memory address. Which is why many people have concerns when it is used.

By the way. You could lock down the PRUs the same way /dev/mem/ is locked down. /dev/mem/ requires root privileges.

Which is essentially the problem and why I do not like /dev/mem for application development. For testing out a concept or debugging kernel code, then it is a powerful tool.

There are several points about security in this discussion, and I would like to make what I think is a good point. With Linux, you can specify how a system can be accessed. Meaning, if you’re doing something that could potentially be considered a security hazard, then you can lock that system down tight as a drum.

You can do various things, such as:

  • Completely disable root
  • Limit access to a terminal( local access only )
  • Run several different IPC mechanisms for system abstraction
  • Run a chroot jail

And a lot more . . .

So in a case where you’re using the PRU’s and / or dev/mem/ for data acquisition. You do NOT allow remote logins - period. Then the data is perhaps saved to a remote NFS share. Then, to maintain your “IoT” system, you log in through a physical serial connection. Pretty simple.

But this is just one of many potential security strategies. However, it should be realized that when a system is internet facing. There will always be a risk . . .

Reading virtio docs, I came across two sentences that may provide a way forward:

  1. "Virtual queues, being virtual, are actually implemented as rings to traverse the guest-to-hypervisor transition. But this could be implemented any way, as long as both the guest and hypervisor implement it in the same way.”
  2. "virtio has also been used in High-Performance Computing (HPC) research to develop inter-virtual machine (VM) communications through shared memory passing."

@Suman, would it be possible to configure the memory resource used by Vrings (possibly DDR memory) so that the PRU memory/shared memory can be reserved as direct shared memory between PRU and ARM processors? The traditional message transported over virtualqueues will be use to setup/configure the shared memory. My purpose is use zero copy techniques by using direct memory sharing between PRU and ARM and avoid the need for vring message passing which require (topic, src, dst, *data & len) parameters for each message transfer. In addition, pru_rpmsg_send does a lot of work, such as pru_virtqueue_get_avail_buf(), preparing the message buffer and then finally pru_virtqueue_kick(). Using direct shared memory, both PRU and ARM can define a message structure/protocol that solves this one use case without breaking the existing framework.

If RPMSG implements mmap(), then we might have a solution that should works for TJF.

One additional thought; if RPMSG embraces zero copy, then this could be extended to eventually support vmsplice which would make for some very powerful features.

Regards,
John

Hello Przemek!

Thanks for joining this discussion.

This is a valid design pattern and you did very impressive things with
it, but it’s not compatible with the basic design of Linux hardware
integration: I don’t know of a single mainline kernel driver designed
this way.

What are you talking about here? Is it libpruio? libpruio isn’t a kernel driver, it’s a userspace library. Jason Kridner wanted to make it mainline, but nothing happened yet. Currently it’s an optional OpenSource package.

… e.g. by disallowing full control of the entire physical address space.

How? We’re talking about the PRUSS (= Programable Realtime Unit SubSystems). They haven’t been designed with security in mind. There’s no way to disallow control of any address space. Again, the PRUSS can access !ALL! CPU memory, indeed also kernel space.

The only way to protect a system against PRU malware is to disable the PRUSS. Not only once, the kernel has to disable them again and again on a regular basis, before any virus can get loaded and executed on any PRU. I don’t see any other way to gain classic Linux security on a board with PRUSS. This is how a secure standard image should work: no PRUSS support at all.

Users who want to benefit from the PRUSS should get informed about the riscs and should install optional drivers to get PRUSS support. I cannot understand how a PRUSS driver can find its way to a standard installation. And I cannot understand how this driver can be enabled by default. And much more, I canot understand why this default driver wasn’t designed with security in mind, in order to work around the hardware unsecurity. This is a massive safety risc, implemented in all TI kernels since major version 4, threatening all SoC families.

On BB it’s pretty bad: the default cape_universal overlay enables the PRUSS and loads the remoteproc driver. And this driver autoloads any malware at boot time. Your system will be hacked before you see any prompt. An agressor only needs to copy a single file and the operation system in standard configuration does the rest, loading and starting the malware at boot time. Not just a simple userspace virus. No, well defined interfaces open doors for PRU malware to infect kernel space.

I’m no expert in virus development, but I’m sure it can work as simple as

  1. PRU searches the CPU memory for RPMsg driver instructions.
  2. PRU overrides these instructions by ARM malware.
  3. PRU triggers a message.
  4. Now the ARM malware is running in kernel space.
    Each interface, like vring or even a simple interrupt handler, is an open door which a PRU virus can use to implement and trigger malware on the ARM side. With this in mind, we all should reconsider the question:

Do we really need PRU support from kernel space? If so, is it all of us?

William said:

The proper solution is to make the remoteproc project as if it’s a working module NOW. Loadable module, that can play nice with the rest of the module world. Then mark it as experimental. Worry about the actual remoteproc implementation after that.

BR

@John

If RPMSG implements mmap(), then we might have a solution that should works for TJF.

You’re again thinking in the wrong direction. It doesn’t matter what you or anybody else add to this framework. It’ll never work for me, since I care about security. The only way to gain a little bit of safety is to remove unsave features.

Regarding PRUSS, for me the best security strategy is that PRU code should never be able to estimate how the software on the ARM side is working. And the number of options to trigger software on the ARM by PRU should be as small as possible, in best case no options at all. (Even then a PRU virus can use brute force attacks to implement kernel or userspace malware, but such a virus activity is more likely to get found. A kernel space virus can work for a long period of time before you even notice it.)

BR

@TJF, I didn’t copy the PRU developers because I believe there is nothing in your response that would be of interest to them.

It is hard to keep up with your changing priorities. Addressing your security issues, you never answered my original question. How are you going to install PRU firmware without root permission. This is what makes the PRU secure. Without root permission, you cannot replace the PRU firmware, period. Now it is up to the PRU firmware developer to make sure their code does not compromise system security and this is true of any kernel module developed to run on the ARM processor.

You originally claimed that RemoteProc/RPMSG was bloated and slow, so I proposed a solution to address this issue by using zero copy techniques, but now this is no longer important to you.

It sounds like you are only interested in getting libpruio into mainline and this seems to be the basis for all your arguments. With that goal in mind, you are using logic to try and back into that conclusion, but your logic doesn’t make sense. I am trying to help you here, but you are not giving me anything I can work with.

Regards,
John

Good morning John!

It is hard to keep up with your changing priorities.

My priorities are

  1. Security
  2. Speed
  3. Resources
    That didn’t change for years.

What you mean is that my focus changed in this discussion. Before that discussion, I didn’t know much about that framework. I read just enough to find out that it doesn’t adress my needs, too slow, too bloated. Then, in this discussion, I learned more about the project and its features, and I found out that it also doesn’t adress my first priority. So I changed my focus from minor to major priority. Since we’re now at the highest priority, there wont be any further changes.

Addressing your security issues, you never answered my original question. How are you going to install PRU firmware …

I wrote it several times, libpruio uses libprussdrv to load the firmware.

This is what makes the PRU secure. Without root permission, you cannot replace the PRU firmware, period.

Period? A standpoint is an intellectual horizon with the radius null.

An agressor needs physical access to the system, but no root permission on that system.

  • When the BB boots from SD card, the agressor puts this card in his laptop where he has root permission to copy the malware to /lib/firmware/am335x-pru$[0|1]-fw on the SD partition.
  • When the BB boots from EMMC, the agressor inserts a bootable SD card and presses the boot button on start up. The system boots from SD and it neither needs a monitor nor a keyboard, the copy process can get executed by a boot script to place the malware on the EMMC partition.
    Welcome to reality!

Sure, any virus can get installed that way. But why should it be more easy to install a worst case kernel virus than to install a simple userspace malware? An why should the operationg system help to get it running?

You originally claimed that RemoteProc/RPMSG was bloated and slow, so I proposed a solution to address this issue by using zero copy techniques, but now this is no longer important to you.

This is a proposal on a detail. Currently it makes no sense to work on details. At the moment, it needs a management decision

  1. General direction
  2. Milestones
  3. and then your important detail work.

It sounds like you are only interested in getting libpruio into mainline …

Is this a bad scenario? Yes, because due to all that kernel changes I’d have to spend a lot of effort to keep it up to date.

What about you? You insists on keeping that framework in mainstreams default configuration, although you know that you could use it as an optional package as well. Why? This sounds like you’re planing to use this weak point for your next malware. This sounds like you want to become an author of a Linux kernel virus.

But instead of listening to mystic sounds, we both should concentrate on facts and arguments.

BR

@TJF

What a silly argument. It is well known that when you don’t have physical security, you don’t have any security. Once you can replace the storage media, you can make the hardware do whatever you want. This is true of your laptop and your servers. Clearly by your argument, you haven’t even begun to understand security. In addition, you should reframe from impugning a person’s motivations or intentions. If you didn’t know, for stability reasons, Linux do not remove fameworks unless they have been replaced by something better and in most cases the new framework is backward compatible with the new framework. No one knows how many developers are using RemoteProc/RPMSG, so there is no ways that Linus or his deputies would permit the removal of this framework. All you can do is attempt to make it better, so stop fighting a loosing battle and join me in fixing what you don’t like.

Regards,
John

Good morning John!

What a silly argument. It is well known that when you don’t have physical security, you don’t have any security. Once you can replace the storage media, you can make the hardware do whatever you want. This is true of your laptop and your servers. Clearly by your argument, you haven’t even begun to understand security.

You’re correct, I don’t know much about the Linux terminologie and definitions. From the Linux point of view, this was evidently a silly argument. Do not fred. You’re still leading by a two-digit score in this discipline.

Does that change much? No, there’re still other ways to get firmware on the PRUSS. As a matter of fact there are solutions to use the PRUSS from user space. No example, because I don’t want to continue in publishing “how to make a virus” tutorials here. If you don’t believe that, just check out the libpruio examples. Most of them work with user privilegues, root permission is necessary for pinmuxing only.

In addition, you should reframe from impugning a person’s motivations or intentions.

It’s confusing when you talk to yourself.

If you didn’t know, for stability reasons, Linux do not remove fameworks unless they have been replaced by something better and in most cases the new framework is backward compatible with the new framework.

Obviously there’re exceptions. The PRU support is neither better nor backward compatible to previous solutions. Oh sorry, not entirely correct: indeed it has better support for PRU virus activities, targeting the kernel space directly.

No one knows how many developers are using RemoteProc/RPMSG …

This is an important point! Just a few unquantifiable developers will use it, but in the current configuration it endangers all BB systems by default.

so there is no ways that Linus or his deputies would permit the removal of this framework.

Is it mainline? I thought it’s a TI feature. Anyhow, mainline isn’t affected. This safety issue concerns boards with PRUSS, only.

All you can do is attempt to make it better …

This is making it optional, or at least making PRUSS support optional in that framework and disable it by default.

so stop fighting a loosing battle and join me in fixing what you don’t like.

That’s a damed good idea. Since here we’ve to wait for a management decision, why don’t you use that time for fixing the issue from our March discussion? Or at least answer the still open question.

BR

Good morning John!

What a silly argument. It is well known that when you don’t have physical security, you don’t have any security. Once you can replace the storage media, you can make the hardware do whatever you want. This is true of your laptop and your servers. Clearly by your argument, you haven’t even begun to understand security.

You’re correct, I don’t know much about the Linux terminologie and definitions. From the Linux point of view, this was evidently a silly argument. Do not fred. You’re still leading by a two-digit score in this discipline.

Does that change much? No, there’re still other ways to get firmware on the PRUSS. As a matter of fact there are solutions to use the PRUSS from user space. No example, because I don’t want to continue in publishing “how to make a virus” tutorials here. If you don’t believe that, just check out the libpruio examples. Most of them work with user privilegues, root permission is necessary for pinmuxing only.

The only reason you can do this from userspace is because of uio_pruss. It is the UIO driver that exposes the PRU memory to user space that creates this security hole. With RPMSG/RemoteProc, there is no uio_pruss and so you cannot install custom code to the PRU without root permission.

In addition, you should reframe from impugning a person’s motivations or intentions.

It’s confusing when you talk to yourself.

If you didn’t know, for stability reasons, Linux do not remove fameworks unless they have been replaced by something better and in most cases the new framework is backward compatible with the new framework.

Obviously there’re exceptions. The PRU support is neither better nor backward compatible to previous solutions. Oh sorry, not entirely correct: indeed it has better support for PRU virus activities, targeting the kernel space directly.

You still haven’t explained how to load custom code onto PRU using RemoteProc without root permission. Looking in /lib/firmware, user and group permissions are always root. There are only two ways to load PRU firmware and that is at boot time because of DT PRUSS driver definition or via sysfs and this requires root permission.

No one knows how many developers are using RemoteProc/RPMSG …

This is an important point! Just a few unquantifiable developers will use it, but in the current configuration it endangers all BB systems by default.

I accept that you may have a way to do this that I have not thought of, but I have yet to see you prove this.

so there is no ways that Linus or his deputies would permit the removal of this framework.

Is it mainline? I thought it’s a TI feature. Anyhow, mainline isn’t affected. This safety issue concerns boards with PRUSS, only.

As I said before, on the BeagleBoard-x15, the DSP, CortexM4 and PRU all have access to the entire memory map, so loading firmware on any of these cores must be secure. It isn’t a PRUSS issue only.

All you can do is attempt to make it better …

This is making it optional, or at least making PRUSS support optional in that framework and disable it by default.

Why, because your argument would apply to any processor, namely DSP, CortexM4, other ARM cores running a different OS or perhaps barebones, etc. As long as you need root access to load firmware onto these other cores, there is no security issue.

so stop fighting a loosing battle and join me in fixing what you don’t like.

That’s a damed good idea. Since here we’ve to wait for a management decision, why don’t you use that time for fixing the issue from our March discussion? Or at least answer the still open question.

BR

Anyway, this discussion is not productive as we are not going to find a solution if you do not want to propose solutions. Your proposed solution of making RPMSG/RemoteProc optional isn’t a solution. However, perhaps there is a way to move the DT PRUSS definition into an DT overlay and not include the PRUSS definition in the base DT tree. This would allow both uio_pruss and RemoteProc to live side by side. Perhaps this is what you should be working on. I’m sure Robert Nelson would be open to a solution like this if you found a way to make this work.

Regards,
John

Personally, I could care less how many people use remoteproc, and I do not think it’s all that important either. I do not recall ever seeing anyone ask how many people use uio_pruss either. But again, I think it’s a moot point. The better software should win out, and for those who do not agree, they will use what they want anyhow. No one can stop them.

So who is this person that needs to stop fighting and conform to someone elses wishes ? Sorry pal, others beliefs aren’t a dictatorship, and whose fighting ? Sounds like someone has built this up as something in their own mind, that it’s not.

Also, it does not matter what is mainline, and what isn’t. At least in this context to me it does not. I know that remoteproc really, has nothing to do specifically with TI hardware. I’ve seen really cool usages of remoteproc on multi core applications processors where 1 or more additional cores were used as bare metal cores. While 1 or more cores were running Linux. Personally, in my mind that’s what remteproc is all about. But what doe this have to do with a specific bit of hardware that only one company has in their processors ? Nothing . . .

Good morning John!

Anyway, this discussion is not productive as we are not going to find a solution if you do not want to propose solutions. Your proposed solution of making RPMSG/RemoteProc optional isn’t a solution. However, perhaps there is a way to move the DT PRUSS definition into an DT overlay and not include the PRUSS definition in the base DT tree. This would allow both uio_pruss and RemoteProc to live side by side. Perhaps this is what you should be working on. I’m sure Robert Nelson would be open to a solution like this if you found a way to make this work.

Sorry, I disagree again. But this time it doesn’t bother me, because your final proposal sounds like the solution most of us are looking for. Just some quotes:

Jason Kridner

I will work to enable uio_pruss functionality, and I think that is what you want, not just getting remoteproc out.

William Herman

I do not think anyone is asking to remove remoteproc, and replace it with uio_pruss. What we’ve been asking, at least I have been asking is give us the option.

Rick Mann

It sure seems to me that if both can exist in the source tree and be selected at runtime with configuration (ideally via device tree, switchable later by loading and unloading modules) …

TJF

I’d try to make both optional and choise in device tree when enabling the PRUSS.

@All

BR

Here is a conversation I had with Robert about this issue:

From: Robert Nelson <robertcnelson@gmail.com>

Subject: Re: uio_pruss coexist with remoteproc
Date: June 30, 2016 at 11:43:29 AM PDT
To: John Syne <john3909@gmail.com>

Hi John,

Hi Robert,

I’m trying to put this issue to bed. Since you had issues getting uio_pruss and remoteproc working together, I’m trying to understand the issue. My guess is that both kernel modules should build without conflict, but the DT has to choose one or the other. Is there a reason why we cannot move the PRUSS specs into a DT overlay, one for uio_pruss and one for remoteproc.

In our weekly meeting today, we talked about it for a good 30 minutes,
even had Suman come in on the line. TI is watching that thread, and
they have even more meetings scheduled.

What i would like to try to do, patch the uio_pruss driver so it can
use the same bindings as remoteproc (they use the same node label),
then users can either blacklist remoteproc or uio_pruss

I know this seemed like a painful process, but in the end it looks like this solution will work for everyone. Thank you everyone for your valuable input.

Regards,
John