Using the PRU to setup PWMSSx/eHRPWMx, problem with pwmssx_tbclken (0x44E1_0664)

Burak_Soner1 · June 28, 2018, 9:33pm

Hello,

I’ve been trying to set up the eHRPWMx outputs on my BeagleBone Black (started out with eHRPWM1 → P9_14) using only the PRU (no intervention/help from the Linux side, just setting registers on PRU code).

I’ve got a BeagleBone Black Wireless with the kernel (uname -r output) 4.9.83-bone10. I’ve posted this here since I think this is irrelevant of the bone being wireless, should apply to all BBBs (there’s also a sad story on the TI forums with a guy trying to solve this and not getting anywhere useful with a BBB, so I think it’s relevant):

https://e2e.ti.com/support/arm/sitara_arm/f/791/t/488134?Unable-to-Run-PWMSS0-PWMSS1-and-PWMSS2-from-PRU-on-Beagle-Bone-Black-

https://e2e.ti.com/support/arm/sitara_arm/f/791/t/478720?Beagle-Bone-Black-PRU-not-able-to-initialize-PWMSS0-or-PWMSS1-

To enable the eHRPWM from the PRU code I do the following in order (the GLB_ registers are just eHRPWM registers in correct addresses):

/////////////////////////////////////////////

///////////////// Code /////////////////
/////////////////////////////////////////////

#define CM_PER_BASE 0x44E00000
#define CM_EPWMSS1_REG ((volatile uint32_t *)(CM_PER_BASE + CM_PER_EPWMSS1_CLKCTRL))
#define CM_EPWMSS0_REG ((volatile uint32_t *)(CM_PER_BASE + CM_PER_EPWMSS0_CLKCTRL))
#define CM_EPWMSS2_REG ((volatile uint32_t *)(CM_PER_BASE + CM_PER_EPWMSS2_CLKCTRL))

#define PWMSS1_BASE 0x48302000
#define SYSCONFIG_REG ((volatile uint32_t *)(PWMSS1_BASE + 0x4))
#define CLKCONFIG_REG ((volatile uint32_t *)(PWMSS1_BASE + 0x8))
#define CLKSTATUS_REG ((volatile uint32_t *)(PWMSS1_BASE + 0xC))

…

// enable OCP (the below registers are defined in the well-known “pru_cfg.h”)
CT_CFG.SYSCFG_bit.STANDBY_INIT = 0;

…

(*CM_EPWMSS1_REG) = 0x2;

(*CLKCONFIG_REG) |= 0x100;

(*GLB_EHRPWM1_TBCTL) = 0xC030;
(*GLB_EHRPWM1_TBPHS) = 0;
(*GLB_EHRPWM1_TBCNT) = 0;
(*GLB_EHRPWM1_TBPRD) = 1000;
(*GLB_EHRPWM1_CMPA) = 500;
(*GLB_EHRPWM1_CMPCTL) = 0x0;

(*GLB_EHRPWM1_AQCTLA) = 0x32;

…

/////////////////////////////////////////////

///////////////// Code /////////////////
/////////////////////////////////////////////

Other than the “signal-the-host” stuff and some other irrelevant things this is the PRU main(). I use the default dtbo file named “BB-PWM1-00A0.dtbo” to test eHRPWM1A on P9_14. After a clean boot, if I run this, it doesn’t work → no output on the pin. I’ve pulled out all the registers that have the name pwm in it from the TRM and checked their values to see what’s wrong. There’s this pwmss1_tbclken bit of the pwmss_ctrl register in control module registers which just won’t turn on by action from the PRU (pwmss_ctrl will stay 0 even though I tried setting it to 0x7 to enable all 3 tbclken’s).

The reason why I think that pwmss1_tbclken was the problem → a while later I realized if after a clean boot, if I do the following on bash (on Linux side):

sudo su
sudo sh -c “echo 0 > /sys/class/pwm/pwmchip0/export”

sudo sh -c “echo 1000000 > /sys/class/pwm/pwmchip0/pwm0/period”

sudo sh -c “echo 500000 > /sys/class/pwm/pwmchip0/pwm0/duty_cycle”

sudo sh -c “echo 1 > /sys/class/pwm/pwmchip0/pwm0/enable”

exit

I get PWM output on P9_14 (like I knew I should). Right after this I check pwmss1_tbclken, it’s set! After this I run the exact same PRU code and I’ve got full control over the eHRPWM module (except pwmss1_tbclken, can’t change that from the PRU). Rerunning it with different prd and cmp values give different pwm waveforms etc.

It’s almost as if the Linux side driver does some kind of enable-control-of-CM_PER_EPWMSS1_CLKCTRL setting which I’m missing. I’ve checked the kernel driver from the repo below but haven’t been able to identify that sort of a register setting:

https://github.com/beagleboard/linux/blob/4.9/drivers/pwm/pwm-tiehrpwm.c

Is there such a setting that I’m missing? Some register that needs to be set before the PRU can access pwmss_ctrl? Or is it just that the PRU is not allowed to do what I’m trying to do? Running the Linux side script once at power-up is ofcourse a workaround that might work but I was thinking of using this in an industrial setting so direct PRU control of the peripheral from power-on onwards would be great.

Thanks in advance for the help,
Burak

Burak_Soner1 · June 28, 2018, 9:58pm

sorry for the typo,

…kind of enable-control-of-pwmss_ctrl setting which I am missing…

RobertCNelson · June 29, 2018, 12:14am

Correct, the kernel is enabling a clock "somewhere" .

Sadly, it's abstracted out, so you won't find it that specific file..

Regards,

Burak_Soner1 · June 29, 2018, 8:23am

Hi Robert,

Thanks for the answer!

Sooo is this a “not going to happen” or if I comment stuff out from this pwm-tiehrpwm.c and and rebuild the kernel a thousand times, I may find that clock somehow? Or is the scope of “somewhere” the whole kernel? (I really want that eHRPWM start from PRU…)

Thanks a lot for your help Robert!
Burak

DTJF · June 29, 2018, 3:58pm

Hi Burak!

RobertCNelson · June 29, 2018, 4:09pm

"Anyway, in order to get the PWM subsystems working in kernel 4.x you
have to make sure that the register doesn't get cleared at boot. Ie.
you can invert the bit-logic. Therefor download the device tree
sources, edit file am33xx-clocks.dtsi, find the following nodes and
make them look like (= add tag ti,set-bit-to-disable)
"

^ that should be patchable as a u-boot overlay..

Regards,

Burak_Soner1 · June 29, 2018, 5:12pm

Hi TJF,

I’ve tried out the 2nd option you’ve mentioned. Did the following in “/opt/source/dtb-4.9-ti”

added the lines " ti,set-bit-to-disable; " to the respective places in eHRPWM0,1,2 in ./src/arm/am33xx-clocks.dtsi
make clean
make all
sudo make install
reboot

I got a bus error while calling prussdrv_open() in host code. Seeing this I tried going back (deleted the lines I added) but couldn’t get rid of the bus error. Do you have an idea as to what I might have screwed up? Since the no-bus-error state of my BBB I didn’t try out anything else other than what I’ve said here so that seems to be the only reason why.

Thanks a lot for your help!
Burak

DTJF · June 29, 2018, 5:53pm

Hi Robert!

zmatt · June 30, 2018, 4:52am

Not possible since the tbclken bits are in the control module, which ignores all unprivileged writes, and PRU cannot perform privileged writes. (Except maybe using a really hacky setup involving EDMA, but that would still need to be setup using privileged writes from the cortex-A8.)

Workaround 1: Set the tbclk bits from linux userspace via /dev/mem by tricking the kernel into performing the privileged write for you, e.g. using process_vm_readv().

Workaround 2: Modify the device tree to have the kernel enable the clocks for you (by changing the base dtb or by using an overlay).

In general, you should usually let the kernel deal with PRCM for you, since typically this is easier than doing it manually, and it avoids the kernel getting angry if it discovers you’ve been messing with PRCM behind its back. To force-enable the main clocks of a module, all you have to set the “ti,no-idle” property in DT:

&epwmss0 {  ti,no-idle;  };
&epwmss1 {  ti,no-idle;  };
&epwmss2 {  ti,no-idle;  };

This removes the need to meddle with any CLKCTRL registers from PRU.

Unfortunately, things are not so simple for the ehrpwm timebase clocks (tbclk), in fact they’re quite a headache. These are not referenced anywhere in the platform clock/hwmod data hardcoded in the kernel, hence not auto-managed by the platform power management code. Instead, they are explicitly requested by the kernel’s ehrpwm driver, which is obviously of no use when not using the kernel’s ehrpwm driver.

In your case you’d want to use DT to either force-enable these clocks, or mark them as being required for pruss. Unfortunately, neither is possible as far as I know.

One solution would be to patch the kernel to make the kernel think each timebase clock is needed for the corresponding pwmss, hence setting “ti,no-idle” on each pwmss would then make the kernel automatically set all the tbclken bits for you:

--- a/arch/arm/mach-omap2/omap_hwmod_33xx_43xx_ipblock_data.c
+++ b/arch/arm/mach-omap2/omap_hwmod_33xx_43xx_ipblock_data.c
@@ -482,7 +482,7 @@ struct omap_hwmod am33xx_epwmss0_hwmod = {
 	.name		= "epwmss0",
 	.class		= &am33xx_epwmss_hwmod_class,
 	.clkdm_name	= "l4ls_clkdm",
-	.main_clk	= "l4ls_gclk",
+	.main_clk	= "ehrpwm0_tbclk",
 	.prcm		= {
 		.omap4	= {
 			.modulemode	= MODULEMODE_SWCTRL,
@@ -495,7 +495,7 @@ struct omap_hwmod am33xx_epwmss1_hwmod = {
 	.name		= "epwmss1",
 	.class		= &am33xx_epwmss_hwmod_class,
 	.clkdm_name	= "l4ls_clkdm",
-	.main_clk	= "l4ls_gclk",
+	.main_clk	= "ehrpwm1_tbclk",
 	.prcm		= {
 		.omap4	= {
 			.modulemode	= MODULEMODE_SWCTRL,
@@ -508,7 +508,7 @@ struct omap_hwmod am33xx_epwmss2_hwmod = {
 	.name		= "epwmss2",
 	.class		= &am33xx_epwmss_hwmod_class,
 	.clkdm_name	= "l4ls_clkdm",
-	.main_clk	= "l4ls_gclk",
+	.main_clk	= "ehrpwm2_tbclk",
 	.prcm		= {
 		.omap4	= {
 			.modulemode	= MODULEMODE_SWCTRL,

An alternative that’s kinda hacky but doesn’t require kernel patching would be to lie to the kernel by telling it each tbclken bit needs to be set to disable the corresponding clock. Since the kernel thinks the timebase clocks are not being used hence should be disabled, it will set the bits for you.

&ehrpwm0_tbclk {  ti,set-bit-to-disable;  };
&ehrpwm1_tbclk {  ti,set-bit-to-disable;  };
&ehrpwm2_tbclk {  ti,set-bit-to-disable;  };

This feels dirty though. Maybe there’s still a better way I’m overlooking, so I’ll keep thinking.

What makes all this even more annoying is that the reason tbclken exists is in the first place is to allow the timebase clocks of multiple ehrpwm modules to be started simultaneously with a single register-write. This register isn’t at all meant for power management, and the way the kernel is handling it right now, with a clock node per bit, is worse than useless. In the absence of a mechanism to set multiple tbclken bits at the same time, it should just enable them all and be done with it.

Matthijs

zmatt · June 30, 2018, 5:11am

Sounds like the pruss node declaration is borked. Possibly those device tree sources were not recent enough? Try reinstalling the kernel with:
sudo apt-get --reinstall install linux-image-$(uname -r)

As Robert mentioned, instead of messing with the main dtb files (and risk problems like these), you can use an overlay to do this since in recent images overlays are applied to the main DT by u-boot rather than being loaded at runtime (which would be too late for this stuff). The easy way is by:

git clone https://github.com/mvduin/overlay-utils
in it, create “pwm-hackery.dtsi” containing the relevant declarations ( &node { property; }; )
run “make pwm-hackery.dtbo” and place the resulting file in /lib/firmware
edit /boot/uEnv.txt to load the overlay.

I don’t know if you’re currently already using any overlay or if you’re using cape-universal, but beware that enabling any overlay will implicitly disable cape-universal (except for certain special-purpose overlays such as the pru overlay given in the “uboot_overlay_pru” variable). I’m not sure what the best way is to use an overlay like this without losing cape-universal.

Matthijs

zmatt · June 30, 2018, 5:19am

On a closer read, it sounds like you used the 4.9-ti device tree sources with a 4.9-bone kernel, though I’m not sure why that would matter in this case, since correct declaration of the pru subsystem is handled by AM335X-PRU-UIO-00A0.dtbo anyway.

… you do have uboot_overlay_pru=/lib/firmware/AM335X-PRU-UIO-00A0.dtbo in your /boot/uEnv.txt, right?

Matthijs

Burak_Soner1 · June 30, 2018, 5:41am

Hi Matthijs,

Thanks for the detailed explanations! What TJF mentioned was also similar to the first thing (meddling with the dtb’s method) you mentioned I guess (editing “am33xx-clocks.dtsi”).

Yes I have that uboot_overlay line at uboot overlay position 4 (right under where it says “###Additional custom capes”)

I’ve checked the dmesg at boot and after I try prussdrv_open(). There is a fault coming up, I guess this is relevant:

[ 185.984870] Unhandled fault: external abort on non-linefetch (0x1018) at 0xb6d9b000
[ 185.992583] pgd = db720000
[ 185.995298] [b6d9b000] *pgd=9b724831, *pte=4a304303, *ppte=4a304a33

I’ll try reinstalling the kernel and then doing the overlay method in ~4-5 hrs and post the result here.

Thanks a lot for your help!
Burak

DTJF · June 30, 2018, 11:46am

Matthijs found nice wording for my statements!

@Burak:

Just for curiosity: What do you target and how many PWM signals do you need?

Burak_Soner1 · June 30, 2018, 4:00pm

Hi Matthijs, TJF, Robert

I’ll try to elaborate as much as possible on what I’ve done with regards to your suggestions. Some seemingly unnecessary details may be useful for other people too. I’ve got something that I can work with in the end.

I’ve reinstalled the kernel according to Matthijs’ suggestion → bus error is gone, dmesg error is also gone. By the way, I previously said I had the “AM335X-PRU-UIO-00A0.dtbo” in a custom capes slot. This was seemingly unnecessary (and also wrong I guess) because there is a line in the uEnv.txt which says “uboot_overlay_pru=/lib/firmware/AM335X-PRU-UIO-00A0.dtbo” rather than what I had → “uboot_overlay_addr4=/lib/firmware/AM335X-PRU-UIO-00A0.dtbo”. I had this open already so that was handling the pru declaration.
After reinstalling the kernel I gave the “am335x-clocks.dtsi” inside /opt/source/dtb-4.9-ti/src/arm another go per TJF’s initial suggestion (“maybe I did something wrong last time?”). Same result, bus error, dmesg problem. I reinstalled the kernel once more to alleviate this and let the /opt/source/dtb-4.9-ti/src/arm folder to be.
I’ve git cloned the overlay-utils repo to my home path, generated pwm-hackish.dtsi inside it as such (just 3 lines):

&ehrpwm0_tbclk { ti,set-bit-to-disable; };
&ehrpwm1_tbclk { ti,set-bit-to-disable; };
&ehrpwm2_tbclk { ti,set-bit-to-disable; };

built it with “make pwm-hackish.dtbo” while still inside overlay-utils repo I’ve cloned, and moved it with “sudo mv pwm-hackery.dtbo /lib/firmware/pwm-hackish.dtbo”. At this point in my /boot/uEnv.txt, I have

…
enable_uboot_overlays=1

…

###Additional custom capes
uboot_overlay_addr4=/lib/firmware/pwm-hackish.dtbo

…

###Custom Cape
dtb_overlay=/lib/firmware/BB-PWM1-00A0.dtbo → default cape inside /lib/firmware, nothing interesting

…

###pru_uio (4.4.x-ti & mainline/bone kernel)
uboot_overlay_pru=/lib/firmware/AM335X-PRU-UIO-00A0.dtbo

…

###Cape Universal Enable
#enable_uboot_cape_universal=1 → commented, so disabled.

…

Did a reboot, run my PRU app configuring the eHRPWM registers and checking (reading and relaying it to the host side via the PRU Shared RAM, host side does printf) the tbclken register values. I’ve got 0x7 for the pwmss_ctrl (means all 3 tbclken’s are set, as expected) and PWM is working as expected. So this hackish overlay works (tbclken for all 3 ehrpwm’s get enabled during boot) since u-boot sets the pwmss_ctrl register to |= 0x1, then |= 0x2, then |= 0x4 early in the boot sequence.

Would be cool if TJF could confirm this → I think if I had the somewhat “correct” dt’s inside /opt/source/dtb-4.9-ti/src/arm, changing the “am33xx-clocks.dtsi” would have done virtually the same thing?

** Matthijs you’ve mentioned this “synchronization of the ehrpwm clocks” via tbclken bits and mentioned that it’s worse than useless right now with the kernel I’m using. You’ve said this since the kernel literally can’t set the whole register to 0x7 or 0x0, thus enabling or disabling all 3 of the ehrpwm clocks simultaneously (at the same clock edge), right? The TRM also explicitly states this intended use case so I guess if the kernel somehow treated this register as a whole and not the separate bits as three clock nodes (and thus could set the whole register to 0x7, 0x0 etc.), using this feature would be possible right? (not that I know if this is possible with the kernel or not, just curious)

** TJF, I’m using the eCAP PWM right now as a busy-on-wait timer. I use that to generate fixed-time tasks (like interrupt service routines on MCUs triggered by a dedicated timer etc.). I guess it’s not possible to truly interrupt the PRUs from an event generated by a timer outside the PRUSS or something, right? Haven’t read the DMTIMERx’s yet on the TRM though. By the way, I mean interrupt in the following sense → the PRU is running its main loop doing stuff, and the ISR kicks in, stops the PRU main loop, does the ISR job, PRU continues from where it left off on the main loop before the interrupt happened.

** TJF, right now I’m aiming at getting the i2c, spi, uart, gpio, ecap, eqep, adc and ehrpwm up and running by using only C on the PRUs, manipulating the registers. Trying to keep the host side intervention minimal. I used to do some power electronics and motion control stuff with the C2000 series ~5-6 years back, the reference manual covered everything there and things were much simpler but the chip capability was not comparable to this one of course (the PRUs are also architecturally different too, I know it’s not the same thing but I wish to use them for similar purposes). Basically I’m trying to get myself acquainted with these connected-to-everything-in-everyway chips and the Linux<->real-time-systems combo. No end goal for the moment. I know about libpruio by the way, it’s GREAT! But I want to learn how to do these myself too and in C. I’m pretty new to Linux too so this is all really good training.

Thanks a lot for your support!
Burak

DTJF · July 1, 2018, 7:50am

Would be cool if TJF could confirm this → I think if I had the somewhat “correct” dt’s inside /opt/source/dtb-4.9-ti/src/arm, changing the “am33xx-clocks.dtsi” would have done virtually the same thing?

Yes, changing the “am33xx-clocks.dtsi” should do the same thing. I wonder why it doesn’t work for you.

AFAIR I tried the Uboot overlay solution as well, with an early 4.1 kernel and in combination with other topics. It didn’t work for me, I thought the tbclken register gets set by the driver only once, when the node gets created. Nice to hear that it works for newer kernels.

** Matthijs you’ve mentioned this “synchronization of the ehrpwm clocks” via tbclken bits and mentioned that it’s worse than useless right now with the kernel I’m using. You’ve said this since the kernel literally can’t set the whole register to 0x7 or 0x0, thus enabling or disabling all 3 of the ehrpwm clocks simultaneously (at the same clock edge), right? The TRM also explicitly states this intended use case so I guess if the kernel somehow treated this register as a whole and not the separate bits as three clock nodes (and thus could set the whole register to 0x7, 0x0 etc.), using this feature would be possible right? (not that I know if this is possible with the kernel or not, just curious)

You can synchronize the output of several PWM modules, ie. to get a specific phase shift between the signals of different PWMSS modules. Therefor the output gets stopped, all related modules get configured and then started all at one with a single write to the tbclken register (in privileged mode). Note: the PWMSS clock has to be enabled all the time, only the output gets stopped by tbclken.

It might be an interesting feature for your power control tasks.

** TJF, I’m using the eCAP PWM right now as a busy-on-wait timer. I use that to generate fixed-time tasks (like interrupt service routines on MCUs triggered by a dedicated timer etc.). I guess it’s not possible to truly interrupt the PRUs from an event generated by a timer outside the PRUSS or something, right? Haven’t read the DMTIMERx’s yet on the TRM though. By the way, I mean interrupt in the following sense → the PRU is running its main loop doing stuff, and the ISR kicks in, stops the PRU main loop, does the ISR job, PRU continues from where it left off on the main loop before the interrupt happened.

Which eCAP? The PRU internal or one of the PWMSS[0-2] eCAPs? (PWMSS1.eCAP isn’t connected to any header pin, so it’s free for such a task. But it has undefined latency due to OCP master access.)

There’s also a timer in the PRU IEP module that can get used for such tasks.

Regards

Burak_Soner1 · July 1, 2018, 9:25am

Hi TJF,

I mixed it up in hurry, you’re right, the one I mentioned was the PRU local eCAP. The PWMSS0 and PWMSS2 eCAP’s are available on pins for external PWM tasks I guess.

I used the local eCAP to not run into that non-deterministic latency due to OCP master access, it’s pretty neat but I didn’t know about the IEP, will check that out, thanks! The PRU local eCAP’s are not routed to any pins either though right? So even if I switch to IEP for the busy-on-wait timer, I don’t gain a PWM output (from the local PRU eCAP)?

Regards,
Burak

DTJF · July 1, 2018, 5:24pm

Hi TJF,

I mixed it up in hurry, you’re right, the one I mentioned was the PRU local eCAP. The PWMSS0 and PWMSS2 eCAP’s are available on pins for external PWM tasks I guess.

I also mixed something up: JT_05 is PWMSS1.eCAP. (The JTack header is normaly used for debugging, it’s 5V output.)

I used the local eCAP to not run into that non-deterministic latency due to OCP master access, it’s pretty neat but I didn’t know about the IEP, will check that out, thanks! The PRU local eCAP’s are not routed to any pins either though right? So even if I switch to IEP for the busy-on-wait timer, I don’t gain a PWM output (from the local PRU eCAP)?

PRU local eCAP is available on P8_15 (mode 5) or P9_42 (mode 3 for ball 59, CPU pin C18), which is also PWMSS0.eCAP (mode 0 for ball 59).

Regards

zmatt · July 2, 2018, 7:54pm

Maybe there’s still a better way I’m overlooking, so I’ll keep thinking.

I asked Tony Lindgren for his thoughts on this. This was his reply:

How about configure it as an optional clock named “tbclk” for the
module with ti-sysc driver in the dts and then tag it with
SYSC_QUIRK_OPT_CLKS_NEEDED (ti,opt-clocks-needed maybe) and
ti,no-idle-on-init?

Unfortunately those parts are not yet implemented in the ti-sysc
driver… I need to get rid of the hwmod platform data first without
regressions so I’m still relying on platform data to verify dts
data and use platform data callbacks for now for reset and idle.

For other alternatives, if you have a proper device driver as a
child of the module, tbclk could be it’s “fck”.

Or if you don’t want Linux to do anything with it, enable the clock
in bootloader and tag the module with status = “disabled”.

BTW, we do now have patches posted for first ever hwmod free
ti-sysc dts using driver for MCAN See thread:

[PATCH v3 0/6] Add MCAN Support for dra76x

That device is super simple though so no idle quirks yet.

My summary of this would “right now not really, but it’s easy enough once we’ve transitioned from hardcoded hwmod data to using ti-sysc driver”.

Enabling tbclk in u-boot and not touching it in the kernel at all would have been by far the best option if it had been done in the first place, but unfortunately it’s a bit invasive to implement now.

DTJF · July 4, 2018, 1:53pm

Thanks for clarification!

zmatt · July 4, 2018, 4:16pm

tbclk is enabled at CPU start-up. Not touching it at all would have been
by far the best option.

I'm pretty sure that's not true. I seem to recall it's disabled by default,
and that's also what the AM335x TRM says.

If it truly is enabled by default, then all you'd need to do is remove the
tbclk definitions from DT (and either put their labels on the parent clock
(&l4ls_gclk) to keep references from breaking, or replace those references
by &l4ls_gclk) and the problem would be solved.

As long as kernel 4.x is still experimental, kernel 3.8 will never go out
of scope.

Ehm? There is nothing "experimental" about 4.x, and although I understand
people who have a working setup with a 3.8 kernel may be reluctant to
invest the effort of moving to 4.x, that doesn't change the fact that 3.8
is very much obsolete and unmaintained.

Matthijs