Weird C problem: PRU doesn't respond to host if two variables aren't set in loop

General Discussion
1

I’ve been pulling my hair out over a really weird problem and after trying everything I know to try, I’m posting it here in hopes of someone seeing the problem.

I am running a Beaglebone Black. The output of version.sh is at the end of this post.
I boot from an SD card.
I’m using remoteproc to communicate between a host program and the PRU code.
The program is long and includes stuff that I don’t think is important here like configured the ADC and IEP counter.

The host program lets the user choose from a menu what they want the PRU to do. Then using RPMSG, the command is sent to the PRU. It receives the message, determines what it is and executes the appropriate action in the PRU code. This is done by reading the message and using an “if-else if-else”. One of the actions is to read three analog inputs and return the values read through RPMSG. Each analog value is stored in a ring buffer-type array, type unsigned int, that is allocated in the PRU Shared memory space.

So once the host sends the message to do the analog input read, it starts to listen for responses from the PRU.

This all works great. I get data from the PRU and the host program puts it in a CSV file that I can analyze with other tools.
Now, though we need to do some basic math on the data in the PRU. I only need to do this on a subset of the data in the arrays and the start and end points in the array can vary. So I have two integer variables that I use store the start and end points. So, the routine that reads the analog data sets the values of these two variables as voltage thresholds are met. The code compiles just fine.

However, if these two variables are not set to a constant within the loop, the host never gets a message from the PRU code. It’s the craziest thing I’ve ever run into I think. I have declared these variables as local and global and nothing seems to matter. If they are set to a constant in the loop, the host doesn’t get a message from the PRU program.

So here’s the snippet… the lines that set variables start_of_pulse and end_of_pulse to zero are the ones that have to be there or no message is returned by the PRU. Really weird.

count = HWREG(SOC_ADC_TSC_0_REGS + TSC_ADC_SS_FIFOCOUNT(0));

						for (i = 0; i < count; i++) 
							{
								Data = HWREG(SOC_ADC_TSC_0_REGS + TSC_ADC_SS_FIFODATA(0));
								StepRead = (Data >> 16) & 0xF;
								RawAnalog = Data & 0xFFF;
								
								switch (StepRead)
									{
										case 0:
										
											DetTSampleSet[pnr]=RawAnalog;
											break;
										case 1:
							      				
											start_of_pulse = 0;
											end_of_pulse = 0;
											
											DetBSampleSet[pnr]=RawAnalog;
											if ((pnr == end_of_pulse) && (in_pulse == E_YES)) // seen a pulse and at end of it analyze signal
											{
												DetBSignalAverage= AnalyzeSignal(start_of_pulse, pnr);
												start_of_pulse = -1;
												end_of_pulse = -1;
												samples_in_pulse = 0;
												in_pulse = E_NO;
											}
											else
											{
												DetBSignalAverage = 0;
											}
											if (RawAnalog > 0xB54)
												{
													__R30 |= PanelYellowLED; // turn yellow LED on; at the sampling rate we're using this may result in just a flicker
													in_pulse = E_YES; // once the leading edge of the pulse passes set this
													samples_in_pulse++; // count the number of samples in the pulse
													DetBSignalAverage = 999; // samples_in_pulse;//just using for debugging
													//
													// here we will check for a max # of samples in pulse while in a pulse to throw out start of fluid or end of fluid or pulses larger than a seed
													//
													if (start_of_pulse < 0) start_of_pulse = pnr;  // set start pointer in ring buffer if it hasn't already been set for this pulse
												}
											else if ((RawAnalog <= 0xAC8) && (in_pulse == E_YES))
												{
													__R30 &= ~CAValve;  // turn compressed air off; stop fluid flow because the pulse has cleared the bottom detector
													in_pulse = E_NO; // pulse is over
													DetBSignalAverage = 1999; //samples_in_pulse;//just using for debugging
													//
													// this is an alternative place to check the number of samples in the pulse to throw out start / end of fluid but we would never stop because Rawanalog
													// will never drop while there is no fluid in the tube
													//
													if (end_of_pulse < 0) end_of_pulse = pnr;  // capture where we are in the ring buffer at the end of the pulse
													
													__R30 |= VacValve; // turn vacuum on to hold fluid in place; ultimately this would read the pressure and attempt to hold it there
													__delay_cycles(40000000);  // put this delay cycle here for now.  will be replaced with a routine to analyze pulse & maintain slight vacuum later
													__R30 &= ~VacValve;
													__delay_cycles(40000000);  // put this delay cycle here for now.  will be replaced with a routine to analyze pulse later
													__R30 |= CAValve;
												}
											else	__R30 &= ~PanelYellowLED; // turn yellow LED off
											
											StepRead = RawAnalog;
											RawAnalog = DetBSignalAverage;
											break;
										case 2:
											Pressure  = RawAnalog;
											if (pnr == E_RING_BUFFER_SIZE-1)
											{
												pnr = 0;
											}
											else
											{
												pnr++;
											}
											RawAnalog=pnr;
											break;
										default:  // flash both LEDS indicating a problem
											for (i=0;i<20;i++)
												{
													__R30 |= PanelGreenLED; // turn green LED on
													__delay_cycles(10000000);
													__R30 |= PanelYellowLED; // turn green LED on
													__delay_cycles(20000000);
													__R30 &= ~PanelGreenLED; // turn green LED off
													__delay_cycles(30000000);
													__R30 &= ~PanelYellowLED; // turn yellow LED off
													__delay_cycles(40000000);
												}
											break;
										
									}

git:/opt/scripts/:[b39ec679648a6be8f25f48bd1c9784c1fc5a0c46]
eeprom:[A335BNLT00C04417BBBK1847]
model:[TI_AM335x_BeagleBone_Black]
dogtag:[BeagleBoard.org Debian Buster IoT Image 2020-04-06]
bootloader:[microSD-(push-button)]:[/dev/mmcblk0]:[U-Boot 2019.04-00002-g07d5700e21]:[location: dd MBR]
bootloader:[eMMC-(default)]:[/dev/mmcblk1]:[U-Boot 2018.03-00002-gac9cce7c6a]:[location: dd MBR]
UBOOT: Booted Device-Tree:[am335x-boneblack-uboot-univ.dts]
UBOOT: Loaded Overlay:[AM335X-PRU-RPROC-4-19-TI-00A0]
UBOOT: Loaded Overlay:[BB-ADC-00A0]
UBOOT: Loaded Overlay:[BB-BONE-eMMC1-01-00A0]
UBOOT: Loaded Overlay:[BB-I2C2-RTC-DS3231]
UBOOT: Loaded Overlay:[BB-W1-P9.12-00A2]
kernel:[4.19.94-ti-r61]
nodejs:[v10.15.2]
/boot/uEnv.txt Settings:
uboot_overlay_options:[enable_uboot_overlays=1]
uboot_overlay_options:[uboot_overlay_addr4=/lib/firmware/BB-W1-P9.12-00A0.dtbo]
uboot_overlay_options:[disable_uboot_overlay_video=1]
uboot_overlay_options:[disable_uboot_overlay_audio=1]
uboot_overlay_options:[uboot_overlay_pru=/lib/firmware/AM335X-PRU-RPROC-4-19-TI-00A0.dtbo]
uboot_overlay_options:[enable_uboot_cape_universal=1]
uboot_overlay_options:[dtb_overlay=/lib/firmware/BB-I2C2-RTC-DS3231.dtbo]
pkg check: to individually upgrade run: [sudo apt install --only-upgrade ]
pkg:[bb-cape-overlays]:[4.14.20210401.0-0~buster+20210401]
pkg:[bb-wl18xx-firmware]:[1.20200322.0-0rcnee0~buster+20200322]
pkg:[kmod]:[26-1]
pkg:[librobotcontrol]:[1.0.4-git20190227.1-0rcnee0~buster+20190327]
pkg:[firmware-ti-connectivity]:[20190717-2rcnee1~buster+20200305]
groups:[debian : debian adm kmem dialout cdrom floppy audio dip video plugdev users systemd-journal bluetooth netdev i2c gpio pwm eqep remoteproc admin spi iio docker tisdk weston-launch xenomai cloud9ide]
cmdline:[console=ttyO0,115200n8 bone_capemgr.uboot_capemgr_enabled=1 root=/dev/mmcblk0p1 ro rootfstype=ext4 rootwait coherent_pool=1M net.ifnames=0 lpj=1990656 rng_core.default_quality=100 quiet]
dmesg | grep remote
[ 70.424168] remoteproc remoteproc0: wkup_m3 is available
[ 70.537289] remoteproc remoteproc0: powering up wkup_m3
[ 70.537322] remoteproc remoteproc0: Booting fw image am335x-pm-firmware.elf, size 217148
[ 70.537592] remoteproc remoteproc0: remote processor wkup_m3 is now up
[ 72.807404] remoteproc remoteproc1: 4a334000.pru is available
[ 72.825531] remoteproc remoteproc2: 4a338000.pru is available
dmesg | grep pru
[ 72.807404] remoteproc remoteproc1: 4a334000.pru is available
[ 72.811832] pru-rproc 4a334000.pru: PRU rproc node pru@4a334000 probed successfully
[ 72.825531] remoteproc remoteproc2: 4a338000.pru is available
[ 72.825738] pru-rproc 4a338000.pru: PRU rproc node pru@4a338000 probed successfully
dmesg | grep pinctrl-single
[ 0.951001] pinctrl-single 44e10800.pinmux: 142 pins, size 568
dmesg | grep gpio-of-helper
[ 0.964886] gpio-of-helper ocp:cape-universal: ready
lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
END

2

Just to close this out in case someone else runs in to this.

I had tried things like declaring these as global vs. local variables.
Declared them as volatile.
Those did not make a difference.

I had also renamed the two variables from start_of_pulse to pulse_start and end_of_pulse to pulse_end and that didn’t work either.
Finally, I renamed them to PulseStart and PulseEnd and it works without the two lines of code.

This is so strange to me because I have another variable named samples_in_loop (two underlines) and it doesn’t seem to make a difference.

I also didn’t get any warnings or errors when compiling and the code loads on the PRU and when those two lines are present, it works.

I even tried something as stupid as a no change change like this.

start_of_pulse = start_of_pulse +1;
start_of_pulse = start_of_pulse -1;

It wouldn’t run with this either. Only changing the names of the variables fixed it.

3

I would check the output of “nm --numeric-sort out.elf” for the two cases: non-working case and the working-with-renaming case.

Variable renaming could be affecting the allocation order your linker is using to place variables. If there is corruption, then it could explain the strange behaviour you are observing. Check the variables placed around start_of_pulse and end_of_pulse in the non-working ELF firmware file.

Regards,
Dimitar

4

Dimitar,

I’m really new to this so is the command to do what you suggest

nm --numeric-sort my_programfilename.elf

Also, where do I find the .elf file? Make doesn’t leave that stuff behind as far as I know.

I’d like to learn how to use this.

Thanks for the suggestions!

5

You seem to be using the proprietary TI compiler. See TI document SPRUHV6B. So try this:

nmpru -n my_programfilename.out

The final firmware file you get is in ELF format. In your case, if you are using TI examples, should be my_programfilename.out.

Regards,
Dimitar

7

Hi @WalterCEden,

Also you might want to check out PRUSS_Bindings to ease the tasks that you mentioned you carry out on the host. It provides CPP bindings that allow easily starting/stopping the PRUs and writing/reading from rpmsg channels, Here’s a simple example of the API functions. You can further use it to create a larger application like making a stepper motor controller which can use the back and forth rpmsgs and repeated starting/stopping of PRUs to an advantage. Functions like mem_read(), mem_write() allow debugging the PRU memory.

This is not related to your initial query, but just thought it might help :slight_smile:

Since you’re new to this, Dr. Mark Yoder’s PRU Cookbook can also help.

Regards,
Pratim.

8

Does remoteproc-pru offer no way to debug the pru core? For uio-pruss my py-uio library includes an example for setting breakpoints and tracing program execution (though currently only for pasm debug format).

Perhaps the old prudebug could be used (via /dev/mem), though doing assembly-level debugging on C code is probably somewhat annoying.

9

The only debugging I have seen requires a debug probe and I don’t want to invest in that. I basically debug with the old painful way of sending messages back to the host program that indicate what is going on in the PRU code. It works but it is slow and cumbersome at times.

10

Thanks. I have been using Dr. Yoder’s book extensively and it’s a great resource. I will check out what you’ve suggested too.

11

If I’m really getting stuck debugging the PRU, I will configure the PRU UART on pins so I can send stuff directly to/from the PRU and monitor it with a serial cable. It’s generally a lot easier than trying to process special messages in your Linux/PRU bridge. Especially if that bridge is the thing you’re trying to debug.

CAUTION 1: printf() pulls in all kinds of code machinery. Never use it in your PRU code as you’ll find yourself out of memory in a hurry. Do number conversions and formatting yourself so you understand how big they can get.

Caution 2: UARTs are glacially slow. It’s really easy to suddenly make your PRU program run like molasses by making things wait for UART sends to complete.

12

Thanks. I should probably bite the bullet and hook up with the UART for debugging but my past experiences with that type of serial ports brings back my ticks and twitches! Ha-ha! Our application i fairly straightforward right now but it may warrant the UART route later.

13

Intead of using an UART I’d write the debug data into a ring buffer in memory, this isn’t much more complex than logging to the UART and the bandwidth is HUGE

(Though aggregating the data into fewer large writes (>= 8 words per write) is recommended for maximum bandwidth. If you’re going to flood the DDR3 controller with tiny writes you see some occasional stalls of a few dozen cycles.)