creating astable build for a commercial system

Hello beaglebone community,

This is my first post.

I used beaglebone to develop a prototype for for a self-cleaning smoothie machine that we plan to commercialize (see video below). It has multiple sensors and outputs to run motors/solenoids while ensuring user safety.

https://www.youtube.com/watch?v=76XDJK5V0PA&feature=share&ab_channel=PascalKriesche

Right now we have. BBB running a linux OS and running a script we wrote in JS. One issue we have had was the 1.8V analog sensors losing signal which required us to reboot the system every tie that occurred. We also had an issue flashing 2 out of our 4 prototypes with an image that worked (seems to be a firmware version issue leasing to an incorrect hardware pin mapping).

I know eventually we want to use a chip on board, but for the initial production run we were hoping on sticking to a prebuilt onboard computing module for the initial 1000 units in order to save time and costs. I was wondering if anyone in this forum had experience launching a product with beaglebone and if this instability can be resolved with the correct I/O filtering on our PCB and software functionality.

Thanks in advance!

You might start with hardening the solution by moving the application from JS to a compiled C program.
I never had issues with locking up peripherals when using C.

Traditionally a watchdog feature is used to reboot the system when lockups are preventing the normal continuation of the program.
I think the TI processor has watchdog circuitry, not sure whether you can access it from the Linux OS (it might already use it to ensure OS lockups invoke reboots), quite sure JS will have no features with respect to this.

just my 5c

Gwen

So, you are using the actual Beaglebone board in a production product? I kind of seen them as just prototyping platforms.

How are the sensors connected to the Beaglebone? Direct or do you have some sort of driver between the sensor and the BB GPIO pin? I would think the later is the best option.

How is your software communicating with the sensors via the BB?

Have you checked either ‘dmesg’ or ‘/var/log/syslog’ on BB to see if any events were seen at the time the issue was seen? This may give a clue as to what is happening.

Cheers,

Jon

I personally would not use the AM335x onboard ADC in any form for a production device. There is a known issue (Google FSM_BUSY) where the FSM gets into a hung state that cannot be resolved without POR. I banged my head against that problem for several months before realizing what was happening. My setup involved using a PRU to acquire the data and send it to user space via rpmsg. I’ve moved on to an ADS1299 based solution on a custom cape.

If you use devmem (or similar) to view the ADC registers (ADCSTAT in particular) during the hung state, you can see that bit 5 and bit 4 will both be set: FSM_BUSY=1 and STEP_ID= idle, which obviously isn’t possible, meaning the FSM is hung and can’t recover without POR.

FSM_BUSY is recoverable without a POR
https://e2e.ti.com/support/processors/f/791/t/689926

I have used the ADC without any errors in a product
buffer the input with an opamp for protection and your good to go

Hey John,

Thanks for the input here!

In our recent prototype, our sensors were directly connected with some minor filtering. In the next iteration of our hardware design, we plan to have a driver in between (similar to your suggestion).

Our software is communicating through a hardware server written in JS. The other half of our software is a UI server that interfaces with the user.

We did have log files on the hardware server that allows us to see some of the commands, but that is helpful advice on using dmesg/syslog to check as well!

Cheers,
Morgan

We built and delivered over 1000 ocean buoys in a design cycle that lasted about 9 months based on the pocketbeagle MCM. The original plan was to build a PCA on which the pocketbeagle sat (in order to satisfy some “COTS” spec) but the cost of doing so got prohibitive, so we just squished the boards into one. Not so much difference. Our initial prototypes used BB blue and pocketbeagle, and they worked fine with a tangle of wires connecting to different sensors and interfaces. We had no issues flashing things, ever; that just worked (but on the PB we used SD cards, obviously). I think I’m chiming in with Gwen on some of this here:

In our case we built everything to run on Linux using a mix of python3 and C. I definitely regret not doing all in C. It would have been nice to have compile-time checking of code, as the linter just can’t catch everything we threw at that codebase. But python did let us add some fairly complex features very quickly compared to C. I would absolutely not field a hardware system using JS, but I highly value reliability (says the guy who did half of it in python, so take that for what it’s worth). One of our biggest issues was boot time, which we never got better than 30 seconds (but most of the time I had to do that was off the clock anyway).

If I were doing what I guess you’re doing, I’d move the hardware interfaces into a compiled solution and focus hard on error trapping. A system shouldn’t lock up because a voltage went out of range (unless it went too high and you blew up the input, but that’s bad hardware design). Realistically almost all of the work we did on that job was learning about errors to trap. Python’s duck typing is awesome for prototyping but made my life hell in a tight delivery schedule. Trapping things like “the logs partition is full” seems hard in JS - a bugaboo which made my stuff die in 24 days, if I recall correctly.

Anyway, if you separate the UI (in JS, or whatever it has to be) from the hardware interface, it would be easier to make the system more robust and capable. I would do all the HW interface in C if I had to do it again, and it wouldn’t have taken much longer to implement, might have cost me 10% additional dev time. Define the inputs very well on the hardware side (i.e. clamp them), and in the software reject any signal outside the expected range. If you can completely eliminate the OS and go to a compiled baremetal solution (or a stripped down RTOS like what TI offer for Sitara) you will eliminate the linux cruft (you probably don’t need chrony, syslog, a filesystem, etc.) which makes for a simpler system that is easier to debug. Of course then you’ll spend a lot more time programming, but your yogurt machine won’t need to reboot so much.

The biggest issue we have in this type of job is testing. Software guys know how to build a nice test framework. It works great all the way up till you have to deal with some external piece of hardware that might not always do what you expect. In my case, I had a modem which worked well except when it failed in a variety of exciting ways; a GPS receiver which occasionally reported impossible locations, and a series of different analog- and digital-output sensors which all found interesting ways to work wrongly. I’ve built test hardware for jobs, but I’ve never had to automate testing of a system involving so many disparate parts. That’s really, really hard to automate, and I’ve yet to find an approach which doesn’t require me to turn it into a career. As I said above, defining the range of the inputs is key, but it’s so much easier to do with an analog signal than a digital protocol. Even then, should actuator 1 do X when both sensors 1 and 2 are at their negative limits - sometimes these things aren’t completely obvious until you think them through.

Those are just some thoughts from someone who has been there. Good luck though.

Jim

Thanks Gwen,

That’s good to know about the watchdog feature and thanks for the confirmation. Our plan is to switch to C or C++ once we finalize the hardware controls and design for the system. Really appreciate your input here!

Cheers

I too read that post - I don’t see in there where the procedure to recover from the hung state is described.

Some boards of various models have been known to have the wrong board
ID burned into the EEPROM; this could "fool" u-boot into loading the wrong
device tree for the board.

Jim,

Thanks for the feedback and insight. That is incredibly valuable to know your experience. We did separate the hardware and software, but I agree the next step will be to write the HW server in C/C++.

Given your experience and feedback, I think you may have saved us several headaches, so thanks so much for sharing! Hopefully that sets us on the right track!

Cheers,