Hardware watchdog for BBB

HI, I have been working for a wile on safe power supply for BBB with backup power provided by supercapacitors. In case of power failure there is just enough time to safely and nicely shut down BBB. For some reason BBB does not always wake up fully. I need hardware dogwatch. Did anybody design such a thing? I was able to find some design for ardunio: http://www.playwitharduino.com/?p=291.
Anybody has any experience with hardware dogwatch for BBB??
Thanks in advance
Robert

HI, I have been working for a wile on safe power supply for BBB with backup power provided by supercapacitors. In case of power failure there is just enough time to safely and nicely shut down BBB. For some reason BBB does not always wake up fully. I need hardware dogwatch. Did anybody design such a thing? I was able to find some design for ardunio: http://www.playwitharduino.com/?p=291.
Anybody has any experience with hardware dogwatch for BBB??
Thanks in advance
Robert

Hi Robert,

Developing a power supply that ensures a reliable shutdown down in the event of a power failure isn’t a simple design. You really need to monitor the input power supply and the state of the kernel to determine when to remove and reapply power to the BBB. You have to consider the corner cases such as:

  1. power failure could occur during the boot up sequence
  2. power failure occurred, triggering a shutdown sequence and then power is restored during the shutdown sequence.

With Linux, you cannot arbitrarily remove power during the boot up sequence and you cannot simply reapply power during the power down sequence. In the first case, when would it be safe to simply remove power to the BBB and in the second case, when would it be safe to recycle the power to the BBB. Currently there is no external info to determine the state of the kernel so you would have to add a kernel driver which will control a GPIO to signal when the kernel is in a safe mode (all volatile info written to non-volatile memory) and also monitor a GPIO used to interrupt the kernel when a power failure occurs.

So now, you need an external state machine which tracks the input power supply, state-of-kernel and charge state of super caps. Timers are also required to ensure a proper power recycle.

I hope I have covered everything you need to consider in your design, but perhaps others has some insights I haven’t considered.

Regards,
John

I am still not sure why anyone would need / want all this complexity.

Living offgrid, powered by solar panels which charge a battery bank, which then powers our home via an Inverter . . . I am not sure why the same concept can not be used on the BBB.

  1. Power the BBB via a small rechargeable ~5V power source.

  2. charge this ~5V power source via AC mains, solar power, whatever.

  3. Monitor power on the charge input, and when absent send a message to the kernel to shutdown / hybernate.

Then, all you need is to make sure your power source can work a few minutes with no input power applied. Perhaps even double this value for “safety”.

The way I see things, there is nothing to complex about all this at all.

I am still not sure why anyone would need / want all this complexity.

Living offgrid, powered by solar panels which charge a battery bank, which then powers our home via an Inverter . . . I am not sure why the same concept can not be used on the BBB.

  1. Power the BBB via a small rechargeable ~5V power source.

  2. charge this ~5V power source via AC mains, solar power, whatever.

  3. Monitor power on the charge input, and when absent send a message to the kernel to shutdown / hybernate.

Then, all you need is to make sure your power source can work a few minutes with no input power applied. Perhaps even double this value for “safety”.

The way I see things, there is nothing to complex about all this at all.

Not a problem when you are there to push the on button or reset your BBB when it locks up. What happens when there is no human intervention and the BBB is in some remote location? What happens during brown outs, power surges, power fluctuations, auto reclosure operations, power bypass, etc? What happens when you have 10K, 100K or even 1 Million devices running. Even a 0.1% failure rate will be a disaster.

Cost is a primary factor so you cannot spend 10x on the power supply (batteries, solar panels; really?). Batteries are expensive and have a limited number of charge cycles, typically less than 1,000 cycles (less than 3 years). Actually, the circuit is even more complex because supercaps have a max voltage of 2.5 or 2.7 volts, so you have to stack them. Now you need an energy balance circuit to make sure all caps in series maintain an equal charge. During power fail, the voltage across these supercaps decrease, but you need to maintain a constant voltage, so you need a boost switching supply. The switcher cannot start until the supercaps have a minimum charge. I could go on, but yes the complexity is necessary to ensure a reliable supply.

Your solution may be perfect for your requirements, but I think we are talking about a different operating environment.

Regards,
John

What happens when you have 10K, 100K or even 1 Million devices running.

Now we know where all the BBBs went!

For home use I’ve rigged two BBBs together so that each can monitor and reset the other. Every 5 minutes each board tries to send itself a message via an ssh connection to the other board. If it fails to receive that message, it assumes the other board has crashed somehow and sends a reset. If it still fails to get a response it carries out a power cycle.

In conjunction with a simple UPS such as the OP describes, this would probably be enough for normal use.

What happens when you have 10K, 100K or even 1 Million devices running.

Now we know where all the BBBs went!

Very funny. BBB wouldn’t work for my application but I do draw from Gerald’s brilliance :wink:

Yeah after I thought about it, after making my post I realized I did not include a way to bring the BBB back up.

For bringing the BBB back up after input power is back up I suppose I would use an MSP430 to monitor the input power, and a “keep alive” signal from the BBB to the MSP430. A Value line MSP430 such as the MSP430G2553 is low cost ( ~$2.5 in quantities of 1 ) can run off a single button cell for years. the MSP430G2553 also has SPI, I2C, GPIO’s, and UART, as well as a few other niceties( hardware WDT, and Timer(s).)

So perhaps more complex than I originally led on, but perfectly doable, and not really all that complex. Just off the top of my head, I would use either a regular timer, or perhaps even use the hardware watchdog timer to cycle a reset on the BBB through a GPIO. With the keep alive signal being sent out over either SPI or UART.

Is this on track with what you had in mind, or are you thinking of something else, or is this too complex for your application ?

Yeah after I thought about it, after making my post I realized I did not include a way to bring the BBB back up.

For bringing the BBB back up after input power is back up I suppose I would use an MSP430 to monitor the input power, and a “keep alive” signal from the BBB to the MSP430. A Value line MSP430 such as the MSP430G2553 is low cost ( ~$2.5 in quantities of 1 ) can run off a single button cell for years. the MSP430G2553 also has SPI, I2C, GPIO’s, and UART, as well as a few other niceties( hardware WDT, and Timer(s).)

So perhaps more complex than I originally led on, but perfectly doable, and not really all that complex. Just off the top of my head, I would use either a regular timer, or perhaps even use the hardware watchdog timer to cycle a reset on the BBB through a GPIO. With the keep alive signal being sent out over either SPI or UART.

Is this on track with what you had in mind, or are you thinking of something else, or is this too complex for your application ?

Hi William,

I like your solution. I used a GreenPak from http://www.silego.com/ which are really low cost $0.35 in small quantities. They are tiny (about 2mm square) and very robust; no need for WDT. Also, they work down to 1.8V, which is required when working with supercaps.

Regards,
John

Hi John,

Yeah, the MSP430G2553 can go down to at least 1.8v, and I am thinking a good bit lower. I am thinking perhaps 1.2V at minimal clock / periph’s( I’d have to read the datasheet again ) Now just because I am relatively new to embedded devices, and I know the MSP430’s fairly well, I would choose these for myself. The MSP430 value line products can not beat or even meet that price by a long shot in small quantities. I think the lowest my buddy got a tube of 10 for ~$1.35 each a bit over a year ago. One or two off, personally I think this price is fair enough.

I haven’t heard of the devices you’re linking to, and the link doesn’t work for me. So i can not even look to see exactly what it is. I would assume the MSP430G2553 would be overkill by comparison, feature wise.

So, I am not much of an EE, but my buddy is. Perhaps I could get him to design something up while I’ll tie things together in software. This is something I personally have interest in as well.

Hi John,

Yeah, the MSP430G2553 can go down to at least 1.8v, and I am thinking a good bit lower. I am thinking perhaps 1.2V at minimal clock / periph’s( I’d have to read the datasheet again ) Now just because I am relatively new to embedded devices, and I know the MSP430’s fairly well, I would choose these for myself. The MSP430 value line products can not beat or even meet that price by a long shot in small quantities. I think the lowest my buddy got a tube of 10 for ~$1.35 each a bit over a year ago. One or two off, personally I think this price is fair enough.

I haven’t heard of the devices you’re linking to, and the link doesn’t work for me. So i can not even look to see exactly what it is. I would assume the MSP430G2553 would be overkill by comparison, feature wise.

So, I am not much of an EE, but my buddy is. Perhaps I could get him to design something up while I’ll tie things together in software. This is something I personally have interest in as well.

Hi William,

Strange, I just clicked on the link below and it works for me. Search Google for silego and greenpak. It’s really like a miniature mixed signal FPGA. They have a really nice software tool/simulator and dev board. I use these all the time instead of using discrete logic. They are really good for small state machines with inputs from timers, counters, analog comparators, lookup tables, macrocells, etc.

Regards,
John

I’ve just come across this conversation in my own search for a rock-solid, embeddable configuration for the BeagleBone Black. I’m trying to develop an embedded controller device that needs to live behind walls, in ceilings and in other inaccessible places. ( It is for the automation of art & other electronic installations.)

From what I gather here, the BBB is not quite up to the task, without an external watchdog circuit (please correct me if I’m misreading this thread).

@John3909: Your suggestion of the GreenPak prompted my own discovery of that tech — it looks great, esp the ecosystem of tools around the platform.

In looking around, I found some Silego application notes that implement a hardware watchdog for MCUs. http://www.silego.com/products/352/312/AN-1058.html This might be a useful starting point for anyone using GreenPak for a hardware watchdog.

@John3909: Does this design look like it might be a good fit for the BBB? (Not knowing how to read GreenPak internals, it is not obvious to me)

Alternately, I’m wondering in the two years that have passed since this thread started, if anyone has developed a hardware watchdog design for the BBB they’d be willing to share. An open-source hardware watchdog for the BBB would go a long way towards ameliorating the hardware issues with the PMIC on RevC, and allow it to prosper as a base for applications where long-term reliability matters.

Although I’m first and foremost a software engineer, I’ve got some electronics chops (albeit mostly digital), but (sadly) very limited hardware design equipment (oscilloscope, etc). [That said, I have iron, and will solder!] I’d be happy to develop & contribute the software components for such a system (I’d envision a library + device tree overlay) if someone(s) else would like to partner up to design the hardware side.

Best,

ST

We use an external msp430 for our intelligent watchdog

I haven’t worked on this in a while, but the circuitry I proposed was for a voltage monitoring and safe shutdown and startup. The AM3358 already has a watchdog timer and will reset the board if the watchdog is not serviced in a predefined period. My proposed circuit included supercaps to power the board during power failure so that the board can shutdown safely. You need a state machine to deal with all the corner cases, such as

  1. what happens if you have a power fail, you trigger a shutdown and then the power is back on before the shutdown completes. In that case, you have to power off the BBB, wait for the supercaps to charge and then reapply the power to the BBB.
  2. what happens if the power is on and the board begins to boot, but then the power fails before the board has completed it’s bootup.

There are several more corner cases when you think through all the scenarios.

Also, you need a supercap charging circuit since supercaps are normally rated at 2.7V so you have to put them in series but one supercap may have lower impedance than the other so one supercap may exceed it’s max voltage and may in fact go negative during discharge. Your charge circuit must prevent both conditions.

You will also need a boost regulator to keep the voltage on the processor constant as the supercaps discharge.

Lithium batteries would be easier, but then you are faced with the limited number of charge cycles and the limit life expectancy of these batteries.

Just some ideas to think about.

Regards,
John

So who or what monitors the MSP430? Since it is a micro-controller, it is easy to get it into a lock situation. All you need is a programable power supply which will ramp up and down the voltage into the micro-controller at predefined times and it will lock and become completely unresponsive. Granted this will rarely happen, but in our applications where 100K or more devices are installed, we cannot accept some devices locking up because of a power failure.

Regards,
John

msp430 has an internal watchdog

Sorry guys. I might be confusing my terms here, or misusing the phrase ‘watchdog timer’. Would ‘hardware watchdog circuit’ fit? “Intelligent power switch?” I’m not quite sure what to search for!

The circuit I’m looking for is what John3909 is describing; something to address the corner-cases around power and ensure graceful, and hands-off, system recovery amidst all of the corner cases (brownouts, drop+restart (incl at ‘inconvenient’ times, ie during shutdown, etc)).

Fwiw, my application is already kicking the onboard watchdog, and relying on its reboot if the software system fails. I need to make sure that that reboot ALWAYS happens, no matter what the power throws at it, as the system will be installed in walls, ceilings and such.

Does anyone know of a good public-domain/open-source external circuit design that might work around the BBB (or something close – a good starting point)? Or, even better, would anyone be able & willing to share their circuit design? I realize that this kind of circuit requires some solid engineering to get right. The EE part is a stretch for me, capability-wise, but I can offer software services in trade for hardware help.

I’m personally blocked by this issue, and just trying to work out the best path forward, but I’d love to see this problem definitively solved for everyone using the BBB. It is such a great platform. I’d love to ensure it’ll always be online and ready for what we all throw at it.

Best,
ST

Sorry guys. I might be confusing my terms here, or misusing the phrase
'watchdog timer'. Would 'hardware watchdog circuit' fit? "Intelligent
power switch?" I'm not quite sure what to search for!

I generally call a watchdog timer (and it can be external if desired)
as a timer that must be continually refreshed, if it times out, it
resets the microprocessor. However, the processor remembers that it
has been reset by a watchdog (most of them ought to), and then your
bootup routine will know that the program hung somewhere and didn't
reset the watchdog timer in time.

What the BBB needs is not so much a watchdog timer, but an intelligent
power monitor. If that processor (and it could be very simple) does
*not* have an operating system but simply runs embedded firmware, then
it will not suffer from shutdown problems as does the BBB.

It could have an operating system, but should not depend on file
systems being set up properly (a la windows or linux, I think).

You could use something as simple as an Arduino.

Harvey

What the BBB needs is not so much a watchdog timer, but an intelligent
power monitor. If that processor (and it could be very simple) does
not have an operating system but simply runs embedded firmware, then
it will not suffer from shutdown problems as does the BBB.

It could have an operating system, but should not depend on file
systems being set up properly (a la windows or linux, I think).

You could use something as simple as an Arduino.

Harvey

I would probably never use an Arduino for this purpose. The cost is too high for starters.

An MSP430G2553 is perfect for this usage because it is an extremely low power MCU, that has hardware SPI, UART, I2C, WDT, ADC, PWM . . . and at least 14-16s pin available for use after the bare minimum are used for power, gnd, reset, etc. This MCU is also well supported in CCS, as well as through gcc, and the MCP430 MCUs are well known, and proven.

So, the G2553 has it’s own WDT in case it ever get’s stuck. It has it’s own hardware ADC so it can monitor input power it’s self. Then it has GPIO’s which can be used to power down a BBB though toggling the power button, or reset the board by disconnecting power from input as needed. Plus it has it’s own POR, and BOR features built in. As well as being an MCU with internal flash for storage it is for all intents and purposes immune to power loss.

Trust me though. My buddy and I have discussed this a lot over the coarse of the last 3 or so years, and even more lately as we’ve realized that reset on the BBB is essentially broken since it’s only a soft reset. That requires the input power to be completely disconnected for a short amount of time.

Anyway, if I knew PIC, or NXP MCU’s as well as the value line MSP430’s, I might consider one of those too. But seriously, beating the power usage of a valueline MSP430 would be really hard. They can operate on a single button cell for over 10 years.

ooops, I forgot the main concern. The MSP430 could also be use as an external smart watchdog. Just by poking a GPIO pin once in a while from a beagelbone. In code this would be very easy, and not processor intensive at all.

  • BBB twiddles GPIO pin
  • Pin interrupt fires on MSP430, and toggles a bit flag.
  • timer period ends, and sees bit toggled, or not.
  • if toggled, repeat cycle.
  • if not toggled disconnect / reconnect power to the Beaglebone, and then toggle the reset switch.

Another nicety of the G2553 MCU is that it has an on die temp sensor. Then since the MCU will spend 90% + of it’s time in sleep mode, core temperature will not be above ambient temperature by very much, if at all. So you get a temperature sensor for free, if you care to hook up UART, SPI, or I2C back to the beaglebone.

I think that honestly, the MSP430G2553 is even too much hardware for the particular use case. But it’s the only MCU I know of that uses so little power, costs so little, nd has all the required features for an external smart watchdog that can “physically” interact with an SBC, as well as monitor power on it’s own.

Hi,

I use this:

wd.png

pins.png