Bone VDD_3V3EXP Disable Issues

You seem to be talking about the BBB, correct?

Correct, though it seems nearly all posts in this thread are referring to the BBB issue?

My initial work was on BBW back in 2012 (prior to BBB release).

It sounds like BBB has similar issues to BBW with running on battery but
that the issues are slightly different.

Well the similarity is that the regulator was being kept alive by leakage currents.

Your original question is actually very easy to answer: disabling 3V3A like you were is really really not allowed, and if leakage currents didn’t keep it logic-high then the result would have been pulling nRESET low, game over.

Shutting down 3V3B is something you may be able to do if appropriate pad configuration is done (high-Z with receiver disabled, pull-up disabled, maybe pull-down enabled) depending a bit on the rest of the BBW schematic (I didn’t study it in any detail), but that doesn’t help you since it doesn’t control 3V3EXP.

Bottom line: shutting down 3V3EXP on the BBW (other than in off-state or rtc-only sleep) is not going to happen without a hardware patch.

I posted a bit too hastily since I missed the second half of your original post.

If external hardware unconditionally drives or pulls up to 3V3EXP (in case of BBW) or 3V3B (in case of BBB) then at shutdown they are indeed applying power to IOs while the corresponding power domain has already been shut down, which is not allowed.

The BBB SRM (but not the BBW SRM) warns that “no pins are to be driven until after the sys_reset line goes high” but this statement isn’t quite right since it only addresses boot and not shutdown (it should say “unless the sys_reset line is high”) and pull-ups can also cause problems. Powering processor IO banks and attached hardware using separate supply lines is tricky to get right, all versions of beaglebones get it wrong, and it was wrong to dump responsibility for these issues onto CAPE developers by putting a statement like that in the SRM. The mandate given in the SRM is actually also violated by on-board peripherals of the BBB, for example by the UART level shifter.

I think the best solution would have been to have the regulator controlled by the PMIC_PGOOD output which also controls the power-on-reset input of the AM335x, but this too would need to be carefully analyzed to verify it doesn’t cause issues.

If you take a BBW and cut the trace coming from U8 pin 5 (the LDO enable
line) and then wire U8 pin 5 to the PWR_LED net on page 2 of the
schematic, then the U8 LDO will shutdown correctly when on battery.
This fix was incorporated into BBB but apparently there are other
problems, too.

This is the only mod required to fix my original problem. No fancy
software needed.

Thanks,
Andrew

If you take a BBW and cut the trace coming from U8 pin 5 (the LDO enable
line) and then wire U8 pin 5 to the PWR_LED net on page 2 of the
schematic, then the U8 LDO will shutdown correctly when on battery.

It will shut down.
It will however not shut down correctly, neither on battery nor on dc power.

So, the BBW has a slightly different power scheme, including a different variant of the PMIC which affects timing and use of strobes. For reference, the relevant changes are:

  • strobe 15: LDO1 (rtc)
    and
  • strobe 1: DCDC1 (vdds, 1v8, ddr)
  • [5 ms delay]
  • strobe 2: LDO2 (power led)
  • strobe 3: LDO3 (3v3a, enables 3v3exp)
  • strobe 4: LDO4 (3v3b)
    This works for boot, and it may have been okay for shutdown if strobe 3 directly controlled 3v3exp (some other TI PMICs have GPIOs in the power sequence for such purposes), though it’d still make me slightly nervous and I would have preferred a later strobe. Using 3v3b to enable 3v3exp would certainly have made more sense to me than 3v3a, and I think has a decent chance of working correctly.

In all cases it would be highly desirable to have the 3v3exp actively discharged when disabled like the PMIC LDOs do, but neither the BBW 3v3exp regulator nor the BBB 3v3b regulator does this. This means that even if they are shutdown before the 3v3a, their output rails may not discharge quickly enough and still end up getting (partially) discharged via processor IOs when the 3v3a shuts down. This is of course highly dependent on CAPE(s) present (if any), especially on the BBW.

Moving the enable to strobe 2 as you’re suggesting however means that 3v3exp gets powered up a full milisecond before the 3v3a at boot. Any pins driven or strongly pulled high to 3v3exp by an external CAPE will heavily leak to the 3v3a through processor IOs. During shutdown the same thing happens again, plus some remaining discharge after the 3v3exp is disabled. This is not good for the long-term health of the processor.

This fix was incorporated into BBB but apparently there are other problems, too.

It was incorporated into BBB, and then undone in rev A6 for exactly the reason stated above. Unfortunately, instead of actually fixing the problem they reverted to using the 3v3a as enable-signal, even though the early posts in this thread show they knew about the problems this causes for battery-powered operation (though maybe they forgot).

To make matters worse, while the BBW needs a CAPE to trigger the problem since the 3v3exp is used exclusively for those, on the BBB the 3v3b and 3v3exp have been merged and the on-board peripherals powered by it already get the job done.

Matthijs

Hi Matthijs,

If you take a BBW and cut the trace coming from U8 pin 5 (the LDO enable
line) and then wire U8 pin 5 to the PWR_LED net on page 2 of the
schematic, then the U8 LDO will shutdown correctly when on battery.

It will shut down.
It will however not shut down *correctly*, neither on battery nor on dc
power.

So, the BBW has a slightly different power scheme, including a different
variant of the PMIC which affects timing and use of strobes. For reference,
the relevant changes are:

   - strobe 15: LDO1 (rtc)

and

   - strobe 1: DCDC1 (vdds, 1v8, ddr)
   - [5 ms delay]
   - strobe 2: LDO2 (power led)
   - strobe 3: LDO3 (3v3a, enables 3v3exp)
   - strobe 4: LDO4 (3v3b)

This works for boot, and it may have been okay for shutdown if strobe 3
directly controlled 3v3exp (some other TI PMICs have GPIOs in the power
sequence for such purposes), though it'd still make me slightly nervous and
I would have preferred a later strobe. Using 3v3b to enable 3v3exp would
certainly have made more sense to me than 3v3a, and I think has a decent
chance of working correctly.

In all cases it would be highly desirable to have the 3v3exp actively
discharged when disabled like the PMIC LDOs do, but neither the BBW 3v3exp
regulator nor the BBB 3v3b regulator does this. This means that even if
they are shutdown before the 3v3a, their output rails may not discharge
quickly enough and still end up getting (partially) discharged via
processor IOs when the 3v3a shuts down. This is of course highly dependent
on CAPE(s) present (if any), especially on the BBW.

Moving the enable to strobe 2 as you're suggesting however means that
3v3exp gets powered up a full milisecond before the 3v3a at boot. Any pins
driven or strongly pulled high to 3v3exp by an external CAPE will heavily
leak to the 3v3a through processor IOs. During shutdown the same thing
happens again, plus some remaining discharge after the 3v3exp is disabled.
This is not good for the long-term health of the processor.

This fix was incorporated into BBB but apparently there are other problems,

too.

It was incorporated into BBB, and then undone in rev A6 for exactly the
reason stated above. Unfortunately, instead of actually fixing the problem
they reverted to using the 3v3a as enable-signal, even though the early
posts in this thread show they knew about the problems this causes for
battery-powered operation (though maybe they forgot).

To make matters worse, while the BBW needs a CAPE to trigger the problem
since the 3v3exp is used exclusively for those, on the BBB the 3v3b and
3v3exp have been merged and the on-board peripherals powered by it already
get the job done.

Thanks for the info!

I'm sure wonky power up and down timing is not helping the silicon.

For reference, I've not seen any failure of the am335x with this rework
when using a custom cape at my previous job. That's not saying there
isn't damage, just that I didn't see any. We had built up probably 200
BBW+cape systems. The custom cape used the 3V3EXP as an enable for
cape-side power converters, so that likely added a slight delay to the
cape actually starting and may mitigate some of the issue you're
pointing out.

Thanks,
Andrew

I’m also not sure how easily the am335x would get damaged in this way… it’ll be very much dependent on the external hardware anyhow. This is nicely illustrated by what I encountered in my continued investigation of the leakage caused by BBB on-board hardware:

I’ve captured the shutdown sequence on DC versus BAT power in more detail by zooming in, omitting the 5V rails, and including the 3V3A. This is with no external hardware connected to the BBB other than the power supply and measurement probes:

Since location of jumps/bends in the signals made me suspicious about the location of some strobes, I also measured LDOs 1 and 2 (not included here to avoid clutter). It turned out strobe 15 follows strobe 1 on shutdown, in contradiction of figure 5 of the TPS datasheet. Given the bend in 3V3A around strobe 15 it also seems that some leakage from one of the LDO1 supplies (vrtc, vio, vdds) to 3V3A is also happening? That would be rather weird and unexpected (not to mention highly undesirable), so maybe I’m misreading this or it’s just a coincidence.

In the BAT-supply case, the current measured through BAT after shutdown is around 35 mA (somewhat dependent on temperature). I’ve analyzed the schematic and it appears there’s no specific culprit: there are many IO lines with pull-ups to 3v3b:

  • 2× 4.75k for am335x boot mode (EMU0, EMU1)
  • 11× 10k for eMMC
  • 7× 10k for SD card
  • 1× 10k for HDMI irq
  • 4× 1.5k for ethernet (MDIO and phy mode select)
    Since the total leakage is not huge and diffusely spread across dozens of processor IOs, and afaict no absolute maximum ratings are violated, it seems intuitively unlikely the cpu will get damaged this way. This will probably then also be true of the brief leakage currents occurring during boot and shutdown on DC power. (Not that I’m 100% sure about either)

Of course 35 mA @ 3.6V = 126 mW of power consumption is pretty bad, so even if the processor doesn’t get damaged it seems to me that people who run into this problem and are unable or unwilling to patch it in hardware should focus on deepsleep modes (which keep the PMIC in active-state) instead. Optimizing these will be hard work though.

One thing one should however definitely never do, is leave a serial cable attached to a BAT-powered BBB whose 3v3b is stuck high. The uart buffer U15 is powered by 3v3b and directly drives the cpu UART0_RXD pin (even though ironically its intended purpose was to isolate the uart pins when the cpu is powered down). It has a pull-down on the associated input, but if you connect a serial cable it will be driven high instead. Here’s the result:

Lots of thoughts going on staring at those pictures: Much steeper slopes in the DC power case already indicate much higher currents. On battery power the 3V3A ends up a full volt higher than before, whoa, and wtf is that sudden surge on the 3V3A shortly after shutdown? Red warning lights start flashing in my head. Uhh, 3V3A is exceeding the 1.8V supplies by more than 2V there, a situation the am335x datasheet repeatedly and emphatically warns should be avoided under all circumstances.

Current through BAT (right before I quickly turned the PSU off) read 80 mA, 45 mA higher than before. Wait, does that mean this 45 mA flows through a single pin of the cpu? What’s the specified latch-up limit anyway? Looks it up… 45 mA. Uh oh…

So here we have a case where the situation was changed from “hmm, annoying power consumption” to “DANGER WILL ROBINSON, DANGER” by something as seemingly innocuous as attaching a serial cable to the console port.

In the DC-powered case everything is happening rather quickly and I have no idea how to estimate how big the risks are to the CPU, or whether cumulative damage may occur. At first glance it seems to avoid overvoltage stresses in this case (which I know are cumulative), and overcurrent situations are afaik mainly thermal in nature hence dependent on their duration. But even 1 ms may very well be an epic age to a processor like the am335x, I just have no idea.

In any case, the leakage currents occurring due to enabling 3v3b 1ms before 3v3a were sufficient motivation for a patch in BBB rev A6, so presumably they are not something to be ignored.

When CAPEs are attached, everything will depend on what exactly they are doing.

As far as hardware patches go, probably the most elegant solution would be to have the 3v3b (3v3exp in case of BBW) regulator actively track the 3v3a voltage rather than using it as enable-input. This avoids a problematic voltage difference under all circumstances, and all nets needed for this are already available at the site of the regulator, which simplifies things. (If the gods are in a very generous mood, perhaps there’s even some pin-compatible chip that does it… yeah right, just dreaming here.)

It would also allow the 3v3b to take advantage of the better calibration the 3v3a voltage seems to have, and the ability to adjust it at runtime for fine-tuning. In my pictures the 3v3b is evidently set slightly too high, and as a result gets poorly regulated on 3.6V supply.

The next best thing would be using a proper enable-signal occurring somewhere in the power sequence after the 3V3A but no later than PMIC_PGOOD (= am335x PORz). The PGOOD signal itself still seems like the best choice, unless some CAPE ends up unintentially pulling down SYSBOOT pins as a result (though iirc the cpu samples sysboot some time after PORz goes high, but this would need to be verified).

Using an enable-signal asserted before and/or deasserted after the 3V3A (such as LDO2 feeding the power led) is suboptimal but may work adequately. I’d still especially try to avoid strongly driving cpu pins during boot or shutdown (as happens on the BBB when console rxd is high), since that would leave me worried about the cpu’s long-term health.

If one plans some ugly hack to make working enable-signal out of the 3V3A itself using voltage dividers or such, keep in mind it will still shut off later than desirable (3V3A only needs to drop 0.3V below 3V3B for leakage to start occurring) and the regulator isn’t guaranteed to shut off unless EN drops below:

  • 0.50 V (BBW regulator)
  • 0.40 V (BBB regulator, 25 ͏°C)
  • 0.18 V (BBB regulator, whole temperature range)

Anyhow, I’m inclined to declare this issue “fully clarified” now, or at least I’ve done all investigation I’m probably going to. To those suffering from this issue I’d advise carefully reading this thread (or at least this branch thereof) and weighing the pros and cons of the various workarounds suggested, since the most suitable one will depend the individual situation.

Good luck to everyone!

Matthijs

Just a quick clarification to avoid confusion:

Unlike the scope images in my original post, these were captured with BATMON connected to BAT even in the DC-powered case. As I mentioned in my original post this somewhat changes the behaviour of the TPS65217, and in particular prevents BAT from ever dropping below 0V, which would be out of view in these images.

The vertical range of these images is 0V…4V, grid lines are spaced 0.5V and 5ms, minor ticks 0.125V and 1ms.

The images in my original post have vertical range -1V…6V, grid lines spaced 1V and 20ms, minor ticks 0.25V and 4ms. They were taken with serial cable attached.

Matthijs

Matthijs,
Did you try any change in the EN pin of U4 (enable signal of 3V3B)?
I’m about to try SYS_RESETn (PMIC_PGOOD after U16), but I’m concerned about the 20ms turn on delay (plus 10ms due to the RC).
The other option is to go back to use 3V3AUX, and add a 1k load resistor to reduce the discharge time.
Thanks,
Max

Did you try any change in the EN pin of U4 (enable signal of 3V3B)?

No, we replaced U4 by this state-of-the-art voltage-tracking regulator:


It can't supply as much current as the original, but our needs are modest
enough.

I'm about to try SYS_RESETn (PMIC_PGOOD after U16), but I'm concerned
about the 20ms turn on delay (plus 10ms due to the RC).
The other option is to go back to use 3V3AUX, and add a 1k load resistor
to reduce the discharge time.

I would advise against both of these choices: Only PORz guarantees that the
AM335x IOs will be high-Z, SYS_RESETn is deasserted considerably later
during power-on. Using it would also case 3v3b to be power-cycled during
warm reset, which may have pros and cons but in any case if I'm reading the
TRM right it is asserted too early in this case and therefore there's a
risk the AM335x ends up driving into unpowered external peripherals.

Any reason why you're not considering PMIC_PGOOD itself? In general I'd
restrict to PMIC signals that transition somewhere between strobe 4 and
PGOOD, which makes PGOOD itself basically the only signal that qualifies.

If you do use LDO2 (3v3aux) for whatever reason, try to avoid having any
signal to the AM335x being driven high from the 3v3b during boot. In
particular, avoid having a console cable connected at boot (at least during
production use) given the stress this would put on the UART0_RXD pin. It
would then also be a good idea to reprogram LDO2 to strobe 7 as soon as
possible during early boot (preferably in the SPL).

A load resistor on 3v3b may be a good idea anyway since it lacks the active
discharging that the PMIC LDOs have.

tracking-regulator.jpg

Thanks for your response.
I see your point about not using SYS_RESETn signal… it can be driven low by the processor.I was considering that one because it’s a 3.3V logic signal. PGOOD is a 1.8V signal and the EN pin of U4 has a minimum of 2V to consider it high.
image.png

I will probably have to add an extra buffer to have PGOOD in 3.3V level to drive the U4-1 and also use it to enable a regulator in my cape (it’s actually more like a motherboard). Maybe in the next rev of my custom BBB I will make U16 a dual gate.

The reason I’m investigating these PMIC details is that I’m using a custom BBB for am industrial application, and we had a couple boards (BBB and custom one) fail. I recently found warnings in new BBBs to avoid “damage in the board”. https://groups.google.com/forum/#!topic/beaglebone/CKuTbHepHYE

tracking-regulator.jpg

my cape (it’s actually more like a motherboard).

The reason I’m investigating these PMIC details is that I’m using a custom BBB for am industrial application, and we had a couple boards (BBB and custom one) fail. I recently found warnings in new BBBs to avoid “damage in the board”.https://groups.google.com/forum/#!topic/beaglebone/CKuTbHepHYE

I’m investigating the PMIC details basically to avoid similar things happening to us :wink: Especially since there’s no way in hell our end-users are going to perform clean shutdowns.

The quirky PMIC and the power supply structure of the BBB are both headache-inducing material.

When I find a moment I’ll be sure to post some observations on shutdown behaviour in that thread.

Heh, it's a pity U3 is an inverter rather than a buffer... it's currently
not actually used (hardware reset is not enabled in eMMC), it's located at
the top of the PCB (where you can also find PGOOD) while its output is
available via R162 on the bottom of the PCB (where you'd need the enable
signal for the regulator), and since its existing input and output connect
to pins which are high-Z/don't-care by default you wouldn't even need to
disconnect those.

But there's no useful inverted signal available to use as enable afaict...