BeaglePlay frozen after suspending to ram

General Discussion
,
1

Hello,

I’m having a hard time making the BeaglePlay go into a low power state and waking afterwards, hope someone here can help.
When I try to suspend to RAM (“echo mem > /sys/power/state”) the kernel starts to shut down parts of the system, but it doesn’t seem to finish the suspend process and it is no longer responsive to any wakeup sources I tried (user button, rtcwake, uart, usb keyboard).

I’ve tried both on Debian, and on a custom TI SDK 09.01 based system built with Yocto.

I measured the current both while idle and after issuing the suspend command, the difference was about 30mA (230mA down to ~200mA). This suggests to me that suspend failed to finish properly.

Here are the logs after suspending on the TI SDK based OS:

echo mem > /sys/power/state
[   36.172436] PM: suspend entry (deep)
[   36.180186] Filesystems sync: 0.004 seconds
[   36.185548] Freezing user space processes
[   36.191431] Freezing user space processes completed (elapsed 0.001 seconds)
[   36.198448] OOM killer disabled.
[   36.201680] Freezing remaining freezable tasks
[   36.207469] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[   36.214881] printk: Suspending console(s) (use no_console_suspend to debug)

Loaded modules:

lsmod

Module                  Size  Used by
overlay               122880  0
wl18xx                114688  0
wlcore                217088  1 wl18xx
mac80211              548864  2 wl18xx,wlcore
libarc4                16384  1 mac80211
xhci_plat_hcd          16384  0
cfg80211              385024  3 wl18xx,wlcore,mac80211
rfkill                 24576  2 cfg80211
dwc3                   98304  0
crct10dif_ce           16384  1
wlcore_sdio            16384  0
dwc3_am62              16384  0
rtc_bq32k              16384  0
at24                   20480  0
cryptodev              53248  0
fuse                  131072  1
drm                   561152  0
drm_panel_orientation_quirks    28672  1 drm
ipv6                  475136  38

And the boot logs:
boot.log (28.4 KB)

1 Like
2

Hi @sdo ,

I think that the Beagleplay has an AM625 GP SoC (Non-HS). However TI-SDK has stopped support for GP devices in the latest SDK that you’re talking about.

This release supports High Security - Field Securable (HS-FS) devices. For migration guide and other info, refer GP to HS-FS Migration Guide

Also, the firmware revision that I see from your logs seems to be from a much older release (08.06?)
What version of bootloaders are you using? Are you building on your own or using the default one’s on the emmc?

There are alot of entities involved in Low Power Mode support (TF-A, DM Firmware, TIFS firmware, U-Boot, Linux kernel) which makes things a bit harder to debug if there are version mismatches.

3

Hi @dhruvag2000,

This release supports High Security - Field Securable (HS-FS) devices. For migration guide and other info, refer GP to HS-FS Migration Guide

Thanks for the information. I tried the steps for GP systems from the migration guide but it didn’t change the suspend to ram behavior.

Also, the firmware revision that I see from your logs seems to be from a much older release (08.06?)
What version of bootloaders are you using? Are you building on your own or using the default one’s on the emmc?

I built the default TI SDK and added a few libraries on top of that, I did not change the boot loader or kernel. (u-boot-ti-staging/1_2023.04+gitAUTOINC+b0d717b732-r0_tisdk_1_edgeai_2, git hash e5972306f9ac7ab9733ded53321dfb5dffba8d02 of last commit)

4

Since my last post I changed the yocto MACHINE=“am62xx-evm” to MACHINE=“beagleplay” and rebuilt everything.

The “User key” interrupt shows up in /proc/interrupts, and works as long as a suspend to RAM is not started.

352:         **13**          0          0          0      GPIO  18 Edge    -davinci_gpio  User Key
5

@sdo Can you please try with one of Robert’s latest images once?

The image I have has following versions:

debian@BeaglePlay:~$ uname -a
Linux BeaglePlay 6.1.69-ti-arm64-r23 #1bookworm SMP PREEMPT_DYNAMIC Wed Jan 31 06:22:50 UTC 2024 aarch64 GNU/Linux
debian@BeaglePlay:~$
debian@BeaglePlay:~$
debian@BeaglePlay:~$ cat /sys/power/mem_sleep
s2idle [deep]
debian@BeaglePlay:~$ dmesg  | grep -i "ti-sci 44043000.system-controller"
[    1.251255] ti-sci 44043000.system-controller: ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)')
root@BeaglePlay:~# rtcwake -s 10 -m mem -d rtc1
rtcwake: assuming RTC uses UTC ...
[2024-02-14 10:56:14] rtcwake: wakeup from "mem" using rtc1 at Thu Jan  1 00:03:58 1970
[2024-02-14 10:56:14] [  205.093668] PM: suspend entry (deep)
[2024-02-14 10:56:14] [  205.097503] Filesystems sync: 0.000 seconds
[2024-02-14 10:56:14] [  205.104801] Freezing user space processes
[2024-02-14 10:56:14] [  205.110710] Freezing user space processes completed (elapsed 0.001 seconds)
[2024-02-14 10:56:14] [  205.117772] OOM killer disabled.
[2024-02-14 10:56:14] [  205.121029] Freezing remaining freezable tasks
[2024-02-14 10:56:14] [  205.127008] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[2024-02-14 10:56:14] [  205.134439] printk: Suspending console(s) (use no_console_suspend to debug)

with the above I am definitely able to hit deepsleep because I can see the power consumption by beagleplay drop significantly as shown: 1.2 W in active and 0.09W in deepsleep

Linux active (not suspended)

deepsleep

I am first trying to make sure that you’re able to atleast enter deepsleep before we debug the wakeup/ resume path.

6

This will not work as a wakeup source from deepsleep, simply because it isn’t configured to be a wakeup source.

Please go through the IO Daisychain concept that is described here: 3.2.2.11.6. Wakeup Sources — Processor SDK AM62x Documentation

You will need to make required DT changes to chain the wake IRQ as described in the above documentation.

7

@dhruvag2000 Please kindly share a link to the Robert’s image to avoid a possible mistake on my part.

8

Hi @sdo,
You can take whatever is latest from Index of /rootfs/debian-arm64-12-bookworm-minimal-v6.1-ti

However, it seems to me that ti-u-boot has some patches that are not yet upstreamed, particularly this:

commit 91886b68025c7ad121e62d1fc1fa4601eeb736cd
Author: Kamlesh Gurudasani <kamlesh@ti.com>
Date:   Mon Jun 12 15:39:09 2023 +0530

    arm: dts: k3: binman: am625: add support for signing FSSTUB images

    Add support for signing of FSSTUB images for HSSE, HSFS and GP devices
    and include them in tispl.bin and tispl.bin_unsigned.

    Signed-off-by: Kamlesh Gurudasani <kamlesh@ti.com>

I’d advise you to replace the Boot binaries from Robert’s images with boot binaries built from ti-u-boot: ti-u-boot-2023.04-next

For build instructions please refer to 3.1.1.1. General Information — Processor SDK AM62x Documentation .
I will follow up internally in the meanwhile to check on what the timeline will be to upstream these missing patches.

I have locally tested with the ti-u-boot and I am able to suspend resume multiple times on my beagleplay:

(Make sure to rmmod optee_rng as it’s currently known to case crashes in GP devices after a suspend resume)

Do make sure to use the updated firmware components as well from ti-linux-firmware (As is mentioned in the docs I pointed to above)

9

Hello @dhruvag2000,

Thanks for the suggestions. I tried one of the newer builds and the power usage while suspended seems much lower than what I saw in my original build.
You were also right about the firmware version being old, I solved that issue now.

I managed to make suspend to ram/resume work by applying the UART daisy chain changes mentioned in the documentation you shared.
Unfortunately waking only works once per boot, after resuming once it no longer works.

I added multiple combinations of kernel modules to a blacklist (including optee_rng), but this did not seem to improve the problem I see now. Even with no kernel modules loaded it still freezes after the first suspend/resume pair.

I see the following errors after resuming the first time:

[ 1138.729127] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[ 1138.731339] CPU3 is up
[ 1138.732030] ti-sci 44043000.system-controller: ti_sci_resume: wakeup source: 0x80
[ 1138.808410] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[ 1139.421575] RTL8211F-VD Gigabit Ethernet gpio-0:00: phy_poll_reset failed: -110
[ 1139.423594] am65-cpsw-nuss 8000000.ethernet eth0: failed to start: -110
[ 1139.423616] am65-cpsw-nuss 8000000.ethernet: PM: dpm_run_callback(): am65_cpsw_nuss_resume+0x0/0x154 returns -110
[ 1139.423643] am65-cpsw-nuss 8000000.ethernet: PM: failed to resume: error -110
[ 1139.483765] OOM killer enabled.
[ 1139.483779] Restarting tasks ... done.
[ 1139.500935] PM: suspend exit

Attached the full log:
boot.log (31.0 KB)

10

@sdo please can you share fuller logs, right from the point your first UBoot print comes?

I want to make sure all the OPTEE, TF-A are correct. Also, I see the date of your Linux compilation to be Nov 30th which seems a bit old? Can you try with the latest ti-linux kernel please?

Also, the following is a known issue:

am65-cpsw-nuss 8000000.ethernet eth0: failed to start

And a fix is underway, however it shouldn’t affect overall system suspend resume.

11

Attaching logs with u-boot lines:
boot.log (53.8 KB)

I’ll try the new kernel a bit later.

12

@dhruvag2000 Here is the log from the new kernel:
boot-6.1.69+.log (52.3 KB)

With all submodules enabled the system won’t allow suspend to happen after the first resume is done. There is an error mentioning an invalid address when writing “mem” to /sys/power/state.

After blacklisting every submodule it behaved in a similar manner to the older kernel, the system froze after a few successful suspend/resume pairs.

13

@sdo I tried multiple times on my end, I am not able to reproduce the issue, can you try this versions of binaries: ( v2.9(release):v2.9.0-614-gd7a7135d32)

U-Boot SPL 2023.04-00001-g1a59dc85127f (Feb 19 2024 - 11:41:54 +0530)
SYSFW ABI: 3.1 (firmware rev 0x0009 '9.2.1--v09.02.01 (Kool Koala)')
SPL initial stack usage: 13400 bytes
Trying to boot from MMC2
Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was d
Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was d
Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was d
Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was d
Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was d
Starting ATF on ARM64 core...

NOTICE:  BL31: v2.9(release):v2.9.0-614-gd7a7135d32-dirty
NOTICE:  BL31: Built : 09:34:15, Aug 24 2023

U-Boot SPL 2023.04-00001-g1a59dc85127f (Feb 19 2024 - 11:42:00 +0530)
SYSFW ABI: 3.1 (firmware rev 0x0009 '9.2.1--v09.02.01 (Kool Koala)')

Seems like the TF-A (BL31) is the only major delta, also in your logs be sure to give dmesg after a suspend resume otherwise at the uboot prompt when it asks to select you can choose 1:

BeaglePlay microSD (extlinux.conf) (swap enabled)
1:      microSD Recovery
2:      microSD disable BCFSERIAL
3:      copy microSD to eMMC
4:      microSD (default)

This gives more logs on the console.