NMI watchdog soft lockup

I have started to encounter a relatively new issue on my Beaglebone Black that I have been developing an app on for quite some time now. I am not quite sure when I first started seeing this and what may have changed, so I am looking for advice on how to troubleshoot this further:

`

Message from syslogd@beaglebone at Apr 19 19:55:10 …
kernel:[244638.300687] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]

Message from syslogd@beaglebone at Apr 19 19:55:10 …
kernel:[244666.198366] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]

Message from syslogd@beaglebone at Apr 19 19:55:10 …
kernel:[244694.095031] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]

Message from syslogd@beaglebone at Apr 19 19:55:10 …
kernel:[244721.991558] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]

`

This smells like an ancient kernel..

Run this command and share the output..

sudo /opt/scripts/tools/version.sh

Regards,

`

git:/opt/scripts/:[9df8deb6b2ae7570c9102ab58f795d3bed592194]
eeprom:[A335BNLT000C1829BBBG0182]
model:[TI_AM335x_BeagleBone_Black]
dogtag:[BeagleBoard.org Debian Image 2017-03-19]
bootloader:[microSD-(push-button)]:[/dev/mmcblk0]:[U-Boot 2019.04-00002-gc8b5ad3a1f]:[location: dd MBR]
kernel:[4.9.147-ti-r120]
nodejs:[v4.8.0]
device-tree-override:[dtb=am335x-boneblack-emmc-overlay.dtb]
uboot_overlay_options:[enable_uboot_overlays=1]
uboot_overlay_options:[disable_uboot_overlay_video=1]
uboot_overlay_options:[disable_uboot_overlay_audio=1]
uboot_overlay_options:[disable_uboot_overlay_wireless=1]
pkg check: to individually upgrade run: [sudo apt install --only-upgrade ]
pkg:[bb-cape-overlays]:[4.1.20170309-0rcnee0~bpo80+20170309]
pkg:[bb-wl18xx-firmware]:[1.20170309-0rcnee1~bpo80+20170309]
pkg:[kmod]:[18-3]
pkg:[roboticscape]:[0.3.4-git20170307-0rcnee2~bpo80+20170307]:[GOT_REPLACED_BY_NEXT]
WARNING:pkg:[librobotcontrol]:[NOT_INSTALLED]
WARNING:pkg:[firmware-ti-connectivity]:[NOT_INSTALLED]
groups:[debian : debian adm kmem dialout cdrom floppy audio dip video plugdev users systemd-journal netdev i2c bluetooth admin spi tisdk weston-launch xenomai]
cmdline:[console=ttyO0,115200n8 root=/dev/mmcblk0p1 ro rootfstype=ext4 rootwait coherent_pool=1M net.ifnames=0 quiet cape_universal=enable]
dmesg | grep remote
dmesg | grep pru
[229286.908037] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[229314.805534] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[229342.703050] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[233094.641593] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[233122.539507] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[233150.437176] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[233178.334647] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[236945.719703] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[236973.617889] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[237001.515392] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[237029.412906] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[240790.547160] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[240818.444960] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[240846.343106] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[240874.240610] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[244638.308181] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[244666.205825] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[244694.102489] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[244721.999013] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[248482.167326] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[248510.065194] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[248537.963199] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
[248565.860624] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss
dmesg | grep pinctrl-single
dmesg | grep gpio-of-helper
lsusb
Bus 001 Device 018: ID 05c6:90b2 Qualcomm, Inc.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
END

`

git:/opt/scripts/:[9df8deb6b2ae7570c9102ab58f795d3bed592194]
eeprom:[A335BNLT000C1829BBBG0182]
model:[TI_AM335x_BeagleBone_Black]
dogtag:[BeagleBoard.org Debian Image 2017-03-19]
bootloader:[microSD-(push-button)]:[/dev/mmcblk0]:[U-Boot 2019.04-00002-gc8b5ad3a1f]:[location: dd MBR]
kernel:[4.9.147-ti-r120]

ah yes, 4.9.x upgrade to 4.14.x or downgrade to v4.4.x...

nodejs:[v4.8.0]
device-tree-override:[dtb=am335x-boneblack-emmc-overlay.dtb]

This can be removed as in v4.14.x, the next 4 line's enable the same thing..

uboot_overlay_options:[enable_uboot_overlays=1]
uboot_overlay_options:[disable_uboot_overlay_video=1]
uboot_overlay_options:[disable_uboot_overlay_audio=1]
uboot_overlay_options:[disable_uboot_overlay_wireless=1]

cd /opt/scripts/tools/
git pull
sudo ./update_kernel.sh --ti-channel --lts-4_14

pkg check: to individually upgrade run: [sudo apt install --only-upgrade <pkg>]
pkg:[bb-cape-overlays]:[4.1.20170309-0rcnee0~bpo80+20170309]

and also run, as ^ is going to break something..

sudo apt install --only-upgrade bb-cape-overlays

Regards,

Awesome, thank you Robert