Need help debugging OpenPlotter crashes

[bitsailor]

Setup is a RPI4 with OP4 connected to NMEA with Actisense NGT-1. Power supply is Apple USBC cable connected to USBC PowerDelivery rated plug. Its a new clean install. InfluxDb and Grafana are installed and running as per the docs. Nothing else added or modified.

When the boat is not moving, openplotter runs fine, and I have had it running for up to a week with zero issues. 

However, as soon as I turn the motor on, a few minutes later, Openplotter goes offline. The only way the motor can affect Openplotter is that I have engine data on the NMEA bus via a YachtDevices module connected to the yanmar motor.

Whats odd is that the raspberry PI does not appear to log any errors. I need help trying to identify the issue, so any tips on where I can look for the right debug and log info is appreciated.

I dont believe its a power issue, I setup a cron job running vcgencmd get_throttled and outputting to a log file every minute and it always just shows 0x0.

I am including samples of my get_throttled cron job and the journalctl log below.

Note that I tunred the engine on the boat on at exactly 9:29, and the raspberry PI went offline at 9:31. It looks like it may have rebooted at 10:56 - but no idea why.

Is there anywhere else I can look to try to work out the cause?

Cron job running every minute of vcgencmd get_throttled  (note that its super wierd that the subsequent outputs all have the same timestamp... its like the raspberry pi is somehow frozen:

Code:

2024-06-15 09:21:01 - throttled=0x0
2024-06-15 09:22:01 - throttled=0x0
2024-06-15 09:23:01 - throttled=0x0
2024-06-15 09:24:01 - throttled=0x0
2024-06-15 09:25:01 - throttled=0x0
2024-06-15 09:26:01 - throttled=0x0
2024-06-15 09:27:01 - throttled=0x0
2024-06-15 09:28:01 - throttled=0x0
2024-06-15 09:29:01 - throttled=0x0
2024-06-15 09:30:01 - throttled=0x0
2024-06-15 09:31:01 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0
2024-06-15 10:56:27 - throttled=0x0

Output of Journalctl - looks like the raspberry PI restarted at 10:56 - but no idea why it restarts at this specific time. There was no user input.

Code:

Jun 15 09:31:19 openplotter kernel: ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
Jun 15 09:31:20 openplotter wpa_supplicant[671]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-52 retry=1
Jun 15 09:31:20 openplotter kernel: ieee80211 phy0: brcmf_run_escan: error (-52)
Jun 15 09:31:20 openplotter kernel: ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
Jun 15 09:31:21 openplotter kernel: ieee80211 phy0: brcmf_run_escan: error (-52)
Jun 15 09:31:21 openplotter kernel: ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
Jun 15 09:31:21 openplotter wpa_supplicant[671]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-52 retry=1
Jun 15 09:31:22 openplotter kernel: ieee80211 phy0: brcmf_run_escan: error (-52)
Jun 15 09:31:22 openplotter kernel: ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
Jun 15 09:31:22 openplotter wpa_supplicant[671]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-52 retry=1
Jun 15 09:31:23 openplotter kernel: ieee80211 phy0: brcmf_run_escan: error (-52)
Jun 15 09:31:23 openplotter kernel: ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
Jun 15 09:31:23 openplotter wpa_supplicant[671]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-52 retry=1
Jun 15 09:31:24 openplotter wpa_supplicant[671]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-52 retry=1
Jun 15 09:31:24 openplotter kernel: ieee80211 phy0: brcmf_run_escan: error (-52)
Jun 15 09:31:24 openplotter kernel: ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
Jun 15 09:31:25 openplotter wpa_supplicant[671]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-52 retry=1
Jun 15 09:31:25 openplotter kernel: ieee80211 phy0: brcmf_run_escan: error (-52)
Jun 15 09:31:25 openplotter kernel: ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
Jun 15 09:31:29 openplotter kernel: brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
-- Boot c0383e3d00cd4487b538c72a3d744f57 --
Jun 15 10:56:28 openplotter kernel: Booting Linux on physical CPU 0x0000000000 [0x410fd083]
Jun 15 10:56:28 openplotter kernel: Linux version 6.6.31+rpt-rpi-v8 (serge@raspberrypi.com) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian)
2.40) #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29)
Jun 15 10:56:28 openplotter kernel: KASLR enabled
Jun 15 10:56:28 openplotter kernel: random: crng init done
Jun 15 10:56:28 openplotter kernel: Machine model: Raspberry Pi 4 Model B Rev 1.4
Jun 15 10:56:28 openplotter kernel: efi: UEFI not found.
Jun 15 10:56:28 openplotter kernel: Reserved memory: created CMA memory pool at 0x000000000dc00000, size 512 MiB
Jun 15 10:56:28 openplotter kernel: OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
Jun 15 10:56:28 openplotter kernel: OF: reserved mem: 0x000000000dc00000..0x000000002dbfffff (524288 KiB) map reusable linux,cma
Jun 15 10:56:28 openplotter kernel: OF: reserved mem: 0x000000003ef64560..0x000000003ef64641 (0 KiB) nomap non-reusable nvram@0
Jun 15 10:56:28 openplotter kernel: Zone ranges:
Jun 15 10:56:28 openplotter kernel:  DMA      [mem 0x0000000000000000-0x000000003fffffff]
Jun 15 10:56:28 openplotter kernel:  DMA32    [mem 0x0000000040000000-0x00000000ffffffff]
Jun 15 10:56:28 openplotter kernel:  Normal  [mem 0x0000000100000000-0x00000001ffffffff]
Jun 15 10:56:28 openplotter kernel: Movable zone start for each node
Jun 15 10:56:28 openplotter kernel: Early memory node ranges
Jun 15 10:56:28 openplotter kernel:  node  0: [mem 0x0000000000000000-0x000000003b3fffff]
Jun 15 10:56:28 openplotter kernel:  node  0: [mem 0x0000000040000000-0x00000000fbffffff]
Jun 15 10:56:28 openplotter kernel:  node  0: [mem 0x0000000100000000-0x00000001ffffffff]
Jun 15 10:56:28 openplotter kernel: Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
Jun 15 10:56:28 openplotter kernel: On node 0, zone DMA32: 19456 pages in unavailable ranges
Jun 15 10:56:28 openplotter kernel: On node 0, zone Normal: 16384 pages in unavailable ranges

[Sailoog]

It seems like a power problem to me. I would remove any data connection to see if the problem persists. Maybe your dc/dc converter does not allow more than 12V when the engine is on.

[bitsailor]

Hi Sailoog,

Thanks for the quick response.

With the Actisense-NG1 disconnected, openplotter does not have any issues when the engine is turned on. In addition, turning the instruments on/off (which activates the NMEA bus) also has no effect. 

The issue only happens when the Actisense is connected and the engine is turned on, and the only variable that changes at this point is NMEA data being collected.

Perhaps this is still causing some wierd edge-case power issue, but if that was the case wouldnt the 'get_throttled' command show something other than 0x00?

I am still trying to find if there are additional places I can look for logs/clues?

Thanks

[bitsailor]

@Sailoog - just want to provide an update on this issue as its not related to power

I believe I may have managed to track down the issue to the notifications module. When the engine is running this is what shows in htop - dozens of notifications-visual scripts maxing the CPU out - it runs for a few mins then just dies:



I cant find any documentation on the notifications module, but is there a way to debug and potentially disable or filter it out?

Thanks!