2026-07-05, 06:05 PM
Hi all, sharing this in case it saves someone else a LOOOONG debugging session.
Setup: Raspberry Pi 5, OpenPlotter, MacArthur HAT with the ICM-20948 IMU module plugged on it, pypilot in "IMU Only" mode publishing attitude to Signal K. (Note: the issue is mode-independent — it affects the full autopilot mode equally, see below).
Symptoms:
What it was NOT (all tested and ruled out): magnetic interference, bad calibration, a second process reading the IMU over I2C (verified with kernel i2c tracing — only pypilot touches the bus), the Kalman filter, or faulty hardware. The flips even survived disabling the compass in the fusion entirely, which was the key clue: whatever it was, it wasn't magnetic.
Root cause:
the @signalk/set-system-time plugin, which ships with Signal K server and was enabled with its default 60-second interval. Every minute it runs date -u -s "<GPS time>" with one-second resolution, truncating the fractional second. That steps the system clock backwards by up to ~1 s (in my case a consistent ~0.8 s); since I also have working NTP over 4G, systemd-timesyncd immediately steps it back forward. Two clock steps per minute, forever.
RTIMULib (used by pypilot) timestamps IMU samples with gettimeofday() — wall clock, not monotonic. Each clock step corrupts the time delta fed to the fusion filter, which then integrates the gyro over a bogus interval and the attitude goes wild for a few seconds before reconverging. I confirmed it with a small script watching wall-clock steps vs. roll excursions: perfect 1:1 correlation, every 60.0 seconds.
Fix:
disable the Set System Time plugin in Signal K (Server → Plugin Config), or set its update interval to 0 (= set time once at startup only, which keeps it useful as a fallback for boats booting without internet/RTC while removing the periodic steps). Result: zero clock steps, zero attitude flips, wave detection back to sanity.
Takeaways:
This is not specific to IMU Only mode: the corrupted dt happens inside RTIMULib/fusion, so a full pypilot autopilot install suffers the same attitude excursions — except there they feed the steering loop, potentially commanding spurious rudder corrections once per minute. If your autopilot makes a small unexplained yaw on a fixed period, check for clock steppers.
If your IMU "goes crazy" on a fixed period, suspect the system clock before the sensor. Anything stepping the clock (GPS time-setting plugins/scripts fighting with NTP) will corrupt pypilot attitude.
One boat, one time authority: either NTP, or GPS-once-at-boot, or gpsd+chrony (which slews instead of stepping) — not two of them fighting.
Longer term, RTIMULib using CLOCK_MONOTONIC (or clamping dt) would make pypilot immune to this. Happy to file it upstream — since set-system-time ships with Signal K, I suspect quite a few installations are silently affected.
Huge thanks to @aitonos who pointed me at clock synchronization as the likely culprit.
Setup: Raspberry Pi 5, OpenPlotter, MacArthur HAT with the ICM-20948 IMU module plugged on it, pypilot in "IMU Only" mode publishing attitude to Signal K. (Note: the issue is mode-independent — it affects the full autopilot mode equally, see below).
Symptoms:
- imu.roll / imu.pitch spiking to ±150° in short bursts, exactly once every 60 seconds, even moored in flat calm water
- imu.warning = compass distortions appearing intermittently
- Compass calibration never converging while sailing circles ("insufficient coverage" forever), and implausible calibration results (field strength ~27 µT, inclination -19° — should be ~45 µT / +58° here)
- A Signal K plugin computing wave motion from the IMU reported "heavy seas" in a flat calm marina (the periodic spikes inflate any RMS-based statistics)
What it was NOT (all tested and ruled out): magnetic interference, bad calibration, a second process reading the IMU over I2C (verified with kernel i2c tracing — only pypilot touches the bus), the Kalman filter, or faulty hardware. The flips even survived disabling the compass in the fusion entirely, which was the key clue: whatever it was, it wasn't magnetic.
Root cause:
the @signalk/set-system-time plugin, which ships with Signal K server and was enabled with its default 60-second interval. Every minute it runs date -u -s "<GPS time>" with one-second resolution, truncating the fractional second. That steps the system clock backwards by up to ~1 s (in my case a consistent ~0.8 s); since I also have working NTP over 4G, systemd-timesyncd immediately steps it back forward. Two clock steps per minute, forever.
RTIMULib (used by pypilot) timestamps IMU samples with gettimeofday() — wall clock, not monotonic. Each clock step corrupts the time delta fed to the fusion filter, which then integrates the gyro over a bogus interval and the attitude goes wild for a few seconds before reconverging. I confirmed it with a small script watching wall-clock steps vs. roll excursions: perfect 1:1 correlation, every 60.0 seconds.
Fix:
disable the Set System Time plugin in Signal K (Server → Plugin Config), or set its update interval to 0 (= set time once at startup only, which keeps it useful as a fallback for boats booting without internet/RTC while removing the periodic steps). Result: zero clock steps, zero attitude flips, wave detection back to sanity.
Takeaways:
This is not specific to IMU Only mode: the corrupted dt happens inside RTIMULib/fusion, so a full pypilot autopilot install suffers the same attitude excursions — except there they feed the steering loop, potentially commanding spurious rudder corrections once per minute. If your autopilot makes a small unexplained yaw on a fixed period, check for clock steppers.
If your IMU "goes crazy" on a fixed period, suspect the system clock before the sensor. Anything stepping the clock (GPS time-setting plugins/scripts fighting with NTP) will corrupt pypilot attitude.
One boat, one time authority: either NTP, or GPS-once-at-boot, or gpsd+chrony (which slews instead of stepping) — not two of them fighting.
Longer term, RTIMULib using CLOCK_MONOTONIC (or clamping dt) would make pypilot immune to this. Happy to file it upstream — since set-system-time ships with Signal K, I suspect quite a few installations are silently affected.
Huge thanks to @aitonos who pointed me at clock synchronization as the likely culprit.
