*newbie* /dev/root full after 24hrs - Printable Version +- OpenMarine (https://forum.openmarine.net) +-- Forum: OpenPlotter (https://forum.openmarine.net/forumdisplay.php?fid=1) +--- Forum: Bug Reports (https://forum.openmarine.net/forumdisplay.php?fid=4) +--- Thread: *newbie* /dev/root full after 24hrs (/showthread.php?tid=1216) |
RE: *newbie* /dev/root full after 24hrs - Sailoog - 2018-08-08 AFAIK this issue is only present in kplex/SK interaction so I propose to wait for the fix in SK node server that tkurki propose. If this happens again in other scenario we could apply solutions proposed by e-sailing and stripydog. To summarize... Provisional fix Manually edit ~/pi/.kplex.conf Code: ###defaults Fix in SK https://github.com/SignalK/signalk-server-node/pull/581 If this happens again changes in kplex? heartbeat in kplex heartbeat in openplotter watchdog in openplotter closing connections RE: *newbie* /dev/root full after 24hrs - tkurki - 2018-08-09 I just released SK server version 1.4.3, which no longer reconnects tcp if the connection is idle. It should be available in npm shortly. https://github.com/SignalK/signalk-server-node/blob/master/CHANGELOG.md#v143-20180809-1821-0000 Please let me know if this fixes the issue. RE: *newbie* /dev/root full after 24hrs - Sailoog - 2018-08-10 Signal K server 1.4.3 is already available in npm, please could you test and report? Type: sudo npm install --unsafe-perm -g signalk-server and reset OP RE: *newbie* /dev/root full after 24hrs - PaddyB - 2018-08-11 (2018-08-10, 05:45 PM)Sailoog Wrote: Signal K server 1.4.3 is already available in npm, please could you test and report? Updated and this from running overnight > Code: kplex 1436 pi 5u IPv4 17290 0t0 TCP *:30330 (LISTEN) RE: *newbie* /dev/root full after 24hrs - stripydog - 2018-08-15 (2018-08-08, 03:36 PM)tkurki Wrote: How about a watchdog thread that closes connections that the client has closed (that are in CLOSE_WAIT), no matter what? Not that simple. Each connection in kplex has its own thread and each thread is responsible for all the data associated with that connection. If you simply close the socket the next client connection will re-use the closed file descriptor which its (old) controlling thread doesn't realise has been closed under it and gets you into a world of pain. Better to signal the controlling thread to die but then this is becomes a non-trivial architectural change. Again, this is not ideal and hands up, no output with thousands of unfulfilled clients was not a scenario I considered, but a kludge as a workaround while awaiting a complete re-architecture is probably the most straight forward solution here. RE: *newbie* /dev/root full after 24hrs - tkurki - 2018-08-16 (2018-08-11, 10:21 AM)PaddyB Wrote: Updated and this from running overnight > So better but not perfect: still a few of those pesky CLOSE_WAIT connections. A scenario that could produce this is SK server crashing for some unrelated reason and getting restarted by systemd. This is actually a realistic case re: kplex and CLOSE_WAIT in general: a crashing client that gets restarted will produce dangling connections. Do you see SK server restarting in syslog? You can also get SK tcp provider debug logging by running the server with environment variable DEBUG set Code: DEBUG=signalk-provider-tcp This will log all connects, disconnects, errors and reconnects. |