Sorry, let me clear it up a bit, I can understand I didn't give a complete picture.
I usually work remotely, this is what I meant with headless. On the boat I have a physical screen, that's why I know it is stuck trying to close that service. So, I don't see the VNC screen, I look at a physical screen. In the bottom-left corner it will stay active, I have to cut off the power to boot it properly again.
- When I do a sudo reboot -f it will definitely reboot, will not get stuck
- When I click on the Raspberry icon, click logoff and reboot it will always get stuck
Question, I wanted to give you a screen dump but not sure how. Is this logged somewhere somehow?
Thanks again for looking into this.
Ok i get it now. I would, parallel to the VNC session, start an ssh session (on windows: with putty.exe) to the same box and see what happens when you do the reboot through the vnc session and it is 'stuck'. At that point, list all processes that have pypilot in their name (ps -ef | grep pypilot), check out top, a few seconds of vmstat 1, check the status of the pypilot service (systemctl status pypilot.service or systemctl list-units|grep pypilot); check out /var/log/syslog. Copy all the output here, and I'm curious to see a regular picture of that screen at stuck time.
FYI I have a fresh openplotter image here, headless though vnc; I installed pypilot through the Raspberry --> open plotter --> settings facility, and have no problems with the Raspberry -->restart. Which, btw, I never used until today, because I always use sudo reboot ;-)
2021-02-02, 11:25 AM (This post was last modified: 2021-02-02, 12:11 PM by jamos.tan@gmail.com.)
(2021-02-02, 10:46 AM)ironman Wrote: Ok i get it now. I would, parallel to the VNC session, start an ssh session (on windows: with putty.exe) to the same box and see what happens when you do the reboot through the vnc session and it is 'stuck'. At that point, list all processes that have pypilot in their name (ps -ef | grep pypilot), check out top, a few seconds of vmstat 1, check the status of the pypilot service (systemctl status pypilot.service or systemctl list-units|grep pypilot); check out /var/log/syslog. Copy all the output here, and I'm curious to see a regular picture of that screen at stuck time.
FYI I have a fresh openplotter image here, headless though vnc; I installed pypilot through the Raspberry --> open plotter --> settings facility, and have no problems with the Raspberry -->restart. Which, btw, I never used until today, because I always use sudo reboot ;-)
Cheers,
-
Ok thanks! Will try it! And, thanks that you have tested it as well! Did you use 2020-12-16-OpenPlotter-v2-Headless.img and do you have Pypilot installed on it as well?
Maybe also a good idea, if I disconnect all the hardware from the Pi. Or at least the Arduino. And test if there is any difference. I will post the output of your suggestions as soon as possible.
Ok so I cannot SSH over internet at the moment, so here's what I found out by looking into the syslog.
I've looked for anything that has to do with Pypilot, I am not sure by looking at the log whether this was during boot or reboot. But I might have found something.
Sometimes I can see this above the pypilot.service stopping, and sometimes below.
Code:
openplotter systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
Also what I can see is that when the pypilot.service is topping, it is immediately started again.
Code:
Feb 1 15:15:40 openplotter systemd[1]: openplotter-pypilot-read.service: Service RestartSec=2s expired, scheduling restart.
Feb 1 15:15:40 openplotter systemd[1]: openplotter-pypilot-read.service: Scheduled restart job, restart counter is at 1.
Feb 1 15:15:41 openplotter systemd[1]: Stopped openplotter-pypilot-read.
Feb 1 15:15:41 openplotter systemd[1]: Started openplotter-pypilot-read.
Feb 1 15:29:48 openplotter systemd[1]: openplotter-pypilot-read.service: Succeeded.
Feb 1 15:29:49 openplotter systemd[1]: openplotter-pypilot-read.service: Service RestartSec=2s expired, scheduling restart.
Feb 1 15:29:49 openplotter systemd[1]: openplotter-pypilot-read.service: Scheduled restart job, restart counter is at 1.
Maybe I'm looking at it in the wrong way, so I've included the log here: syslog
I'll do the top and vmstat 1 when I can SSH during reboot.
For now, this is all I've got.
2021-02-02, 09:27 PM (This post was last modified: 2021-02-02, 10:19 PM by ironman.)
Yeah on my desk here at home I have a pi4 with that image, but I see now I installed a pypilot from source on this machine, so it is not out-of-the-box. I logged some behaviour in attached file, so you can compare your log with it. There is some strange things in it that don't look right, but that might be due to the fact that I overruled the openplotter distribution. If you really want to get down to it, check out https://github.com/openplotter/openplott...terPypilot.
If you don't like a puzzle, I'd suggest you burn a spanking new image on a new sd card, and see if that gets stuck at reboot. If it does, post it on the openplotter forum. If it does not, do any configuration steps and test your reboot after each step. With a bit of luck, it does not get stuck anymore.
BTW I don't really use the openplotter pypilot; I have a separate tinypilot in use on the boat.
Your log file does not seem to reveal more than the snipplet. That snipplet, btw, concerns this pypilot-read-service that synchronises data from the local pypilot to signalk. That mechanism seems to be obsolete now pypilot recently got zeroconf added.
Thanks for the investigation and the attached file. I can see you've done some digging and found some weird things also.
Do I understand that during your reboot you've experienced the service not being properly closed? Did this also block your reboot?
I have also done some more testing and it seems my conclusion that when doing a normal reboot "always" results in pypilot.service being stuck is not correct.
It happens often, but not always. I did not do any configuration changes so I cannot say if some special configuration makes the difference. I was thinking maybe it has something to do with how long the system has been up, or have I been using Pypilot or something else. At this moment I don't know, it seems to be sporadic.
Also something else, Pypilot's Webapp service is also not connected. So can't use the webapp, it loads but the service is not connected.
I'll keep investigating.
I reverted to an older image I had lying around. Now the pypilot.service is no longer stuck. However, it seemed before that it was just the last process in line before the "kill" command shutting down the PSU or doing the reboot command. But I am really not sure now. It seems to be working again.
