+- OpenMarine (https://forum.openmarine.net)
+-- Forum: Pypilot (https://forum.openmarine.net/forumdisplay.php?fid=17)
+--- Forum: Feature requests (https://forum.openmarine.net/forumdisplay.php?fid=21)
+--- Thread: Pypilot on Micropython? (/showthread.php?tid=5319)
Pages:
1
2
Pypilot on Micropython? - dave42w - 2024-04-09
Hi,
I'm wondering whether there is any potential for getting the core of Pypilot running on Micropython on a Pi Pico (or similar embedded board)?
I wonder if it would have some advantages by not needing a full OS. There are some quite powerful boards available with more memory etc available. Goals would include power requirements, security and plug&play.
Thanks
Dave
RE: Pypilot on Micropython? - CapnKernel - 2024-04-09
I'm very interested in not using a board with an OS because of the time it takes to boot. If the PyPilot we have today loses power, it's around 40 seconds before the autopilot is back on line. That's a loooong time if you're on your boat and hell's breaking loose, and you really need the autopilot to take the helm now.
I've looked into the possibility of a MicroPython port. I see six main obstacles:
- Multiprocessing / multithreading. As currently written, PyPilot spawns Linux processes, and uses the Python multiprogramming library. Neither of these is supported by MicroPython. MicroPython programs should be written to use async/await, and/or use the _threading library, although support for the latter is still experimental. Rewriting to use these would result in code that would be too different to keep in one codebase, although code written to use async/await may work In CPython and MicroPython.
- PyPilot as written uses TCP sockets, unix domain sockets, and does I/O by doing reads/writes on Linux character device files. The way MicroPython programs do networking, interprocess communication, and device I/O is quite different. See above re divergent codebases.
- Some of the autopilot algorithms use the TensorFlow AI library. Not a dealbreaker as the main algorithm doesn't.
- PyPilot as written uses the RTIMULib2 IMU library. This library would either need to be ported to an MCU, or replaced with something else.
- PyPilot as written calls SciPy to do IMU calibration (least squares calculation, and Orthogonal Distance Regression). SciPy is never going to run on an MCU. This code would need to be replaced with something else.
- Sean may give his blessing for such a port, and may not. Who would work on a port if Sean did not?
I would be very interested to hear what others think.
RE: Pypilot on Micropython? - dave42w - 2024-04-09
Hi,
Thanks, that's really helpful.
I agree the boot speed is potentially very significant.
The points you raise all make sense. I would have hoped to avoid a divergent codebase but that is a lot of functionality that needs to be re-written.
re 5. (SciPy) not sure if this will help https://micropython-ulab.readthedocs.io/en/latest/ulab-intro.html
I'd like to contribute (with potentially a lot more time to do so coming in about 15 months).
Dave
RE: Pypilot on Micropython? - seandepagnier - 2024-04-12
(2024-04-09, 12:13 PM)CapnKernel Wrote: I'm very interested in not using a board with an OS because of the time it takes to boot. If the PyPilot we have today loses power, it's around 40 seconds before the autopilot is back on line. That's a loooong time if you're on your boat and hell's breaking loose, and you really need the autopilot to take the helm now.I can agree about the boot speed being an issue. It could in theory be optimized a little, the boot speed is already much faster on quad core. To a lesser extent, power consumption (and cost) is a bit lower on these boards too but not really significant. It would take a lot of work to support certain features of pypilot, but a working autopilot should be the goal first.
There are 2 other potential things that could help with this:
run pypilot with it's own power pack offering it sort of like a UPS to last at least an hour or so (including moving the motor) using one or more 18650 cells. You said "if pypilot loses power" that is going to be an issue for any autopilot, so if by this you mean a fuse blows or something like that and you replace it, then yes boot speed is annoying, but having backup power pack is already better no matter what the autopilot boot speed.
standby pypilot -- It is in planning stages, but to offer the ability to have 2 (or more) pypilots running at the same time. Only one is working, however if the standby pilot detects the main one is not working, eg:
1) software crash
2) power failure (wires powering stuff on primary pilot break)
3), motor controller fails for whatever reason, eg: fuse blows, or water gets on it etc
4) inertial sensors not working, or sudden distortion (someone moved a magnet near the compass sensors)
5) just generally fails to hold course (wrong settings)
the standby pilot would then activate the clutch output on its motor controller which is wired to a relay that allows it to take over the motor without back driving the primary motor controller, and it would take over steering. Other configurations could be to have 2 hydraulic pumps for added redundancy in drives where this is possible, but the idea is having a standby pilot. This could of course have one or more pilots with #1 (backup battery pack)
Quote:I've looked into the possibility of a MicroPython port. I see six main obstacles:[*]There is at least preliminary work in place to decouple this. So to sequentially execute the critical parts. It is not really tested, but search for "use_multiprocessing" and multiprocessing in the code and there are some initial tests in place to make pypilot work as a single process. This is not really better if processes can be used. In more recent micropython it seems it _is_ possible to spawn multiple threads each running as a separate freertos thread (can utilize dual core) and execute python code this way. Not sure how viable it is without external ram (psram) It might make sense to use fewer processes but not only a single one in this way, to ensure that still the main process that performs the control loop is prioritized.
[*]Multiprocessing / multithreading. As currently written, PyPilot spawns Linux processes, and uses the Python multiprogramming library. Neither of these is supported by MicroPython. MicroPython programs should be written to use async/await, and/or use the _threading library, although support for the latter is still experimental. Rewriting to use these would result in code that would be too different to keep in one codebase, although code written to use async/await may work In CPython and MicroPython.
Quote:[*]PyPilot as written uses TCP sockets, unix domain sockets, and does I/O by doing reads/writes on Linux character device files. The way MicroPython programs do networking, interprocess communication, and device I/O is quite different. See above re divergent codebases.[*]Not sure this is correct. At least not in the sense that it would not work because the read/write is fully implemented on micropython using the same calls. Already the pypilot_hat works on micropython as a control interface to pypilot.
Quote:[*]Some of the autopilot algorithms use the TensorFlow AI library. Not a dealbreaker as the main algorithm doesn't.[*]Yes, not important at all, the tensorflow is not even working yet (still)
Quote:[*]PyPilot as written uses the RTIMULib2 IMU library. This library would either need to be ported to an MCU, or replaced with something else.[*]I dont see a huge reason why this library could not be given micropython support wrapper. Another option that could be easier/more efficient would be to rewrite the icm20948 driver specifically for esp32 and utilize dma, then either wrap this in something that works like rtimulib2, or just support this new library.
Quote:[*]PyPilot as written calls SciPy to do IMU calibration (least squares calculation, and Orthogonal Distance Regression). SciPy is never going to run on an MCU. This code would need to be replaced with something else.[*]Yes, it is relatively straightforward to implement the regression using matricies. Essentially a matrix inverse, and while would use less processing power it is also less flexible (requires partial derivatives) I think instead a different algorithm could be implemented though to replace the scipy.optimize.leastsq or perhaps even just port this one function (since all of scipy is too big to consider)
Quote:[*]Sean may give his blessing for such a port, and may not. Who would work on a port if Sean did not?[*]I do want such a port, but if possible I would like to separate the esp32 specific parts into different python files, and reuse as much of pypilot as possible in regular python, similarly to the pypilot_hat port. This way it is possible to maintain pypilot adding new features and support micropython in the main code base without a separate fork.
RE: Pypilot on Micropython? - dave42w - 2024-04-12
Thanks for your reflections and help!
Quote:I can agree about the boot speed being an issue. It could in theory be optimized a little, the boot speed is already much faster on quad core. To a lesser extent, power consumption (and cost) is a bit lower on these boards too but not really significant. It would take a lot of work to support certain features of pypilot, but a working autopilot should be the goal first.
This is what interests me (a basic working autopilot). I'd be happy to work on that (albeit with not a lot of time available yet).
The battery backed autopilot is also of interest to me but that it should work whatever the hardware platform for pyPilot and the motor is going to be the biggest energy user.
The standby pilot is not so interesting for me as I plan to have windvane self steering to give the second option and would be planning to use that mostly on longer passages.
Quote:There is at least preliminary work in place to decouple this. So to sequentially execute the critical parts. It is not really tested, but search for "use_multiprocessing" and multiprocessing in the code and there are some initial tests in place to make pypilot work as a single process. This is not really better if processes can be used. In more recent micropython it seems it _is_ possible to spawn multiple threads each running as a separate freertos thread (can utilize dual core) and execute python code this way. Not sure how viable it is without external ram (psram) It might make sense to use fewer processes but not only a single one in this way, to ensure that still the main process that performs the control loop is prioritized.
I have done a quick search and read. I had hoped that the microcontrollers had got powerful enough to be able to achieve pypilot directly onto the board with micropython. If we had to add an OS and extra hardware then it seems like a whole lot of coding work while losing most of the potential advantage.
Looking around it seems that there are Pi Pico clones with up to 16Mb flash memory and direct support for LiPo eg charging https://coolcomponents.co.uk/products/pico-lipo-16mb
So given the RP2040 has two cores I wonder if that makes the idea of a basic autopilot in just micropython more feasible (maybe as the standby, with an instant power up and battery backup it might be a good fit?)
But this seems a little different from the use_multiprocessing which is either multiple processes or a single thread. As there is (at least experimental) support in Micropython for threads and the RP2040 has 2 cores would it be possible to use the second core for a thread handling a queue of tasks currently handled by external processes?
Given that python is progressing toward eliminating the GIL maybe a threading alternative to multiprocessing might make sense on future Python releases as well as Micropython (providing the code handles both 2 and n cores)?
Quote:[*][*][*]Quote:[*]PyPilot as written uses the RTIMULib2 IMU library. This library would either need to be ported to an MCU, or replaced with something else.[*]I dont see a huge reason why this library could not be given micropython support wrapper. Another option that could be easier/more efficient would be to rewrite the icm20948 driver specifically for esp32 and utilize dma, then either wrap this in something that works like rtimulib2, or just support this new library.
Is this already available? see https://micropython-icm20948.readthedocs.io/en/latest/api.html
Quote:Yes, it is relatively straightforward to implement the regression using matricies. Essentially a matrix inverse, and while would use less processing power it is also less flexible (requires partial derivatives) I think instead a different algorithm could be implemented though to replace the scipy.optimize.leastsq or perhaps even just port this one function (since all of scipy is too big to consider)
It is more than 40 years since I did matrix maths for A-Level :-) Googling seems to show that various people have needed this function and done some work on it. But I don't know enough to even ask sensible questions :-(
Quote:I do want such a port, but if possible I would like to separate the esp32 specific parts into different python files, and reuse as much of pypilot as possible in regular python, similarly to the pypilot_hat port. This way it is possible to maintain pypilot adding new features and support micropython in the main code base without a separate fork.
Thanks. I am interested in this and in contributing towards it. I am absolutely uninterested in forking PyPilot and would want to keep the changes as minimal as possible and as useful for the standard PyPilot as possible.
I guess my first step is to setup a Pi5 and get pypilot building and running on it. We have an original 1977 Neco with broken control panel and I want to get PyPilot working with that. Once I have a working PyPilot then the next step would be to start porting onto a microcontroller (which would get me nicely into understanding how the code base works).
Timing wise we move to live aboard in August 2025 and have a lot of basic boat refit to finish first. So it might be that the Neco project doesn't get much love before then.
Dave
Sustainable Sailing
RE: Pypilot on Micropython? - seandepagnier - 2024-04-12
So.. I think the battery-backed autopilot would eliminate most of the issues with slow boot speed (from power lost) as well as provide even more reliability. I will look into setting this up on my boat and begin testing so that I can advise people on how to do it. It is really a separate system and could of course be used with an esp32 system as well.
As for the icm20948 driver on micropython... It may exist, I am sure several people have implemented one, but I worked specifically on the rtimulib one and there are 2 factors:
1) I optimized the driver heavily. It samples the sensors much faster (100hz) and averages them using the fifo. Without this it would use more cpu and/or worse data output, so I would likely have to modify any driver and/or verify it to ensure the data is as good as possible.
2) RTIMULib also implements sensor fusion, using a kalman filter (although optionally other filters) This is all done in c++ which benefits pypilot from reduced cpu on the computation and reduced latency in the control loop. So sampling the raw sensors alone is not enough it needs this as well.
As for a micropython port, I think starting out, disabling certain features that are in separate processes could be appropriate like zeroconf, signalk, and initially even nmea. Honestly if it can be shown that multiple threads in micropython can run as separate freertos threads and therefore utilize separate cores, then this might be the way to go, and just implement python multiprocessing this way (avoid changing the main code and just add a special module that emulates everything needed including inter-process pipes) but the limited ram on these boards could be the bottleneck and it is unlikely to work without more memory. In fact when running pypilot_hat on micropython I did get it to work without external ram, but had to do some careful optimizing of the display driver. I have some esp32 dev boards which have 4mb PSRAM and it could be determined that this is the way to go as it should be plenty of memory.
From what I recall, without external memory, there is < 100kb of free ram for the program itself in a single python thread. it is just not going to be able to support several python processes without PSRAM. I think at minimum there should be 2 processes, one with critical control loop ensuring the best reaction time, and another with everything else. Right now pypilot uses several processes, with 2 in the critical loop (so it can begin to read from the sensors for the next cycle and they will be ready as soon as possible while it is still performing control loop computations) For example, once it completes the control loop and sends the servo command to move the rudder, there is still a bunch of other stuff it needs to do like figure out if it should drop the mode (gps lost) but also to send and receive data from the server to communicate with other processes, and during this time, the other critical control process (the one that basically just runs RTIMULIB ) can already be reading the sensor data so that there is minimal delay when the main control loop process wants it. Perhaps it could be reorganized into a single critical process though, this is something to consider but testing would be needed to verify it, and instrumentation to verify the delay. a few milliseconds might not seem like much but every millisecond counts in this control loop as any delay in reaction results in larger corrections needed which is almost always bad. To be clear, if pypilot were purely c++ (no longer pypilot) it would shave some milliseconds off and improve reaction time slightly, and use far less ram and therefore already work on esp32, but become un-portable, unmaintainable and likely to suffer from bugs that cause it to crash whenever new features are added (unstable) which to me is not worth it. Even after 20 times the development effort, opencpn crashes.. pypilot does not really crash, but sometimes prints (usually informative) information to the log about exceptions when there are bugs. An autopilot that occasionally crashes from software bugs is a bad thing indeed.
Other features of pypilot are just not going to fit without more memory like running the hat service and the web. As for the web service.. likely an entirely new code to service it would be required, but fortunately the web servers for micropython already exist and could likely still use the same java and pypilot/web.py is already only 272 lines so a re-implemented web-esp32.py would probably be most sensible here. It is unclear if a single esp32 has the processing power even with dual cores to handle everything. Currently the pi zero uses about half the cpu to run all pypilot processes, and the pi zero has 3-4x the processing power of an esp32.
Another possibility is to use 2 or more esp32 to run separate services, with one running the core pilot, another for the web service and so on. Another way could be to slow down the data handling to non-critical paths, eg: service data at 4hz instead of 20hz, to external processes and so forth.
There is an upcoming esp32-p4 that might be the most suitable for pypilot. It offers superior processing power but potentially still not enough ram, it is hard to say at this stage.
RE: Pypilot on Micropython? - mgrouch - 2024-04-12
There is already existing open source autopilot made on microcontroller.
See: https://fenix-autopilot.com/
And: https://github.com/spascual90/Fenix
RE: Pypilot on Micropython? - dave42w - 2024-04-13
Hi Sean,
Thanks again. I have learned a great deal from this thread. It has been awesome to learn more of the scope of the pypilot project, such as the work on rtimulib.
It is only recently that I have started experimenting with micro-controllers and micropython with plans for a variety of tasks around our boat. This discussion has lead to a lot of new discovery for me. I had dipped my toes in with the Pi Pico W and now see so many alternatives with lots of potential.
I'm a long time Linux user (since 1998), initially as a software development but diverted 20 years ago into becoming a Methodist Minister, now with retirement approaching I'm getting back to the joy of coding with some Rust and Python.
My conclusion from this is that while interesting, porting pypilot to micro-controllers with micropython isn't a great use of time/resources. By the time a top spec ESP32 is tricked out with all the memory it is way more expensive than a Pi Zero. Plus as you say the combination of a battery backed Pi Zero (with potentially a standby pypilot) solves the key issues.
I did find this thought interesting:
Quote:Another possibility is to use 2 or more esp32 to run separate services, with one running the core pilot, another for the web service and so on.
The idea of packaged micro controllers for the individual services is interesting and potentially a nice way to add redundancy. But again the time to make the changes to the way the services within pypilot communicate may not be a great return.
Dave
RE: Pypilot on Micropython? - seandepagnier - 2024-04-13
Faster boot speed and lower power consumption would be achieved and I wont ignore these facts. Especially on really small boats with very limited power.
the slow boot speed of pypilot has long been annoying but mostly tolerable because of the stability of the system. Once I support pi zero 2 in the near future, the boot speed would be under 20 seconds and while still a while, this isn't that bad... There are likely potential tricks to get it working even faster in 10 seconds or less that could be explored, but an esp32 without linux would still be potentially operational in 1 second from power which is impossible to beat using linux as far as I know.
RE: Pypilot on Micropython? - dave42w - 2024-04-13
Sean,
A couple of things.
Pi zero 2: What needs doing for this? When I buy for our system it seems sensible to buy the Zero 2, some places not selling the zero any more. In terms of getting a working system I have a Pi5 handy, will that work.
If there are pypilot tasks that a rusty, ancient developer can help with let me know.
Meanwhile if there is a particular microcontroller that you think it would be worth getting to experiment with then let me know (just looking and there is a bewildering list of potential boards from different companies).
Dave