This forum uses cookies
This forum makes use of cookies to store your login information if you are registered, and your last visit if you are not. Cookies are small text documents stored on your computer; the cookies set by this forum can only be used on this website and pose no security risk. Cookies on this forum also track the specific topics you have read and when you last read them. Please confirm whether you accept or reject these cookies being set.

A cookie will be stored in your browser regardless of choice to prevent you being asked this question again. You will be able to change your cookie settings at any time using the link in the footer.

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Connect 2 Signalk servers via node-red / MQTT for cloud access
#1
I have been thinking about the best way to get some of my signalk data into the cloud with intermittent internet. The requirement is that while there is no internet connection, the data accumulates on the raspberry pi and automatically gets uploaded when a connection is established.

If anyone has a solution for this please let me know.  My thought was to run a separate Signalk server in the cloud, and use node-red and MQTT to pass signalk messages to the cloud instance.  This would essentially mean the cloud instance would receive raw signalk data from the boat.

I have managed to get the infrastructure up and running, with the following flow.

ON BOAT -> Signalk-node-red -> singalk-on-delta -> MQTT

IN CLOUD -> MQTT -> signalk-send-delta

I have setup debugging on both ends and can confirm that the messages are being received on the cloud server, but for some reason they are not being processed correctly. 

I believe the issue is that for some reason the structure of the signalk delta is modified during sending, and it is not being reconstructed correctly on the cloud end - I have tried changing and editing various things, but cannot get it working.

Has anyone tried anything similar, or have a different solution for getting the data into the cloud on an intermittent internet connection?

Thx
Reply
#2
This app does just that, though limited to propulsion and navigation paths so far. https://www.npmjs.com/package/signalk-to...-buffering
You need a cloud instance of influxdb running.
Reply
#3
Here is an implementation of Signal K over MQTT cloud bridge, with local buffering in Mosquitto: https://github.com/tkurki/signalk-mqtt-gw/pull/17

It is work in progress, but working. Well, more like stalled to be more precise.

I can continue with that if there is interest.
Reply
#4
(2021-03-05, 06:18 PM)tkurki Wrote: Here is an implementation of Signal K over MQTT cloud bridge, with local buffering in Mosquitto: https://github.com/tkurki/signalk-mqtt-gw/pull/17

It is work in progress, but working. Well, more like stalled to be more precise.

I can continue with that if there is interest.

@tkurki - I managed to get this working with a vanilla setup.  I had to do a bit of fiddling to get the signalk delta into exactly the right format - it was getting changed slightly during the transfer over MQTT which was creating issues on the receiving end.

The other issue I ran into was that node-red's MQTT wrapper has a bug that means the MQTT QoS is ignored.  Even if you set it to QoS level 1, if the client is not connected to the server, node-red will not queue the messages and they get lost.

The issue is tracked here https://github.com/node-red/node-red/issues/2864

There is a temporary workaround that seems to be working for me which you can find in the issue tracker

This solution seems to be working quite well so far - messages are queued on the raspberry PI on the boat and as soon as the MQTT connection is established, all the messages are sent to the signalk server in the cloud

Im wondering if the cloud bridge you are working on is trying to achieve the same thing, or is it different?

Happy to elaborate on any areas if there are questions
Reply
#5
Are the messages queued in memory or on disk? Meaning do they survive a node-RED restart and is it limited by memory or disk space?

My solution runs a local broker that is bridged to the cloud broker and does the queuing on disk and ddoes not strain memory. I first experimented with buffering in mqtt client, but there is a fundamental flaw in that mechanism that may cause data loss.
Reply
#6
(2021-03-05, 11:30 AM)PaddyB Wrote: This app does just that, though limited to propulsion and navigation paths so far. https://www.npmjs.com/package/signalk-to...-buffering
You need a cloud instance of influxdb running.

This app now does what you asked.
Reply
#7
(2021-03-07, 11:10 PM)tkurki Wrote: Are the messages queued in memory or on disk? Meaning do they survive a node-RED restart and is it limited by memory or disk space?

My solution runs a local broker that is bridged to the cloud broker and does the queuing on disk and ddoes not strain memory. I first experimented with buffering in mqtt client, but there is a fundamental flaw in that mechanism that may cause data loss.

Messages are queued in memory and will not survive a rode-RED restart - your solution does sound more robust.

@paddyB - thanks will check out the repo - great to see it being actively developed
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)