Pypilot and Signal K

[tkurki]

Sean posted on Cruiser's Forum:

The pypilot implementation of signalk differs from the node implementation.   The differences are not great, but it is unclear if an agreement can be found to make them compatible in the near future.

I'd like to dig into this a bit deeper. Where should I start? Are you talking about pypilot being different from the Signal K specification http://signalk.org/ or some minor implementation details?

[seandepagnier]

When I started pypilot I really needed a way to communicate autopilot data between various processes.   At the time signalk specification was even less worked out and the few disagreements I had were never resolved.   Most of my disagreements are related to how large the actual messages (bytes over tcp) compared to what could actually be sent instead.

for example:

Code:

{
  "self": "urn:mrn:signalk:uuid:c0d79334-4e25-4245-8892-54e8ccc8021d",
  "version": "0.9.0",
  "vessels": {
    "urn:mrn:signalk:uuid:c0d79334-4e25-4245-8892-54e8ccc8021d": {
      "uuid": "urn:mrn:signalk:uuid:c0d79334-4e25-4245-8892-54e8ccc8021d",
      "navigation": {
        "courseOverGroundTrue": {
          "value": 3.61562407843144,
          "$source": "ttyUSB0.GP",
          "timestamp": "2017-04-03T06:14:04.451Z"
        }
      }
    }
  }
}

Would be in pypilot signalk:

Code:

{ "gps.track": {"value": 3.6156, "timestamp": "2017-04-03T06:14:04.451Z"}}

I am confused why it needs to specify the vessel uid (twice!) when there is only ever one vessel!

Anyway, I hope there is some way that pypilot can use a compatible subset of signalk  so the above sentence could be valid.

I also now have the issue of sharing larger binary files between processes but maybe signalk is not right for this?

---

btw, I really do like the "minperiod" option for subscriptions. This is not yet implemented and clients have to poll data if they want less frequent updates.


So I am really just looking for a way to simplify the protocol in the most common case (single vessel single data source) but I don't know if it's possible, and pypilot can certainly be extended to support more of the signalk standard, changing key names or methods is no problem, but for now they are purposefully different.

[seandepagnier]

An idea...

A short term solution pypilot could support both formats (listen for tcp sockets on two different ports) and proper comparison/evaluation can be made. If there are no significant or measured disadvantages to the more verbose format it should be adopted.

[tkurki]

So I am really just looking for a way to simplify the protocol in the most common case (single vessel single data source) but I don't know if it's possible, and pypilot can certainly be extended to support more of the signalk standard, changing key names or methods is no problem, but for now they are purposefully different.

For sending updates between processes the delta format is just the thing:

{"updates": {"$source": "GPS", "values": [{"path": "navigation.courseOverGround", "value": "3.14159"}]}}
OR
{
   "updates": {
       "$source": "GPS",
       "values": [
           {
               "path": "navigation.courseOverGround",
               "value": "3.14159"
           }
       ]
   }
}
A bit more verbose, but context is optional (assumed to be "self") and timestamp also. A bit more complexity in the format allows the format to be used to send several updates at a time. In the end this is key-values, with a bit of metadata on the envelope.

The format you quoted is the full format, that I practically never use when coding, except when debugging stuff, because it is very easy to use a browser to see what's going on and get a snapshot of the data. 

With http you can request

everything: http://demo.signalk.org/signalk/v1/api/
just "self" data: http://demo.signalk.org/signalk/v1/api/vessels/self
a subtree: http://demo.signalk.org/signalk/v1/api/v...nvironment

but for streaming updates you want to do tcp or ws and deltas.

[seandepagnier]

Thanks for the pointers.  

It is definitely more complex but also more flexible than what pypilot currently is using.  It really isn't that far off and would be exciting to make fully compatible.  

I think the starting point for pypilot output would be the "subtree" with updates because at least for now, the intention is not to relay signalk data.   It might convert a few things from nmea but that is all, the rest of the data will originate from pypilot itself.

 Obviously the initial message should contain meta data (it does already in pypilot) to describe the unit type and some other information.

What is a little bit different currently in pypilot is that the "source" for gps, wind and other data is actually a data field of its own that can be subscribed to.   This is really useful to for display, and also the watchdog plugin of opencpn can subscribe to the wind.source without caring about the wind data itself and so this data isn't sent, but it wants to sound an alarm if the sensor is lost and the source becomes "none"    This is also used by nmea parsing which happens in a separate process (because pypilot uses realtime processes for critical control loops) and it can know not to bother sending the data if it received nmea from a lower priority source than what is already being used.


I don't think this is a problem for signalk and should probably be a feature of pypilot, so it doesn't need to affect the standard or anything like that.   It is a different field in the sense that it is the source the autopilot is actually going to use which may be different from other data passing through.

I am wondering, if the $source field is omitted, would the client just assume the source is whoever they connected to, or is it whatever the last value for $source was set to?

Code:

{"wind":
    {"speedApparent":
        {"meta": {"units":"m/s","description":"Apparent wind speed"},
         "value":8.498624375207065,
         "$source":"nmeaFromFile.II",
         "timestamp":"2020-01-27T00:13:23.047Z",
         "sentence":"MWV"}
}
}

Would this then update with?

Code:

{"updates": {"$source": "nmeaFromFile.II", "values": [{"path": "wind.speedApparent", "value": "3.14159"}]}}

Could it just be instead:

Code:

{"wind.speedApparent": {"value":8.498624375207065, "timestamp":"2020-01-27T00:13:23.047Z"}}
   

This is really just the subtree format with fields omitted which is incidentally the exact format pypilot already uses
I am trying to understand the logic of "updates" is it to share fields like $source and timestamp such as:

Code:

{"updates": {"$source": "nmeaFromFile.II", "timestamp":"2020-01-27T00:13:23.047Z", "values": [{"path": "wind.speedApparent", "value": 3.14159},{"path": "wind.directionApparent", "value": 123}]}}

Is this correct?   and if so, what about instead:

Code:

{"wind": {"$source": "nmeaFromFile.II", "timestamp":"2020-01-27T00:13:23.047Z", "speedApparent": {"value": 8.498}, "directionApparent": {"value": 123}}}

This seems more intuitive to me, and does away with the extra rules needed for "updates" and "values" since everything is supported as the standard format with fields omitted.
It could even possibly be simplified to:

Code:

{"wind": {"$source": "nmeaFromFile.II", "timestamp":"2020-01-27T00:13:23.047Z", "speedApparent": 8.498, "directionApparent": 123}}

But this means that if there is one field rather than the dictionary it is assumed to be "value" and this adds an extra rule which is maybe not better or maybe ok?
To share source from fields that do not share other fields:

Code:

{"$source": "nmeaFromFile.II", "wind.speedApparent": {"value": 8.498, "timestamp":"2020-01-27T00:13:23.047Z"}, "navigation.speedThroughWater": {"value": 3.53, "timestamp":"2020-01-27T00:24:09.421Z"}}}

I would like to open the discussion to everyone's input.  My intention is to keep the format intuitive, have as few "rules" as possible making it much easier to implement software, but also keep the actual data transmitted minimal (avoid redundancy)

Just because a previous format was already used, or some device exists using it is not really a good enough reason to keep it in my opinion.   The standard can remain dynamic.    I am sure I am overlooking some important factors which need to be taken into consideration.

[seandepagnier]

Hi,

I've dug into the signalk stuff a bit more and closer to finding a solution for pypilopt.

Basically I can have a driver for the server which processes messages in the same format as signalk node server without any real downsides,

My biggest question is.. If pypilot connects to signalk-node, and/or provides signalk as a piped provider, how will it get subscriptions from other clients about the data it provides?

There are a lot of keys pypilot can provide that might be expensive to compute as well as transmit, so it would need the subscriptions of clients.

[tkurki]

Hi,

I've dug into the signalk stuff a bit more and closer to finding a solution for pypilopt.

Basically I can have a driver for the server which processes messages in the same format as signalk node server without any real downsides,

My biggest question is..  If pypilot connects to signalk-node, and/or provides signalk as a piped provider, how will it get subscriptions from other clients about the data it provides?

There are a lot of keys pypilot can provide that might be expensive to compute as well as transmit, so it would need the subscriptions of clients.

An interesting use case, something that hasn't come up before exactly in this form.

So would pypilot act as a Signal K client or a server? Upon connection it could advertise a list of keys it has data for. Then we could add a feature to the SK server where you can activate subscription forwarding, whereby SK server's clients' relevant subscriptions would be passed on to pypilot. With most SK clients that would be the default "all for self" subscription.

A reasonably complicated feature, as SK server would need to keep track of overlapping subscriptions, or just default to subscribing/unsubscribing to all whenever there is some client connected.

What do you think?

[seandepagnier]

An interesting use case, something that hasn't come up before exactly in this form.

So would pypilot act as a Signal K client or a server? Upon connection it could advertise a list of keys it has data for. Then we could add a feature to the SK server where you can activate subscription forwarding, whereby SK server's clients' relevant subscriptions would be passed on to pypilot. With most SK clients that would be the default "all for self" subscription.

A reasonably complicated feature, as SK server would need to keep track of overlapping subscriptions, or just default to subscribing/unsubscribing to all whenever there is some client connected.

What do you think?

From what I can tell, pypilot needs to act as both a provider and consumer of data to work correctly with signalk.   A starting point would be to have program which hosts a website that you can configure which sentences get forwarded in each direction.

A more flexible solution would be preferred, which is needed for a better integration.

If there were a way to register that you can provide data for a certain field, then the server could notify you with subscribe messages if there are subscribers.  If there are any then the client can produce the data at the fastest subscribers rate.

signalk could no longer care about client vs server messages for subscriptions.  Instead both clients and serers can send subscriptions.  This would also make it easy to link signalk servers and keep subscriptions if the server could create a client which forwarded all of it's subscriptions both ways.

[seandepagnier]

It has taken me a while.. I rewritten pypilot data handling to make it more efficient/flexible and added initial support for signalk node.

The signalk node server is discovered and connected by a websocket and messages are translated in both directions depending on what data is available.   For example, if rudder feedback comes from the motor controller of pypilot, then it outputs on signalk, but if not, and it is found on signalk, then it will use this as input to pypilot.  The same is true of wind sensor.. if nmea0183 sensor is plugged directly into usb of tinypilot it is going to use that and output wind to signalk, but otherwise it can read wind from signalk.   This should eliminate the need to use nmea0183 over tcp sockets completely (but still supported)

The subscriptions to signalk-node are managed by pypilot to minimize network traffic.   It would be really interesting to do various latency tests such as routing wind over wifi through signalk. if the latency is more than 100 milliseconds compared to a direct connection it will have performance implications to the autopilot, but at least it's going to be supported.   For gps this probably doesn't matter.


What I would really like to see in signalk in the future is the server sending subscriptions to clients.    pypilot now has a bi-directional format using subscriptions so clients do not need to produce data faster than needed.   It also allows various processes to produce data (such as automatic compass calibration or machine learning background training) without interfering with the critical control loop.  Without this the cpu and data use can be very high.   This also makes it possible to link servers quite easily while maintaining the minimum data traffic.   Once signalk node supports this, the full pypilot values tree can be "mounted" into signalk providing more than 100 values such as:

vessels.self.pypilot.ap.heading_error
vessels.self.pypilot.servo.amp_hours
vessels.self.pypilot.rudder.range

These could be accessible by signalk programs to subscribe or set these values, node red alarms etc...    The current translation would be used to convert units (pypilot uses degrees not radians) for generic keys not specific to pypilot, but these would benefit from subscriptions as well to reduce data and cpu load.  For example, pypilot is always producing signalk navigation.attitude and navigation.headingMagnetic even if there are no clients to signalk-node that consume these.

I think the change is fairly straightforward.  I just need signalk-node-server to set the period minperiod and policy for subscriptions in a way that allows it to provide the subscriptions it receives.   This would support connecting two signalk-node servers to be linked by a connection maintaining minimal bandwidth and processing load.

[seandepagnier]

tkurki,

I have been working on signalk integration and now pypilot detects signalk with zeroconf, finds the url for the websocket using http requests, then connects to the websocket.

pypilot then subscribes to signalk for wind, gps, rudder, or autopilot messages etc if it doesn't have any other higher priority input (like a direct serial connection) It unsubscribes from signalk if a higher priority source is found. This seems to be working.

This is a small annoyance since the server rejected unsubscribe requests, I have to unsubscribe from all and re subscribe to all the keys. Is this by design or just lacking implementation? Also, subscribing to a signalk key the server sends the last value even if it is days old.

So now I have produced various signalk outputs and send them down the websocket as "updates", but this is not working. On the web site localhost:3000 I see:

ws
::1 needs authentication


Is this the reason my data seems to be ignored? The current pypilot signalk integration in openplotter just sends a udp packet to a port configured as input in signalk server but I would like to avoid this for a number of reasons, and I already have the websocket connection.