OpenMarine

Full Version: Signal k TCP data source ... help please
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
I am using the version 1.16.0 of Signal k server.
I've configured a connection under Server/Connections/Add:
The options which I've selected are:
Input type = Signal K
Signal K Source = TCP
I've configured the address & port and the Server makes a connection to a TCP server as confirmed by the connection status on the dashboard.

However the Dashboard indicates zero deltas/second.
The source is converting XDR data and delivering it via a TCP server.
It is being received and logged.
Here is an example of a delta which is being delivered to the Signal K connection.
Can anyone advise why it is not being handled?
I've used the Playground to confirm that the delta format is correct.
I'm at a loss although I suspect that it is something to do with the source section, the significance of which I admit I don't really understand at this point.

Many thanks

{
  "context": "vessels.urn:mrn:imo:mmsi:601270700",
  "updates":
  [
    {
      "source":
      {
        "label": "XDR_Data",
        "type": "NMEA0183",
        "talker": "II",
        "sentence": "XDR"
      },
      "values":
      [
        {
      "path": "environment.outside.air.temperature",
      "value": 279.85
        }
      ]
    }
  ]
}
The input must be one complete delta per line.

How does your input look if you use netcat to connect to your source?




Sent from my iPhone using Tapatalk
(2019-06-30, 11:00 PM)tkurki Wrote: [ -> ]The input must be one complete delta per line.

How does your input look if you use netcat to connect to your source?




Sent from my iPhone using Tapatalk

The data displays in netcat exactly as I have posted it. It also displays in the Signalk log file exactly the same (with the timestamp info at the start of each line).

I can try to remove the '\n', '\t' and spaces from the code but I'd be surprised given that the Playground accepts it and also all of the documentation shows the "beautified" json.

Thanks for the reply, much appreciated.

(2019-07-01, 01:07 AM)garysmith Wrote: [ -> ]
(2019-06-30, 11:00 PM)tkurki Wrote: [ -> ]The input must be one complete delta per line.

How does your input look if you use netcat to connect to your source?




Sent from my iPhone using Tapatalk

The data displays in netcat exactly as I have posted it. It also displays in the Signalk log file exactly the same (with the timestamp info at the start of each line).

I can try to remove the '\n', '\t' and spaces from the code but I'd be surprised given that the Playground accepts it and also all of the documentation shows the "beautified" json.

Thanks for the reply, much appreciated.

Actually the input/output is a stream of data so I've attached a logfile (not the Signal k logfile) captured using netcat

[attachment=468]
This time with a valid attachment
I'd be surprised if putting the whole delta on a single line would not work given that I am one of the primary authors of the server and know how things work underneath the hood. If you know better feel free to ignore my advice.

The issue here is that a TCP stream is just a stream of bytes, there is no concept of a message in TCP. The receiver either needs to parse the input on the fly, balancing parentheses and breaking the input to delta messages, or we can choose a message delimiter. We have chosen newline as the delimiter, making the receiver much simpler, it just splits the input to messages by lines.

Once the input is split to individual messages newlines and whitespace no longer matter, each message is parsed as a JSON document. Documentation is formatted for easy reading and reflects this. Playground parses a single message as JSON. If you paste more than one message to Playground it will complain about invalid JSON - it is trying to handle just a single message, not the contents of a stream.

If you start the server in demo mode with sample data playback and netcat to port 8375 you will see deltas, one per line. This should be your reference - this is what the Signal K tcp input handles.
(2019-07-01, 09:08 AM)tkurki Wrote: [ -> ]I'd be surprised if putting the whole delta on a single line would not work given that I am one of the primary authors of the server and know how things work underneath the hood. If you know better feel free to ignore my advice.

The issue here is that a TCP stream is just a stream of bytes, there is no concept of a message in TCP. The receiver either needs to parse the input on the fly, balancing parentheses and breaking the input to delta messages, or we can choose a message delimiter. We have chosen newline as the delimiter, making the receiver much simpler, it just splits the input to messages by lines.

Once the input is split to individual messages newlines and whitespace no longer matter, each message is parsed as a JSON document. Documentation is formatted for easy reading and reflects this. Playground parses a single message as JSON. If you paste more than one message to Playground it will complain about invalid JSON - it is trying to handle just a single message, not the contents of a stream.

If you start the server in demo mode with sample data playback and netcat to port 8375 you will see deltas, one per line. This should be your reference - this is what the Signal K tcp input handles.
And after that authoritative answer I'd be surprised if that wasn't the solution :-). Thank you, much appreciated.

And just for everyone else ... It works. No newlines in a delta or full message.
(2019-07-01, 01:24 PM)garysmith Wrote: [ -> ]
(2019-07-01, 09:08 AM)tkurki Wrote: [ -> ]I'd be surprised if putting the whole delta on a single line would not work given that I am one of the primary authors of the server and know how things work underneath the hood. If you know better feel free to ignore my advice.

The issue here is that a TCP stream is just a stream of bytes, there is no concept of a message in TCP. The receiver either needs to parse the input on the fly, balancing parentheses and breaking the input to delta messages, or we can choose a message delimiter. We have chosen newline as the delimiter, making the receiver much simpler, it just splits the input to messages by lines.

Once the input is split to individual messages newlines and whitespace no longer matter, each message is parsed as a JSON document. Documentation is formatted for easy reading and reflects this. Playground parses a single message as JSON. If you paste more than one message to Playground it will complain about invalid JSON - it is trying to handle just a single message, not the contents of a stream.

If you start the server in demo mode with sample data playback and netcat to port 8375 you will see deltas, one per line. This should be your reference - this is what the Signal K tcp input handles.
And after that authoritative answer I'd be surprised if that wasn't the solution :-). Thank you, much appreciated.

And just for everyone else ... It works. No newlines in a delta or full message.

@tkurki

help again please ....

I've attached a log file of the output from my server using netcat to connect to port 8375.

I've grepped it to reduce the volume of data. It only contains data pertaining to the environment path.

There are two sources of data - one is where the TCP server does the conversion from non XDR strings and the other is from my XDR converter

Here is an extract of one of each delta message:
a) The Signal K generated delta:
{"updates":[{"source":{"sentence":"MWV","talker":"II","type":"NMEA0183","label":"nmea_no_xdr"},"timestamp":"2019-06-23T23:42:23.579Z","values":[{"path":"environment.wind.speedApparent","value":11.62},{"path":"environment.wind.angleApparent","value":0.7674212722771316}]}],"context":"vessels.urn:mrn:imo:mmsi:601270800"}

b) The delta from my converter (after passing through the Signal K server
{"context":"vessels.urn:mrn:imo:mmsi: 601270800 ","updates":[{"source":{"label":"xdr_to_signalk","type":"NMEA0183","talker":"II","sentence":"XDR"},"values":[{"path":"environment.outside.temperature","value":303.03},{"path":"environment.pressure","value":101200}],"timestamp":"2019-06-23T23:42:23.581Z"}]}

They both look fine ... But ...

The data in b) is not presented when I use a browser to navigate to 
http://mplex.local:3000/signalk/v1/api/v...nvironment

It is also not available in any of the instruments.

In fact none of the data that I'm converting is presented. This is a step backward as I'd had the environment data available previously. But I just can't find the issue.

You'll note that I excluded the key "outside" from the pressure path. That was intentional as I was trying to discern whether deltas are handled differently with the subtly different paths.

The excerpt below is my input to the server.

{"context": "vessels.urn:mrn:imo:mmsi: 601270800 ", "updates": [ {  "source":{ "label": "XDR_Parser", "type": "NMEA0183", "talker": "II", "sentence": "XDR" },   "values": [ {"path": "environment.outside.temperature", "value": 303.56}]  } ] }
{"context": "vessels.urn:mrn:imo:mmsi: 601270800 ", "updates": [ {  "source":{ "label": "XDR_Parser", "type": "NMEA0183", "talker": "II", "sentence": "XDR" },   "values": [ {"path": "environment.outside.humidity", "value": 23.16}]  } ] }
{"context": "vessels.urn:mrn:imo:mmsi: 601270800 ", "updates": [ {  "source":{ "label": "XDR_Parser", "type": "NMEA0183", "talker": "II", "sentence": "XDR" },   "values": [ {"path": "environment.pressure", "value": 101200}]  } ] }
{"context": "vessels.urn:mrn:imo:mmsi: 601270800 ", "updates": [ {  "source":{ "label": "XDR_Parser", "type": "NMEA0183", "talker": "II", "sentence": "XDR" },   "values": [ {"path": "environment.outside.temperature", "value": 303.56},{"path": "environment.pressure", "value": 101200}]  } ] }

(2019-07-07, 08:20 PM)garysmith Wrote: [ -> ]
(2019-07-01, 01:24 PM)garysmith Wrote: [ -> ]
(2019-07-01, 09:08 AM)tkurki Wrote: [ -> ]I'd be surprised if putting the whole delta on a single line would not work given that I am one of the primary authors of the server and know how things work underneath the hood. If you know better feel free to ignore my advice.

The issue here is that a TCP stream is just a stream of bytes, there is no concept of a message in TCP. The receiver either needs to parse the input on the fly, balancing parentheses and breaking the input to delta messages, or we can choose a message delimiter. We have chosen newline as the delimiter, making the receiver much simpler, it just splits the input to messages by lines.

Once the input is split to individual messages newlines and whitespace no longer matter, each message is parsed as a JSON document. Documentation is formatted for easy reading and reflects this. Playground parses a single message as JSON. If you paste more than one message to Playground it will complain about invalid JSON - it is trying to handle just a single message, not the contents of a stream.

If you start the server in demo mode with sample data playback and netcat to port 8375 you will see deltas, one per line. This should be your reference - this is what the Signal K tcp input handles.
And after that authoritative answer I'd be surprised if that wasn't the solution :-). Thank you, much appreciated.

And just for everyone else ... It works. No newlines in a delta or full message.

@tkurki

help again please ....

I've attached a log file of the output from my server using netcat to connect to port 8375.

I've grepped it to reduce the volume of data. It only contains data pertaining to the environment path.

There are two sources of data - one is where the TCP server does the conversion from non XDR strings and the other is from my XDR converter

Here is an extract of one of each delta message:
a) The Signal K generated delta:
{"updates":[{"source":{"sentence":"MWV","talker":"II","type":"NMEA0183","label":"nmea_no_xdr"},"timestamp":"2019-06-23T23:42:23.579Z","values":[{"path":"environment.wind.speedApparent","value":11.62},{"path":"environment.wind.angleApparent","value":0.7674212722771316}]}],"context":"vessels.urn:mrn:imo:mmsi:601270800"}

b) The delta from my converter (after passing through the Signal K server
{"context":"vessels.urn:mrn:imo:mmsi: 601270800 ","updates":[{"source":{"label":"xdr_to_signalk","type":"NMEA0183","talker":"II","sentence":"XDR"},"values":[{"path":"environment.outside.temperature","value":303.03},{"path":"environment.pressure","value":101200}],"timestamp":"2019-06-23T23:42:23.581Z"}]}

They both look fine ... But ...

The data in b) is not presented when I use a browser to navigate to 
http://mplex.local:3000/signalk/v1/api/v...nvironment

It is also not available in any of the instruments.

In fact none of the data that I'm converting is presented. This is a step backward as I'd had the environment data available previously. But I just can't find the issue.

You'll note that I excluded the key "outside" from the pressure path. That was intentional as I was trying to discern whether deltas are handled differently with the subtly different paths.

The excerpt below is my input to the server.

{"context": "vessels.urn:mrn:imo:mmsi: 601270800 ", "updates": [ {  "source":{ "label": "XDR_Parser", "type": "NMEA0183", "talker": "II", "sentence": "XDR" },   "values": [ {"path": "environment.outside.temperature", "value": 303.56}]  } ] }
{"context": "vessels.urn:mrn:imo:mmsi: 601270800 ", "updates": [ {  "source":{ "label": "XDR_Parser", "type": "NMEA0183", "talker": "II", "sentence": "XDR" },   "values": [ {"path": "environment.outside.humidity", "value": 23.16}]  } ] }
{"context": "vessels.urn:mrn:imo:mmsi: 601270800 ", "updates": [ {  "source":{ "label": "XDR_Parser", "type": "NMEA0183", "talker": "II", "sentence": "XDR" },   "values": [ {"path": "environment.pressure", "value": 101200}]  } ] }
{"context": "vessels.urn:mrn:imo:mmsi: 601270800 ", "updates": [ {  "source":{ "label": "XDR_Parser", "type": "NMEA0183", "talker": "II", "sentence": "XDR" },   "values": [ {"path": "environment.outside.temperature", "value": 303.56},{"path": "environment.pressure", "value": 101200}]  } ] }

Although I've found the issue, I'm not going to delete the post in case it helps anyone elselater on.

The problems was the whitespace in the input delta message ... no newlines, no whitespace of any kind allowed ....
The edited message reads like you found the issue?


Sent from my iPhone using Tapatalk