This forum uses cookies
This forum makes use of cookies to store your login information if you are registered, and your last visit if you are not. Cookies are small text documents stored on your computer; the cookies set by this forum can only be used on this website and pose no security risk. Cookies on this forum also track the specific topics you have read and when you last read them. Please confirm whether you accept or reject these cookies being set.

A cookie will be stored in your browser regardless of choice to prevent you being asked this question again. You will be able to change your cookie settings at any time using the link in the footer.

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
how to manipulate incoming data stream?
#1
I have some old seatalk instrument, and have the data coming in over GPIO. However I find that the seatalk network is very noisy and I frequently get momentary zero values for wind or depth metrics.

As such it throws off the autopilot to the extent that is can be quite dangerous, resulting in unplanned tacks or gybes when sailing with autopilot in wind tracking mode.

Now while there is likely to be a physical problem, or several, I would like to try ignoring the obvious bad data and use last value when a zero comes through.

I think this must be fairly trivial but not sure where to start.

I'm wondering if Node Red is the answer, but will need to guarantee that the outgoing canbus data stream is using the cleaned data stream.

Do I need to write a plugin like the derived data plugin?
Reply
#2
Well here's what I've come up with so far in node red, but I still get alarms for 0 depth on the NMEA 2000 chart plotter, although it's no longer showing up in grafana.

I will try updating the nmea 2000 output directly.
   


Code:
[{"id":"38336563.2cc8e2","type":"tab","label":"Flow 2","disabled":false,"info":""},{"id":"71b3bf61.5dd4f8","type":"inject","z":"38336563.2cc8e2","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":"1","topic":"environment.wind.angleApparent","payload":"3.14","payloadType":"num","x":90,"y":140,"wires":[["8c8dd0de.697ad"]]},{"id":"3b7621c1.ff17e6","type":"inject","z":"38336563.2cc8e2","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":"1","topic":"environment.wind.angleApparent","payload":"NaN","payloadType":"str","x":90,"y":200,"wires":[["8c8dd0de.697ad"]]},{"id":"8c8dd0de.697ad","type":"function","z":"38336563.2cc8e2","name":"filter zero and NaN AWA","func":"var payload = msg.payload;\n//node.warn(\"msg.payload = \" + msg.payload);\nlastAwa=context.get(\"lastAwa\"); \n//node.warn(\"lastAwa = \" + context.get(\"lastAwa\"));\n\nif (payload == \"0\"||payload == \"NaN\"||payload == \"3.14\")\n{\nnode.warn(\"BAD WIND DATA \" + payload);\nmsg.payload = lastAwa\nnode.warn(\"lastAwa = \" + context.get(\"lastAwa\"));\nreturn msg;\n} else \n{\n//node.warn(\"GOOD WIND DATA\" + payload);\ncontext.set(\"lastAwa\",payload); \n//node.warn(\"lastAwa = \" + context.get(\"lastAwa\"));\nreturn msg;\n}\n","outputs":1,"noerr":0,"initialize":"","finalize":"","x":470,"y":160,"wires":[["e18508a1.f09518"]]},{"id":"966391c5.dede38","type":"signalk-input-handler","z":"38336563.2cc8e2","name":"AWA input handler","context":"vessels.self","path":"environment.wind.angleApparent","source":"","x":140,"y":280,"wires":[["8c8dd0de.697ad"]]},{"id":"e18508a1.f09518","type":"signalk-input-handler-next","z":"38336563.2cc8e2","name":"back to server","x":880,"y":240,"wires":[]},{"id":"cb274b3f.f47ee8","type":"signalk-input-handler","z":"38336563.2cc8e2","name":"Depth input handler","context":"vessels.self","path":"environment.depth.belowTransducer","source":"","x":130,"y":340,"wires":[["2405a162.722516"]]},{"id":"2405a162.722516","type":"function","z":"38336563.2cc8e2","name":"filter zero and above 150 depth","func":"var payload = msg.payload;\n//node.warn(\"msg.payload = \" + msg.payload);\nlastDepth=context.get(\"lastDepth\"); //to retrieve a variable  \n//node.warn(\"lastDepth = \" + context.get(\"lastDepth\"));\n\nif (payload == \"0\"||payload >= 150)\n{\nnode.warn(\"BAD DEPTH DATA  \" + payload);\nnode.warn(\"lastDepth = \" + context.get(\"lastDepth\"));\nmsg.payload = lastDepth\nreturn msg;\n} else \n{\n//node.warn(\"GOOD DEPTH DATA  \" + payload);\ncontext.set(\"lastDepth\",payload); // to store a variable\nreturn msg;\n}\n","outputs":1,"noerr":0,"initialize":"","finalize":"","x":470,"y":320,"wires":[["e18508a1.f09518"]]},{"id":"38e1dfe7.ae87c","type":"inject","z":"38336563.2cc8e2","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":"1","topic":"environment.depth.belowTransducer","payload":"160","payloadType":"num","x":190,"y":440,"wires":[["2405a162.722516"]]},{"id":"dabf54e0.0a65f8","type":"inject","z":"38336563.2cc8e2","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":"1","topic":"environment.depth.belowTransducer","payload":"0","payloadType":"num","x":190,"y":480,"wires":[["2405a162.722516"]]}]
Reply
#3
Well after many iterations and some very exciting live coding in production while trying to prevent the boat from randomly gybing, this is my damping algorithm which will discard any bad data that frequently comes through from old instruments and uses the average value from a set, with a configurable size.

The damping set and the tolerance for difference from the last average are both configurable. Hopefully this helps someone else!


Code:
[
   {
       "id": "48e91ee0.6e36b",
       "type": "tab",
       "label": "Clean AWA",
       "disabled": false,
       "info": ""
   },
   {
       "id": "cf421cb3.e95bc",
       "type": "inject",
       "z": "48e91ee0.6e36b",
       "name": "",
       "props": [
           {
               "p": "payload"
           },
           {
               "p": "topic",
               "vt": "str"
           }
       ],
       "repeat": "",
       "crontab": "",
       "once": false,
       "onceDelay": "1",
       "topic": "environment.wind.angleApparent",
       "payload": "-3.14",
       "payloadType": "num",
       "x": 170,
       "y": 80,
       "wires": [
           [
               "4b11a19a.2aae8"
           ]
       ]
   },
   {
       "id": "4b11a19a.2aae8",
       "type": "function",
       "z": "48e91ee0.6e36b",
       "name": "filter AWA more than 30% different",
       "func": "let threshold = 17;\nlet maxValue = 3.15;\n\nlet minValue = -3.15;\nlet dampingSet = 5;\nlet modifyMsg = flow.get(\"modifyMsg\");\n\nreturn modifyMsg(msg,threshold,maxValue,minValue,dampingSet);\n",
       "outputs": 2,
       "noerr": 0,
       "initialize": "let modifyMsg = function modifyMsg(message = msg,\n                                   thresholdVal = threshold,\n                                   max = maxValue,\n                                   min = minValue,\n                                   dampingSet = 5) {\n\n    let currentValue = message.payload;\n    let lastKnownGoodValue = flow.get(\"lastKnownGoodValue\");\n    let prevAvgValue = flow.get(\"avgValue\");\n\n    // dampingSet; //number of elements to use for average value\n    // node.warn(\"damping array\"+JSON.stringify(flow.get(\"dampingAry\")));\n    if (flow.get(\"dampingAry\") == null) {\n        node.warn(\"initialising damping array\");\n        let damping = Array.from({length: dampingSet}, () => ({value: 0, update: 0, init: 0,}))\n        damping[0].update = 1\n        let dampingElement = 0;\n        flow.set(\"dampingAry\", damping);\n    }\n\n    let differenceInRads = (Math.abs(lastKnownGoodValue) - Math.abs(currentValue));\n    let percent = ((differenceInRads) / (2 * Math.PI)) * 100;\n\n    if (currentValue != null && !isNaN(currentValue) && currentValue != 0) {\n        //if value is not blank then proceed, otherwise use previous value\n        if ((Math.abs(currentValue) > max) || (Math.abs(currentValue) < min)) {\n            //if value is a crazy number, then use last average value\n            node.warn(JSON.stringify(currentValue* 180/Math.PI) + \" impossible value: \" + JSON.stringify(currentValue* 180/Math.PI));\n            message.payload = prevAvgValue\n            // node.warn(\"corrected message value: \" + JSON.stringify(message.payload));\n            return message;\n        } else if ((percent > thresholdVal)) {\n            //if value is too different from the average in the damping array, then use last known good value\n            // message.payload = lastKnownGoodValue\n            message.payload = prevAvgValue\n            // node.warn(JSON.stringify(percent) + \" % diff \" + differenceInRads + \" in rads  \" +\n            //     \"between \" + JSON.stringify(lastKnownGoodValue) + \" and \" + JSON.stringify(currentValue)\n            // + \" too high using previous average value: \" + JSON.stringify(prevAvgValue));\n            // // node.warn(differenceInDeg+\"  in degs between \" + JSON.stringify(lastKnownGoodValue))\n            node.warn(\"corrected \" + JSON.stringify(currentValue* 180/Math.PI)\n                + \" to: \"\n                + JSON.stringify(message.payload* 180/Math.PI)\n                + \" flow.get(\\\"avgValue\\\"): \"\n                + JSON.stringify(flow.get(\"avgValue\")* 180/Math.PI)\n                + \" lastKnownGoodValue: \"\n                + JSON.stringify(flow.get(\"lastKnownGoodValue\")* 180/Math.PI)\n                // + \" dampingAry: \"\n                // + JSON.stringify(flow.get(\"dampingAry\"))\n            );\n            return message;\n        } else {\n            //if value seems ok then update last known good value, damping array and average value\n            flow.set(\"lastKnownGoodValue\", currentValue);\n            updateDampingArray(currentValue);\n        }\n        //value seems ok so pass it on unmodified\n        // node.warn(\"uncorrected message value: \" + JSON.stringify(message.payload));\n        return message;\n    } else {\n        flow.set(\"avgValue\", prevAvgValue);\n        // message.payload = lastKnownGoodValue\n        message.payload = prevAvgValue\n        // node.warn(\"currentValue null, using last value: \" + JSON.stringify(lastKnownGoodValue* 180/Math.PI));\n        node.warn(\"currentValue null, using last avg: \" + JSON.stringify(prevAvgValue* 180/Math.PI));\n        node.warn(\"average set to: \" + JSON.stringify(prevAvgValue* 180/Math.PI));\n    }\n}\n\nfunction updateDampingArray(currentValue) {\n    //todo convert negatives to 180-360 for averaging and back again\n    let ary = flow.get(\"dampingAry\");\n    ary.forEach((element, index, array) => {\n        if (element.update == 1) {\n            dampingElement = index;\n            element.update = 0\n        }\n    });\n    if (currentValue<0){\n        ary[dampingElement].value = currentValue +(2*Math.PI);\n    } else {\n        ary[dampingElement].value = currentValue;\n    }\n\n    ary[dampingElement].update = 0;\n    if (ary[dampingElement + 1] != null) {\n        ary[dampingElement + 1].update = 1;\n    } else {\n        ary[0].update = 1;\n    }\n\n    let avgVal = ary.reduce(function (sum, element) {\n        return sum + parseFloat(element.value);\n    }, 0) / ary.length;\n\n    if (avgVal != null && !isNaN(avgVal) && avgVal != 0) {\n        //defensively update average\n        if (avgVal>Math.PI) {\n            flow.set(\"avgValue\", avgVal-(2*Math.PI));\n        } else {\n            flow.set(\"avgValue\", avgVal);\n        }\n\n        // if (avgVal != prevAvgValue) {\n        //     node.warn(\"update average from \" + prevAvgValue + \" to:  \" + JSON.stringify(avgVal));\n        // }\n    }\n\n    flow.set(\"dampingAry\", ary);\n}\n\nflow.set(\"modifyMsg\", modifyMsg);\n",
       "finalize": "",
       "libs": [],
       "x": 460,
       "y": 140,
       "wires": [
           [
               "670bd0ea.8ea67"
           ],
           []
       ]
   },
   {
       "id": "58e0a1ba.0ddcb",
       "type": "signalk-input-handler",
       "z": "48e91ee0.6e36b",
       "name": "AWA input handler",
       "context": "vessels.self",
       "path": "environment.wind.angleApparent",
       "source": "",
       "x": 130,
       "y": 140,
       "wires": [
           [
               "4b11a19a.2aae8"
           ]
       ]
   },
   {
       "id": "670bd0ea.8ea67",
       "type": "signalk-input-handler-next",
       "z": "48e91ee0.6e36b",
       "name": "back to server",
       "x": 760,
       "y": 140,
       "wires": []
   },
   {
       "id": "deeb0c7c.106ce",
       "type": "inject",
       "z": "48e91ee0.6e36b",
       "name": "",
       "props": [
           {
               "p": "payload"
           },
           {
               "p": "topic",
               "vt": "str"
           }
       ],
       "repeat": "",
       "crontab": "",
       "once": false,
       "onceDelay": "1",
       "topic": "environment.wind.angleApparent",
       "payload": "2.94",
       "payloadType": "num",
       "x": 170,
       "y": 40,
       "wires": [
           [
               "4b11a19a.2aae8"
           ]
       ]
   }
]
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)