LST

About

LST is the Labor für Systemtechnik at the Munich University of Applied Sciences. They are into aviation and conceive a number of fine projects. Also see some pictures of the laboratory.

Environment

Let’s have a look at the environment:

  • mbed is a popular embedded computing platform used intensively inside the laboratory.
  • A common way of communicating telemetry data and control commands is UDP over Ethernet or WiFi.
  • Data payloads are usually in binary format defined in C/C++ header files.

System overview

// LST system overview
digraph lst {

    // Options
    rankdir=LR;
    ranksep=0.5;

    // Style
    //graph [splines=ortho];
    node [pin=true, shape="box", fontname="Verdana"];
    edge [fontname="Verdana"];


    // Graph nodes represent system components
    "mbed"          [label="mbed MCU"];
    "kotori"        [label="Kotori"];
    {rank=same; "influxdb"; "grafana" };
    "influxdb"      [label="InfluxDB"];
    "grafana"       [label="Grafana"];
    "browser"       [label="Browser"];
    "rickshaw"      [label="Rickshaw"];

    // Graph edges represent communication paths
    "mbed"          -> "kotori"         [label="UDP/Binary"];
    "kotori"        -> "influxdb";
    "kotori"        -> "grafana";
    "influxdb"      -> "grafana";
    "kotori"        -> "browser"        [label="WAMP"];
    "browser"       -> "rickshaw";

}
  • Receive telemetry messages over UDP in binary format.

  • Decode and enrich them by using information from structs of real C/C++ header files.

  • Store measurements to the database, with attribute names matching the header file struct declarations.

  • Automatically create default Grafana panels for instant telemetry data visualization.

  • While doing all this, it should get out of the way by honoring LST best practices:

    • Must handle and dispatch to multiple struct definitions per project. Each struct has a unique ID at the second byte position. When receiving binary payloads from UDP, the messages have to be dispatched appropriately. Example:

      # This is a struct mapped to message ID=1 (with length=9)
      struct struct_position
      {
          uint8_t  length             ;//1        // Length of struct (number of bytes)
          uint8_t  ID                 ;//2        // Struct ID
          // ...
      } position = {9,1};
      
    • Must handle bit fields, e.g.:

      uint8_t  send_ser        : 1;   //  3.0 FlagByte 1
      uint8_t  cfg_loaded      : 1;   //  3.1
      uint8_t  clamped         : 1;   //  3.2
      

Integration

For making a micro controller send UDP messages the LST way, please consider using the fine H2M Telemetry library for mbed by Sebastian Donner.

This library was conceived when working on the telemetry communication layer of the Hydro2Motion project.

Handbook

Channel configuration

Channel configuration is currently done by amending the Kotori configuration file, which is /home/basti/kotori/etc/lst.ini on host kotori-lst.

Overview

A channel is made of ...

  • an UDP port for receiving messages
  • a WAMP topic for publishing received messages to the software bus
  • a filesystem path to the directory of C/C++ header files
  • the names of the header files containing struct declarations describing the messages received on a channel, see Anatomy of a header file

Example:

[lst]
channels     = lst-h2m, lst-sattracker

[lst-h2m]
udp_port     = 8888
wamp_topic   = edu.hm.lst.h2m
include_path = etc/headers
header_files = h2m_structs.h

[lst-sattracker]
udp_port     = 8889
wamp_topic   = edu.hm.lst.sattracker
include_path = etc/headers
header_files = sattracker.h

Add new channel

  • Choose a project/channel name, here “foo”
  • Choose an UDP port, here “8890”
  • Choose a wamp topic, here “edu.hm.lst.foo”
  • Put your C++ header file into the etc/headers directory, here “foo.h”
  • Activate channel configuration by adding configuration section name to the channel attribute of the toplevel [lst] section, here “lst-foo”. This is a comma-separated list.

Example:

[lst]
channels     = lst-h2m, lst-sattracker, lst-foo

# [...]

[lst-foo]
udp_port     = 8890
wamp_topic   = edu.hm.lst.foo
include_path = etc/headers
header_files = foo.h

Anatomy of a header file

In order to make Kotori grok the message structs defined in standard C/C++ header files, there are some specific details to be followed. The initializer flavor to be used currently is the “list-initializer” style as seen below. Further, transformation rules can be attached to specific fields by adding annotations in comments. See also example below and Transformation rules.

Note

We are working on the brace-or-equal-initializer style.

Example:

#include "mbed.h"

struct struct_system
{
    uint8_t length              ;//1
    uint8_t ID                  ;//2
    uint8_t output      : 1     ;//3.0
    uint8_t use_gps     : 1     ;//3.1
    uint8_t flagbyte_2          ;//4
    double  lat_home            ;//12
    double  long_home           ;//20
    int8_t  sync                ;//4
    uint8_t ck                  ;//21
} sys = {21,0};

struct struct_position
{
    uint8_t  length             ;//1
    uint8_t  ID                 ;//2
    uint8_t  flagbyte_1         ;//3
    uint8_t  flagbyte_2         ;//4
    int16_t  hdg                ;//6     // @rule: name=heading; expr=hdg * 20.1; unit=degrees
    int16_t  pitch              ;//8     // @rule: name=pitch; expr=pitch * 10; unit=degrees
    uint8_t  ck                 ;//9
} position = {9,1};

Please consider adding the header file to the git repository by issuing:

git add etc/headers/your_header.h
git commit etc/headers/your_header.h
git push

Transformation rules

The purpose of transformation rules is to apply scaling factors to values and to rename fields. This is achieved by adding a declarative annotation on the same line of the struct field definition. Example:

int16_t  hdg                ;//6     // @rule: name=heading; expr=hdg * 20.1; unit=degrees
This will:
  • rename the field from “hdg” to “heading”
  • perform scaling by applying the factor 20.1 to the raw value
  • configure value unit “degrees” in Grafana (TODO)

The evaluation of expressions is based on the fine SymPy library and can be made of a number of mathematical expressions, see also SymPy Features.

There’s a commandline tool for applying transformation rules to payloads, see Transform message.

Operating

A little bit of linux howto

WinSCP connect basti@raider.isarengineering.de

create a subfolder named after the channel in the folder /home/basti/kotori/etc/headers where the components.h needs to be saved

in the file /home/basti/kotori/etc/lst.ini the Channel needs to be created and the components.h needs to be linked

using the User Basti the command “tmux att -t kotori” attaches to the console where kotori is running, press Strg-C to stop kotori, “kotori –debug” restarts kotori

never close tmux sessions, just detache, them using Strg-B and then d.

Tools

For using the channel operations, the user basti should navigate to /home/basti/kotori and then start “source .venv27/bin/activate”. From this environment, the commandline-based channel operations should be available.

The lst-message command

There is some tooling for conveniently working with native binary message structs defined in etc/headers/*.h.

The general syntax is:

lst-message <channel> <action> <argument> --config etc/lst.ini
  1. <channel> is one of h2m or sattracker, see also “Show list of channels”:

    • h2m: Use structs declared in etc/headers/h2m_structs.h
    • sattracker: Use structs declared in etc/headers/sattracker.h
  2. <action> is one of decode, transform, send or info:

    • decode: Decode binary payload given as hexadecimal string, dispatch to appropriate message matching the ID field and pretty print the contents
    • transform: Same as decode, but also applies transformation rules, see Transformation rules
    • send: Send binary payload given as hexadecimal string to specified target
    • info: Displays schema information and metadata for given struct name

See also

$ lst-message --help
    Usage:
      lst-message list-channels                    [--config etc/kotori.ini]
      lst-message <channel>  decode     <payload>  [--config etc/kotori.ini] [--debug]
      lst-message <channel>  transform  <payload>  [--config etc/kotori.ini] [--debug]
      lst-message <channel>  send       <payload>  --target=udp://localhost:8888 [--config etc/kotori.ini]
      lst-message <channel>  info       <name>     [--config etc/kotori.ini] [--debug]
      lst-message --version
      lst-message (-h | --help)

    Options:
      --config etc/kotori.ini   Use specified configuration file, otherwise try KOTORI_CONFIG environment variable
      --version                 Show version information
      --debug                   Enable debug messages
      -h --help                 Show this screen

Note

The parameter --config etc/lst.ini can be omitted by defining the KOTORI_CONFIG environment variable:

export KOTORI_CONFIG=`pwd`/etc/lst.ini

Show list of channels

$ lst-message list-channels

Channel names:
lst-h2m
lst-sattracker

So, the canonical channel names are “h2m” and “sattracker”. They can be configured in Kotori configuration file, e.g. etc/lst.ini. Use them as the <channel> parameter to lst-message in the following commands.

Decode message

Decode a short message in hex format, display struct name and decoded content:

$ lst-message h2m decode 0x05022a0021
----
name         struct_cap_r
----
hex          0x05022a0021
decimal      [5, 2, 42, 0, 33]
bytes        '\x05\x02*\x00!'
----
length       5
ID           2
voltage_act  42
ck           33

Transform message

Same as decode, but also applies transformation rules, see Transformation rules.

Example:

$ lst-message sattracker transform 0x090100000100000000
----
name        struct_position
----
hex         0x090100000100000000
decimal     [9, 1, 0, 0, 1, 0, 0, 0, 0]
bytes       '\t\x01\x00\x00\x01\x00\x00\x00\x00'
----
length      9
ID          1
flagbyte_1  0
flagbyte_2  0
heading     42.42
pitch       0
ck          0

The field heading was augmented by the transformation rule:

// @rule: name=heading; expr=hdg * 42.42; unit=degrees

Send message

Send a message in hex format to UDP server:

$ lst-message h2m send 0x05022a0021 --target=udp://localhost:8888
Message "0x05022a0021" sent to "udp://localhost:8888"

Use this tool to generate and send binary messages without having the hardware in place.

Display message schema

Display struct metadata information:

$ lst-message h2m info struct_fuelcell_r

------------------------------------------------------------------------------------
                             struct "struct_fuelcell_r"
------------------------------------------------------------------------------------

Header information

name         type        default    bitfield
-----------  --------  ---------  ----------
length       uint8_t          11
ID           uint8_t           4
current_act  uint16_t
current_req  uint16_t
voltage_act  uint16_t
voltage_req  uint16_t
ck           uint8_t


Library information

name         type      symbol    field                                   bitfield
-----------  --------  --------  ------------------------------------  ----------
length       c_ubyte   B         <Field type=c_ubyte, ofs=0, size=1>
ID           c_ubyte   B         <Field type=c_ubyte, ofs=1, size=1>
current_act  c_ushort  H         <Field type=c_ushort, ofs=2, size=2>
current_req  c_ushort  H         <Field type=c_ushort, ofs=4, size=2>
voltage_act  c_ushort  H         <Field type=c_ushort, ofs=6, size=2>
voltage_req  c_ushort  H         <Field type=c_ushort, ofs=8, size=2>
ck           c_ubyte   B         <Field type=c_ubyte, ofs=10, size=1>


Representations

kind     representation
-------  ----------------------------------------------
hex      0x0b04000000000000000000
decimal  [11, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0]
bytes    '\x0b\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00'

Todo

$ lst-message h2m encode
$ lst-message h2m receive

Query InfluxDB

List databases:

$ curl --silent --get 'http://192.168.59.103:8086/query?pretty=true' --user admin:Armoojwi --data-urlencode 'q=SHOW DATABASES' | jq '.'

{
  "results": [
    {
      "series": [
        {
          "name": "databases",
          "columns": [
            "name"
          ],
          "values": [
            [
              "_internal"
            ],
            [
              "edu_hm_lst_h2m"
            ]
          ]
        }
      ]
    }
  ]
}

Query timeseries:

$ export INFLUX_URI=http://192.168.59.103:8086/query?pretty=true
$ curl --silent --get $INFLUX_URI --user admin:Armoojwi --data-urlencode 'db=edu_hm_lst_h2m' --data-urlencode 'q=select * from "02_cap_r";' | jq '.'

{
  "results": [
    {
      "series": [
        {
          "name": "02_cap_r",
          "columns": [
            "time",
            "_hex_",
            "voltage_act"
          ],
          "values": [
            [
              "2015-11-10T13:44:43.945864544Z",
              "05022a0021",
              42
            ],
            [
              "2015-11-10T13:46:35.678928928Z",
              "05022a0021",
              42
            ],
            [
              "2015-11-10T14:48:33.475860964Z",
              "05022a0021",
              42
            ]
          ]
        }
      ]
    }
  ]
}