Import CSV data

cd ~/dev/foss/contrib/

cat 2017-02-25_sds011_sensor_777.csv | http POST Content-Type:text/csv --timeout 500

HTTP/1.1 200 OK
Channel-Id: /mqttkit-1/testdrive/earth/42
Content-Type: application/json
Date: Tue, 28 Mar 2017 22:56:52 GMT
Server: TwistedWeb/17.1.0
Transfer-Encoding: chunked

        "message": "Received header fields ['sensor_id', 'sensor_type', 'location', 'lat', 'lon', 'time', 'P1', 'durP1', 'ratioP1', 'P2', 'durP2', 'ratioP2']",
        "type": "info"
        "message": "Received #22 readings",
        "type": "info"

We added commit 49256945 to make Kotori conveniently grok the CSV format used by

# Convenience hack to support import from
elif first_line.startswith('sensor_id'):
    header_line = first_line

Vendor configuration

Add vendor configuration to Kotori:

# Activate
root@elbanco:/etc/kotori/apps-enabled# ln -s ../apps-available/luftdaten.ini .

# Bounce daemon
systemctl restart kotori


Phase 1

Import CSV data (see above). [29.03.17 02:18:04] Andreas Motl: ready:

Phase 2

Feed data from live data API from to MQTT using

[29.03.17 16:44:09] Richard Pobering: ping. ich habe das dashboard jetzt inhaltlich nochmal ein bisschen angepasst:


  • Link/embed to dashboard by Sensor ID.

  • Check for proper timezone when importing CSV vs. fetching data from live API

  • Filter out fields like sensor_id, location, lat, lon from timeseries data

  • Convert lat/lon or latitude/longitude to geohash (The Grafana Worldmap Panel requires GeoHash)

  • Live data convergence by polling on, see also

  • Resolve geohash to regional/city name

  • Migrate new routines from ( into Kotori

  • Kotori: Re-create database when ERROR: Error processing MQTT message from topic “luftdaten/testdrive/earth/42/data.json”: [Failure instance: Traceback: <class ‘influxdb.exceptions.InfluxDBClientError’>: 404: {“error”:”database not found: “luftdaten_testdrive”“}

  • Worldmap Plugin:

    • Interpolate multiple metric values into popover
    • Add HTML links to popover
    • Embed HTML into popover for generic panel generation
  • Single announcement of non-volatile values to be persisted into MongoDB. Use case: Einmalig Standort registrieren.

  • Generic tag announcements through field name annotations. Proposals:

    1. Send CSV header like this: sensor_id[@tag];sensor_type[@tag];location[@tag];lat;lon;timestamp;P1;durP1;ratioP1;P2;durP2;ratioP2
    2. Send qualification information “out of band”: ## @tags:sensor_id,sensor_type,location

    => Think about how to do it with JSON?:

    Why not just "@tag:sensor_id", ...?
    {temperatute: 42.42, tags: {'standort': 'Niederrhein'}}
    {tags: {'standort': 'Niederrhein'}}
    {temperatute: 42.42}
  • Integrate web-gl globe:

  • Generische tag => filter umsetzung à la Grafana