luftdaten.info


About luftdaten.info

luftdaten.info is a community project from Stuttgart, Germany. Due to its geographical location, they have high particulate matter (PM) levels on a regular basis.

People around the OK Lab Stuttgart, a project supported by code for germany of the open knowledge foundation constructed cost-effective particulate sensors easily to rebuild by everyone. The sensor readings are acquired by the backend application feinstaub-api, displayed in luftdaten.info map and archived at luftdaten.info Archive.

Luftdaten.info is in the process of installing low cost, crowdfunded sensor nodes across many cities in Germany to measure PM levels.

About Luftdatenpumpe

Goal

The goal is to display data from luftdaten.info using a Grafana dashboard for data visualization. Historical data should be able to be displayed by geographical location to make it a meaningful supplement to the luftdaten.info map with more flexible access capabilities compared to the current RRDTOOL-based graphing. luftdaten.info already makes all historical data accessible on the luftdaten.info Archive (CSV format) and live data via luftdaten.info API (JSON format).

There are two different data paths:

Description

The main workhorse, luftdatenpumpe requests data from the live API of luftdaten.info each 10 minutes, enriches it with geographic location information (reverse geocoding) and republishes its results to the MQTT bus. Kotori picks it up from there, stores it into InfluxDB with appropriate tags as pivot points and adds a corresponding Grafana dashboard for data visualization.

As one of the authors of the feinstaub-api outlined here:

You can easily implement a “proxy” that translates json to mqtt. Then you can use this proxy as a custom api.
Historical data can be found here: https://archive.luftdaten.info/
Graphs can be found here: https://www.madavi.de/sensor/graph.php
This file is updated every minute and contains all sensors and values sent to the server in the last 5 minutes.

luftdatenpumpe is such a translator and can be used as an universal MQTT forwarder.

Screenshots and live data

Live data

Screenshots

luftdaten.info - Measurement timeseries by location, with EU-Limits as threshold lines

luftdaten.info - Measurement timeseries by location, with EU-Limits as threshold lines

luftdaten.info - Current measurement value, with EU-Limits (green)

luftdaten.info - Current measurement value, with EU-Limits (green)

luftdaten.info - Current measurement value, with EU-Limits (orange)

luftdaten.info - Current measurement value, with EU-Limits (orange)

luftdaten.info - Location chooser

luftdaten.info - Location chooser

luftdaten.info - Grafana Worldmap

luftdaten.info - Grafana Worldmap

Setup

This section is about running the whole platform on your own hardware. For full installation of the InfluxDB/Grafana/Mosquitto/Kotori stack, please have a look at the Setup page.

InfluxDB

Configure /etc/influxdb/influxdb.conf:

[[udp]]
  # High-traffic UDP
  enabled = true
  bind-address = ":4445" # the bind address
  database = "luftdaten_testdrive" # Name of the database that will be written to
  batch-size = 5000 # will flush if this many points get buffered
  batch-timeout = "30s" # will flush at least this often even if the batch-size is not reached
  batch-pending = 100 # number of batches that may be pending in memory
  read-buffer = 8388608 # (8*1024*1024) UDP read buffer size

Then:

systemctl restart influxdb

Kotori

Activate luftdaten.ini as configuration:

ln -sr /etc/kotori/examples/vendors/luftdaten.ini /etc/kotori/apps-available/
ln -sr /etc/kotori/apps-available/luftdaten.ini /etc/kotori/apps-enabled/
systemctl restart kotori

Luftdatenpumpe

luftdatenpumpe requests data from the live API of luftdaten.info, enriches it with geospatial information and republishes it to the MQTT bus.

Synopsis

/opt/kotori/bin/luftdatenpumpe --mqtt-uri mqtt://mqtt.example.org/luftdaten/testdrive/earth/42/data.json --geohash --reverse-geocode --progress
2017-04-22 03:55:50,426 [kotori.vendor.luftdaten.luftdatenpumpe] INFO   : Publishing data to MQTT URI mqtt://mqtt.example.org/luftdaten/testdrive/earth/42/data.json
2017-04-22 03:55:51,396 [kotori.vendor.luftdaten.luftdatenpumpe] INFO   : Timestamp of first record: 2017-04-22T01:50:02Z
100%|..........................................................................| 6782/6782 [01:01<00:00, 109.77it/s]

Run each 10 minutes

/etc/cron.d/luftdatenpumpe:

# /etc/cron.d/luftdatenpumpe -- forward data from luftdaten.info json api to mqtt

# run luftdatenpumpe each 10 minutes
*/10 * * * * root /opt/kotori/bin/luftdatenpumpe --mqtt-uri mqtt://mqtt.example.org/luftdaten/testdrive/earth/42/data.json --geohash --reverse-geocode --progress

Usage

This is an ad hoc example about how to republish measurement data.

Subscribe to the luftdaten.info feed on MQTT:

mosquitto_sub -h mqtt.example.org -t 'luftdaten/#' -v

Start feeding measurements:

luftdatenpumpe --mqtt-uri mqtt://mqtt.example.org/luftdaten/testdrive/earth/42/data.json --geohash --reverse-geocode --progress

Data payloads will be per-sensor in JSON format, enriched by geographic location information.

Example MQTT messages:

luftdaten/testdrive/earth/42/data.json {"sensor_id": 2115, "sensor_type": "SDS011", "P1": 11.17, "P2": 7.95,               "time": "2017-05-05T01:27:42Z", "location_id": 1064, "geohash": "u33dbz90yu6r", "location_name": "Gr\u00fcntaler Stra\u00dfe, Gesundbrunnen, Berlin, DE"}
luftdaten/testdrive/earth/42/data.json {"sensor_id": 2116, "sensor_type": "DHT22",  "temperature": 12.3, "humidity": 65.5, "time": "2017-05-05T01:27:42Z", "location_id": 1064, "geohash": "u33dbz90yu6r", "location_name": "Gr\u00fcntaler Stra\u00dfe, Gesundbrunnen, Berlin, DE"}

Credits

  • All the people of luftdaten.info.
  • Richard Pobering for thoughtful design decisions, hard work on the beautiful Grafana dashboards and for pushing things forward in general.

Agenda

Todo

Import historical data from the luftdaten.info Archive.

All acquisition infrastructure for reading the CSV format is in place already, but there is still some data munging required. To get results of similar richness as the live data import, some parts of the code should be refactored from the luftdatenpumpe program to the internal data acquisition routines to provide things like reverse geocoding also when importing raw CSV files.

Todo

There are more things on the todo list.