Changelog

in progress

Breaking changes

  1. The updated Tasmota decoder introduces a breaking change as the database field names emitted by the decoder will change.

    For example, AirQuality will now be SonoffSC.AirQuality, Temperature might be called DS18B20.Temperature and so on.

    We have been too lazy in the past. Sorry.

Features

  • Honor “Time” field submitted by devices running Tasmota

  • Make Tasmota decoder grok double-nested payloads from Wemos devices

  • Add decoder for message payloads from Airrohr devices

  • Add pool_size option to [influxdb] configuration section. Thanks, @adrianlzt!

Bugfixes

  • Fix logger

  • Refactor decoder subsystem

  • Add “lan” strategy and “basic” application This implements a flat MQTT topic namespace for simplifying addressing and getting started documentation.

  • Upgrade to influxdb==5.3.0 module and fix tests

Documentation

Packaging

  • Improve Docker image building

  • Add packaging for Ubuntu 18 Bionic Beaver

Tests

  • Add MQTT/InfluxDB integration tests

  • Add MQTT/Grafana integration tests

  • Add tests covering all timestamp formats/units

  • Add tests covering the Tasmota decoder

  • Add test for vendor “hiveeyes”

  • Add MQTT/HTTP integration tests for HTTP data acquisition

  • Add tests for HTTP API export features

  • Add tests for InfluxDB tag heuristics

  • Add test for publishing discrete values to MQTT

  • Add tests for publishing events using MQTT and HTTP

  • Add test covering telemetry from Homie

  • Add test for WeeWX integration

  • Add test for legacy MQTT topic suffix

Clients

  • Add basic data export example for Arduino/HTTP

2020-03-03 0.24.5

  • Adjust packaging again

2020-03-03 0.24.4

  • Adjust packaging again

2020-03-03 0.24.3

  • Adjust packaging again

2020-03-03 0.24.2

2020-03-03 0.24.1

  • Adjust packaging. Thanks, @RuiPinto96 and @Dewieinns!

2020-01-15 0.24.0

  • Adjust packaging for Debian stretch vs. buster

  • InfluxDB adapter: Interpret “latitude/longitude” fields as tags

2020-01-10 0.23.0

  • Fix Terkin telemetry client for PHP5 to PHP7. Thanks, Markus and Christian!

  • Fix sandbox installation

  • Add basic support for ingesting SENSOR telemetry data from devices running the Sonoff-Tasmota firmware. Thanks, Roh!

  • Add support for ingesting STATE telemetry data from Sonoff-Tasmota devices.

  • Add documentation for the Sonoff-Tasmota Decoder

  • Improve error signalling within “boot_vendors”

2019-02-05 0.22.7

  • Slightly refactor ingress timestamp processing

  • Disable timestamp precision hack for WeeWX

  • Fix problem with time precision detection heuristics for ingress timestamps. Thanks, Richard!

2019-02-04 0.22.6

  • Fix Metastore database connection woes

2019-02-04 0.22.5

  • Attempt to fix dependency woes with the “idna” package

2019-02-04 0.22.4

  • Fix HTTP handler after upgrade to Twisted 18.9.0

  • Add missing “service_identity” module Twisted complained about and rebalance dependencies

2019-02-04 0.22.3

  • Fix Debian package dependencies

2019-02-04 0.22.2

  • Improve documentation

2019-02-03 0.22.1

  • Fix wrong Trove classifiers in setup.py blocking the upload to PyPI

2019-02-03 0.22.0

  • [io] Add “datetime” field to list of valid timestamp fields on data ingress

  • [luftdaten.info] Move Luftdatenpumpe to https://github.com/hiveeyes/luftdatenpumpe

  • Delete NaN values from ingress payload. Thanks, wtf!

  • Implement UNIX Epoch time precision detection heuristics for ingress timestamps. Thanks, wtf!

  • [package] Use Docker for Debian package building

2018-04-09 0.21.1

  • [io] Improve unicode support

  • [io] Warn if topic could not be classified

  • [io] Add “measure” topic suffix to topic classifier to retain backwards compatibility

  • [grafana] Properly quote measurement names for annotations in JSON templates

  • [Hiveeyes] Add “hiveeyes” tag to all instant Grafana dashboards

  • [Hiveeyes] Improve per-node Grafana dashboard re. field name classification

2018-04-09 0.21.0

  • [core] Improve component loading robustness

  • [core] Improve log level selection when (not) running with --debug-mqtt-driver

  • [code] Module namespace refactoring

  • [io] Make MQTT client identifier more unique by adding process id

  • [io] Improve resiliency when connecting to InfluxDB and Grafana

  • [io] Fix MQTT error signalling

  • [import] Also accept the CSV field “Date” as field containing timestamp information

  • [command] Add kotori-selftest program, still in its infancy

  • [firmware builder] Add “esp_root” configuration variable to example configuration file

  • [package] Improve list of suggested packages to ease installing on more modern Debian distributions

  • [env] Switch development environment from Docker to native

  • [grafana]

    • Improve Grafana dashboard builder in the context of Hiveeyes

    • Use specific Grafana folder for stuffing instant dashboards into

    • Use topology information for deriving unique dashboard id from

    • Add basic mechanisms for taming the dashboard refresh interval

    • Refactoring. Robustness. Run GrafanaManager as service. Add worker service for taming dashboard refresh intervals.

    • Relocate template resources (.json files)

    • Subsystem refactoring

    • Implement sensible rules for dashboard refresh interval taming

    • Properly talk to Grafana 5: Fix compatibility with dashboards having schemaVersion 6 (old) vs. 16 (new)

    • Properly compute dashboard identity. Don’t use the uid field for stable addressing as it is limited to 40 characters.

    • Allow unicode strings for dashboard names

    • Allow multiple graphing subsystems/managers for a single data acquisition channel driver

  • [luftdaten.info] Improve luftdatenpumpe.py

    • Add option --dry-run

    • Fix filtering by station id

    • Fix access to Nominatim reverse geocoder API

    • Use “appdirs” module for computing cache location

    • Report about cache location at startup

  • [Hiveeyes] Add per-node Grafana dashboard for beekeepers. Thanks, weef!

  • [doc]

    • Add comments to MqttInfluxGrafanaService regarding ingress data enrichment

    • Use ptrace.getkotori.org for documentation media assets

    • Correct GitHub repository url

    • Add documentation about Nginx reverse proxy setup

    • Add installation instructions for Mac OS X

2017-05-05 0.20.1

  • [export] Fix raw data export

  • [export] Properly apply charset encoding for data export

  • [doc] Update documentation for vendors luftdaten.info and WeeWX

2017-05-04 0.20.0

  • Re-add export/influx.py lost during refactoring

  • Fix InfluxDB UDP data acquisition

  • Tap into error signalling over MQTT

  • Minor updates to vendor luftdaten.info docs & co.

2017-05-03 0.19.1

  • Fix regression re. CSV data acquisition

2017-04-25 0.19.0

  • Vendor luftdaten.info:

    • Update Grafana dashboard “grafana-by-location.json”. Thanks, Richard!

    • Update documentation

  • Fix data convergence re. “time” field

2017-04-24 0.18.2

  • Attempt to improve InfluxDB write performance by using UDP protocol for vendor luftdaten.info

2017-04-24 0.18.1

  • Fix variable naming, references, data conversion and exception handling

  • Attempt to improve InfluxDB write performance

  • luftdatenpumpe: Configure Nominatim cache directory conditionally

2017-04-24 0.18.0

  • Improve InfluxDB database creation behaviour

  • luftdaten.info:

    • Add geospatial data acquisition capabilities

    • Improve luftdatenpumpe.py:

      • Generic commandline interface

      • Caching for Nominatim responses

      • Appropriate timestamp mungling

      • Documentation

    • Add LuftdatenGrafanaManager for provisioning appropriate Grafana dashboards

2017-04-21 0.17.0

  • Firmware builder improvements:

    • Don’t pull all build parameters into the artefact filename as this might raise OSError: [Errno 63] File name too long.

    • Automatically derive “MQTT_TOPIC” from telemetry channel address information

    • Update firmware builder configuration for vendor Hiveeyes

2017-04-19 0.16.0

  • Add standalone program “luftdatenpumpe.py” to request data from live data API of luftdaten.info, enrich geospatial information and publish to MQTT bus. Thanks, Richard!

  • Add documentation and configuration for vendor luftdaten.info. Thanks again, Richard!

  • Fix HTTP routing for Firmware builder

  • Fix python module dependency woes re. OpenSSL on Darwin vs. Linux

  • Make Kotori run even without having “pandas” installed (w/o “export” extras)

  • Improve error logging in bootstrapping phase

  • Add support for architecture “ESP” to Firmware builder

  • Make transaction logging interval configurable, default to 60 seconds to reduce log traffic significantly

  • Add vendor WeeWX: Tune MQTT data acquisition to support the weeWX weather station software. Thanks, Jan!

2017-03-29 0.15.0

2017-03-22 0.14.3

  • Properly use MQTT authentication also when publishing MQTT messages received via HTTP

2017-03-22 0.14.2

  • Fix setup.py again

2017-03-22 0.14.1

  • Fix setup.py

2017-03-21 0.14.0

  • Connect to the MQTT broker with authentication (default: kotori/kotori) to be able to apply Mosquitto ACLs to distinct MQTT topics. Thanks, Alex!

2017-03-20 0.13.0

  • Concurrency improvements: Don’t use the main reactor thread for MQTT message processing, use a different thread pool instead.

2017-03-20 0.12.3

  • Ubuntu 16.04 compatibility: Nail some more Python dependency modules. Thanks, Alex!

2017-03-20 0.12.2

  • Work on Ubuntu 16.04 compatibility: Nail some more Python dependency modules

2017-03-20 0.12.1

  • Add Python dependency “python-dateutil”

2017-03-16 0.12.0

2017-03-16 0.11.5

  • Prepare CSV import of http://archive.luftdaten.info/. Thanks, Richard!

    • Allow semicolon (;) as separator for CSV import

    • Allow CSV field timestamp as alias for datetime

  • Publish Grafana v4.2.0-beta1 debian packages to package repository, see Add foundation packages and Setup on Debian and Ubuntu.

  • Fix SSL runtime dependency woes AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1' with Twisted-17.1.0 vs. python-openssl vs. pyOpenSSL. Thanks, Matthias!

2017-02-12 0.11.4

  • Don’t always send http responses as text/plain

  • Catch MongoDB database errors, log and respond appropriately

2017-02-12 0.11.3

  • Upgrade some Python modules

2017-02-12 0.11.2

  • Fix Grafana dashboard builder for vendor Hiveeyes

2017-02-01 0.11.1

  • Data export: Fix addressing with relative timestamp, e.g. /data.txt?from=now-30d as well as proper handling of include and exclude url parameters

2017-01-31 0.11.0

2016-10-31 0.10.10

  • Fix Debian package re. superfluous “local” folder containing a redundant Python virtualenv. Thanks Smilie!

  • Fix receiving discrete measurements via MQTT. Thanks Karsten and Clemens!

  • Update Git repository url for hacking on Kotori

  • Improve documentation

2016-07-12 0.10.9

  • Documentation updates, add system diagrams to vendor Hiveeyes

  • Export csv and json data with ISO format timestamps to satisfy dygraphs rendering in Firefox

  • Don’t add “pad=true” or “backfill=true” when “interpolate=true” parameter was obtained from URL

  • Improve robustness of http api parameter evaluation and passing

  • Improve Vega asset loading: Use https resources, better safe than sorry

  • Add export format “.tsv” (text/tab-separated-values)

2016-07-10 0.10.7

  • Update documentation

  • Rebuild without having “ggplot” installed on the build host

2016-07-10 0.10.6

  • Fix timeseries plotting by using “pad” and “backfill” appropriately

  • Add export parameters “exclude”, “include”, “interpolate” and “sorted”

  • Fix data routing and processing

  • Add license to documentation

  • Quick hack for making Firmware builder endpoint not convert numeric values to floats

  • Packaging fixes

2016-07-05 0.10.5

  • Attempt to fix huge dependency chain when installing with –install-recommends –install-suggests

2016-07-05 0.10.4

  • Fix missing runtime dependency “simplejson” (required by cornice)

2016-07-02 0.10.3

  • Use matplotlib “agg” backend

  • Improve ggplot rendering context, add font for rendering xkcd theme

  • Upgrade to pandas 0.18.1

2016-07-02 0.10.2

  • Honor https scheme in reverse proxy setups

  • Packaging: Remove Python dependency on crossbar, can be installed through pip install crossbar==0.13.0

  • Packaging: Depend on more distribution packages to reduce package size

2016-07-01 0.10.1

Packaging

  • Fix Debian runtime dependencies

Data export

  • Always emit lowercase values from WanBusStrategy.sanitize_db_identifier()

  • When querying InfluxDB, quote table name (series/measurement) if identifier starts with a numeric value

  • Add “exclude” parameter to HTTP API for mitigating scaling/outlier problems when plotting

  • Fix “Excel worksheet name must be <= 31 chars.” by introducing “compact” title

2016-06-29 0.10.0

2016-06-17 0.9.0

2016-06-06 0.8.0

General

  • Add HTTP-to-MQTT protocol forwarder component, see HTTP acquisition API configuration

  • Add Terkin PHP, a HTTP API library for PHP, supports PHP5 and PHP4

  • Relocate configuration blueprints in etc/examples

Bugfixes

  • Update default credentials for Grafana 3.x compatibility (admin/admin)

  • Start HTTP server service only once, even when having multiple HTTP-to-X forwarders defined

Documentation

2016-05-22 0.7.1

  • Update default credentials for Grafana 3.x in Kotori configuration (admin/admin)

2016-05-22 0.7.0

Vendor Hiveeyes

  • Integrate and absorb communication style and subsystems of Hiveeyes/BERadio into core

  • Refactor into generic Twisted service MqttInfluxGrafanaService, then implement the Hiveeyes vendor application on top of it

Vendor LST

  • Improve command line tooling per lst-message <channel> info: Display common information about a data channel like the configuration object and the names of all structs.

  • Improve logging and debugging

  • Optionally put legend on the right hand side of the graph

General

  • Improve configuration, logging, debugging and documentation

  • Improve internal settings handling and application bootstrapping

  • Introduce service-in-service infrastructure

  • Make default Grafana panel not use steppedLines: true, smooth lines are more beautiful when displaying sine curves

  • Add MqttKitApplication, a generic application modeled after and using the Hiveeyes vendor infrastructure

  • Add PahoMqttAdapter: Migrate from twisted-mqtt to the Eclipse Paho MQTT Python client library paho-mqtt, to enable running more than one MQTT adapter instance

  • Introduce concept of “applications”, which are native Twisted services and can be bootstrapped by defining them in the configuration file

  • Add composite application completely declared by configuration settings

  • Adapt Hydro2Motion and LST to infrastructure changes

  • Upgrade libraries Twisted, autobahn, crossbar, msgpack and influxdb

  • Improve Grafana gracefulness when finding a corrupt panel

  • Overhaul configuration subsystem

  • Try to reconnect to MQTT broker in interval if initial connection fails

  • Add license, improve packaging and package publishing

Documentation

  • Document how to Run on PyPy

  • Improve documentation at About and Kotori

  • Add CSS3 Hexagon Buttons 1.0.1 and more static assets

  • Add Entypo pictograms by Daniel Bruce

2016-03-27 0.6.0

Vendor LST

  • resolve collision on parsed C header files when using identical filenames for different channels

  • add project “proptest”

Vendor Hiveeyes

  • improve configuration file “hiveeyes.ini” and logging

  • fix Grafana panel creation re. Grafana 2.6.0 compatibility, Grafana 2.1.3 still works though

  • fix Grafana panel creation re. InfluxDB select expression

  • don’t put global realm “hiveeyes” into Grafana dashboard name

  • improve Grafana panel automation

  • accept single values on mqtt topic

  • tune the default Grafana dashboard and panel

  • documentation updates

Packaging

  • Modularize python dependencies into extra features

  • Debian packaging using FPM

    • Read designated package version from setup.py

    • Use virtualenv-tools for relocating virtualenvs

    • Add systemd service configuration file

Miscellaneous

  • Documentation refactoring and improvements

2015-11-26 0.5.1

  • overhaul configuration files, activate “hydro2motion” channel with vendor LST

  • fix hydro2motion re. database authentication

  • lst: improve documentation

2015-11-26 0.5.0

Vendor LST

  • add sattracker application

  • fix WAMP serialization error when publishing binary data (e.g. “char 0x9c”) by using MsgPack serialization

  • augment c source file before compilation re. #include "mbed.h" vs. #include "stdint.h"

  • parse transformation rules from source code annotation

  • apply transformation rules before publishing to software bus

  • fix grafana dashboard update when having no panels

  • nasty hack to get proper struct initializer data from CParser results

  • show “average” column in Grafana

  • flexible compiler detection re. Linux vs. Mac OSX (MacPorts)

  • improve error handling when using interactive commands

  • explicitly convert values to float when evaluating SymPy expressions

  • influxdb: prevent float<->integer casting errors by converting all numerical values to float

  • upgrade to python influxdb-2.10.0

  • rename etc/lst-h2m.ini to etc/lst.ini

  • generalize h2m-message and sattracker-message into lst-message

  • specify configuration file via KOTORI_CONFIG environment variable

  • add “lst-message list-channels” command

  • wording: change “application” to “channel”

  • refactor configuration mechanics

2015-11-20 0.4.0

Proof-of-concept for vendor LST

  • add struct definitions of h2m project

  • add basic udp message sender in c++ based on h2m struct definitions

  • add infrastructure for parsing schema mappings from c/c++ header files based on pyclibrary

  • instantiate structs from compiled c/c++ header files/libraries

  • introduce struct registry for bookkeeping and runtime dispatching

  • decouple lst/h2m specific struct registry behavior based on ID attribute

  • add initial docs about lst/h2m spikes

  • properly tweak “h2m_structs.h” to be grokked by patched pyclibrary

  • make message receiving actually work in dry-dock, improve pretty-printing

  • add command line entrypoint “h2m-message” with “decode” and “info” actions

  • implement “h2m-message send”

  • lst main application component: receive, decode and store binary messages

  • automatic Grafana dashboard- and panel creation

General improvements

  • add release and documentation infrastructure through Makefile targets

  • fix panel generation for vendor hiveeyes

  • use nanosecond time precision with InfluxDB

  • lst: honour struct field order in Grafana

  • add more details to Grafana dashboard panels

  • improve error messages “h2m-message send/decode”

  • generalize c library adapter, multi-project capabilities for vendor lst

2015-11-06 0.3.2

Proof-of-concept for vendor Hiveeyes

  • upgrade foundation libraries: Twisted, Autobahn, Crossbar

  • receive messages via MQTT and store data points into InfluxDB

  • storage: add support for InfluxDB 0.9

  • storage: minor tweaks to enable influxdb database authentication

  • receive telemetry data from BERadio

  • grafana datasource- and dashboard automation

  • Sort “collect_fields” result before passing to grafana manager

Vendor Hydro2Motion

  • refactor hydro2motion code

User interface

  • add frontend foundation based on Pyramid web framework

  • add jQuery, Bootstrap, Fontawesome, html5shiv and respond.js

  • add material design for bootstrap

  • add prototype html based on SB Admin 2 bootstrap template

  • add modernizr and underscore

  • add foundation for page transitions from codrops

  • http: cache really static resources longer than volatile ones

  • ui: add pages with page transitions, about content, etc.

General improvements

  • refactor project layout

  • use configuration file instead of hardcoded configuration values

  • improve logging

2015-05-21 0.2.2

  • hydro2motion: production improvements from May 2015 in Rotterdam

2015-05-01 0.2.1

Vendor Hydro2Motion

  • ui: set map position to Munich

  • ui: add lat long conversion

  • backend: use InfluxDB on localhost

  • backend: process complete Fuelcell telemetry data package

2015-04-24 0.2.0

Proof-of-concept for vendor Hydro2Motion

  • ui: add d3 and rickshaw

  • ui: add timeseries prototype

  • ui: add cbuffer.js

  • ui: use ringbuffer for telemetry data

  • backend: more convenient default setting: listen on all interfaces

  • sensors: add temp sensor

  • backend: store telemetry data to sqlite database

  • middleware: reduce lag because of debug messages

  • middleware: disable heartbeat

  • backend: add mongodb adapter

  • ui: add leaflet map

  • ui: fix image baseurl for leaflet.js

  • ui: add marker to leaflet widget

  • ui: be graceful to old wire format for telemetry data

  • backend: store latitude and longitude into databases

  • ui: mapview: let the marker follow the position (map.panTo), but disable it

  • backend: add database adapter for InfluxDB and some documentation along the lines

  • improve documentation

2015-03-18 0.1.1

  • ui/backend: add persistent configuration store

  • ui: add bootstrap-editable css

  • namespace refactoring from ilaundry.node to kotori.node

  • upgrade javascript libraries to autobahn 0.10.1, add crossbar configuration

  • partial upgrade to autobahn 0.10.1

  • backend: add udp adapter

2014-01-21 0.1.0

  • node: reactivate heartbeat

  • node: mplayer user-agent hack for correctly spelling umlauts

  • ui: indicate motion activity from node

  • ui: indicate node online/offline state

  • ui: indicate privacy mode

  • ui: button for toggling operator presence

  • ether: refactored node registration, send hostname along

  • ui: layout refactoring, display more details

2014-01-13 0.0.4

  • ui: introduce Bootstrap, jQuery, underscore, etc.

  • ui: reflect multinode capabilities

2014-01-13 0.0.3

  • modularized into three components: master, node, web

  • single-daemon mode

  • first feature set on top of Adafruit_BBIO.GPIO

2014-01-05 0.0.2

  • Multiple nodes for real [NodeRegistry]

2014-01-05 0.0.1

Proof-of-concept for vendor iLaundry

  • Two daemons: master service and node service

  • Communication infrastructure on top of Autobahn using PubSub

  • Text-to-speech on top of Google Translate TTS

  • Basic HTML Dashboard GUI for sending text messages