Changelog

in progress

2017-05-05 0.20.1

  • Fix raw data export
  • Properly apply charset encoding for data export
  • Update documentation for vendors luftdaten.info and weeWX

2017-05-04 0.20.0

  • Re-add export/influx.py lost during refactoring
  • Fix InfluxDB UDP data acquisition
  • Tap into error signalling over MQTT
  • Minor updates to vendor luftdaten.info docs & co.

2017-05-03 0.19.1

  • Fix regression re. CSV data acquisition

2017-04-25 0.19.0

  • Vendor luftdaten.info:

    • Update Grafana dashboard “grafana-by-location.json”. Thanks, Richard!
    • Update documentation
  • Fix data convergence re. “time” field

2017-04-24 0.18.2

  • Attempt to improve InfluxDB write performance by using UDP protocol for vendor luftdaten.info

2017-04-24 0.18.1

  • Fix variable naming, references, data conversion and exception handling
  • Attempt to improve InfluxDB write performance
  • luftdaten-to-mqtt: Configure Nominatim cache directory conditionally

2017-04-24 0.18.0

  • Improve InfluxDB database creation behaviour

  • luftdaten.info:

    • Add geospatial data acquisition capabilities

    • Improve luftdaten_api_to_mqtt.py:

      • Generic commandline interface
      • Caching for Nominatim responses
      • Appropriate timestamp mungling
      • Documentation
    • Add LuftdatenGrafanaManager for provisioning appropriate Grafana dashboards

2017-04-21 0.17.0

  • Firmware builder improvements:

    • Don’t pull all build parameters into the artefact filename as this might raise OSError: [Errno 63] File name too long.
    • Automatically derive “MQTT_TOPIC” from telemetry channel address information
    • Update firmware builder configuration for vendor Hiveeyes

2017-04-19 0.16.0

  • Add standalone program “luftdaten_api_to_mqtt.py” to request data from live data API of luftdaten.info, enrich geospatial information and publish to MQTT bus. Thanks, Richard!
  • Add documentation and configuration for vendor luftdaten.info. Thanks again, Richard!
  • Fix HTTP routing for Firmware builder
  • Fix python module dependency woes re. OpenSSL on Darwin vs. Linux
  • Make Kotori run even without having “pandas” installed (w/o “export” extras)
  • Improve error logging in bootstrapping phase
  • Add support for architecture “ESP” to Firmware builder
  • Make transaction logging interval configurable, default to 60 seconds to reduce log traffic significantly
  • Add vendor weeWX: Tune MQTT data acquisition to support the weeWX weather station software. Thanks, Jan!

2017-03-29 0.15.0

2017-03-22 0.14.3

  • Properly use MQTT authentication also when publishing MQTT messages received via HTTP

2017-03-22 0.14.2

  • Fix setup.py again

2017-03-22 0.14.1

  • Fix setup.py

2017-03-21 0.14.0

  • Connect to the MQTT broker with authentication (default: kotori/kotori) to be able to apply Mosquitto ACLs to distinct MQTT topics. Thanks, Alex!

2017-03-20 0.13.0

  • Concurrency improvements: Don’t use the main reactor thread for MQTT message processing, use a different thread pool instead.

2017-03-20 0.12.3

  • Ubuntu 16.04 compatibility: Nail some more Python dependency modules. Thanks, Alex!

2017-03-20 0.12.2

  • Work on Ubuntu 16.04 compatibility: Nail some more Python dependency modules

2017-03-20 0.12.1

  • Add Python dependency “python-dateutil”

2017-03-16 0.12.0

2017-03-16 0.11.5

  • Prepare CSV import of http://archive.luftdaten.info/. Thanks, Richard!

    • Allow semicolon (;) as separator for CSV import
    • Allow CSV field timestamp as alias for datetime
  • Publish Grafana v4.2.0-beta1 debian packages to package repository, see Foundation packages and Setup on Debian.

  • Fix SSL runtime dependency woes AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1' with Twisted-17.1.0 vs. python-openssl vs. pyOpenSSL. Thanks, Matthias!

2017-02-12 0.11.4

  • Don’t always send http responses as text/plain
  • Catch MongoDB database errors, log and respond appropriately

2017-02-12 0.11.3

  • Upgrade some Python modules

2017-02-12 0.11.2

  • Fix Grafana dashboard builder for vendor Hiveeyes

2017-02-01 0.11.1

  • Data export: Fix addressing with relative timestamp, e.g. /data.txt?from=now-30d as well as proper handling of include and exclude url parameters

2017-01-31 0.11.0

2016-10-31 0.10.10

  • Fix Debian package re. superfluous “local” folder containing a redundant Python virtualenv. Thanks Smilie!
  • Fix receiving discrete measurements via MQTT. Thanks Karsten and Clemens!
  • Update Git repository url for hacking on Kotori
  • Improve documentation

2016-07-12 0.10.9

  • Documentation updates, add system diagrams to vendor Hiveeyes
  • Export csv and json data with ISO format timestamps to satisfy dygraphs rendering in Firefox
  • Don’t add “pad=true” or “backfill=true” when “interpolate=true” parameter was obtained from URL
  • Improve robustness of http api parameter evaluation and passing
  • Improve Vega asset loading: Use https resources, better safe than sorry
  • Add export format ”.tsv” (text/tab-separated-values)

2016-07-10 0.10.7

  • Update documentation
  • Rebuild without having “ggplot” installed on the build host

2016-07-10 0.10.6

  • Fix timeseries plotting by using “pad” and “backfill” appropriately
  • Add export parameters “exclude”, “include”, “interpolate” and “sorted”
  • Fix data routing and processing
  • Add license to documentation
  • Quick hack for making Firmware builder endpoint no convert numeric values to floats
  • Packaging fixes

2016-07-05 0.10.5

  • Attempt to fix huge dependency chain when installing with –install-recommends –install-suggests

2016-07-05 0.10.4

  • Fix missing runtime dependency “simplejson” (required by cornice)

2016-07-02 0.10.3

  • Use matplotlib “agg” backend
  • Improve ggplot rendering context, add font for rendering xkcd theme
  • Upgrade to pandas 0.18.1

2016-07-02 0.10.2

  • Honor https scheme in reverse proxy setups
  • Packaging: Remove Python dependency on crossbar, can be installed through pip install crossbar==0.13.0
  • Packaging: Depend on more distribution packages to reduce package size

2016-07-01 0.10.1

Packaging

  • Fix Debian runtime dependencies

Data export

  • Always emit lowercase values from WanBusStrategy.sanitize_db_identifier()
  • When querying InfluxDB, quote table name (series/measurement) if identifier starts with a numeric value
  • Add “exclude” parameter to HTTP API for mitigating scaling/outlier problems when plotting
  • Fix “Excel worksheet name must be <= 31 chars.” by introducing “compact” title

2016-06-29 0.10.0

2016-06-17 0.9.0

2016-06-06 0.8.0

General

Bugfixes

  • Update default credentials for Grafana 3.x compatibility (admin/admin)
  • Start HTTP server service only once, even when having multiple HTTP-to-X forwarders defined

Documentation

2016-05-22 0.7.1

  • Update default credentials for Grafana 3.x in Kotori configuration (admin/admin)

2016-05-22 0.7.0

Vendor Hiveeyes

  • Integrate and absorb communication style and subsystems of Hiveeyes/BERadio into core
  • Refactor into generic Twisted service MqttInfluxGrafanaService, then implement the Hiveeyes vendor application on top of it

Vendor LST

  • Improve command line tooling per lst-message <channel> info: Display common information about a data channel like the configuration object and the names of all structs.
  • Improve logging and debugging
  • Optionally put legend on the right hand side of the graph

General

  • Improve configuration, logging, debugging and documentation
  • Improve internal settings handling and application bootstrapping
  • Introduce service-in-service infrastructure
  • Make default Grafana panel not use steppedLines: true, smooth lines are more beautiful when displaying sine curves
  • Add MqttKitApplication, a generic application modeled after and using the Hiveeyes vendor infrastructure
  • Add PahoMqttAdapter: Migrate from twisted-mqtt to the Eclipse Paho MQTT Python client library paho-mqtt, to enable running more than one MQTT adapter instance
  • Introduce concept of “applications”, which are native Twisted services and can be bootstrapped by defining them in the configuration file
  • Add composite application completely declared by configuration settings
  • Adapt Hydro2Motion and LST to infrastructure changes
  • Upgrade libraries Twisted, autobahn, crossbar, msgpack and influxdb
  • Improve Grafana gracefulness when finding a corrupt panel
  • Overhaul configuration subsystem
  • Try to reconnect to MQTT broker in interval if initial connection fails
  • Add license, improve packaging and package publishing

Documentation

  • Document how to Run on PyPy
  • Improve documentation at About Kotori and Kotori
  • Add CSS3 Hexagon Buttons 1.0.1 and more static assets
  • Add Entypo pictograms by Daniel Bruce

2016-03-27 0.6.0

Vendor LST

  • resolve collision on parsed C header files when using identical filenames for different channels
  • add project “proptest”

Vendor Hiveeyes

  • improve configuration file “hiveeyes.ini” and logging
  • fix Grafana panel creation re. Grafana 2.6.0 compatibility, Grafana 2.1.3 still works though
  • fix Grafana panel creation re. InfluxDB select expression
  • don’t put global realm “hiveeyes” into Grafana dashboard name
  • improve Grafana panel automation
  • accept single values on mqtt topic
  • tune the default Grafana dashboard and panel
  • documentation updates

Packaging

  • Modularize python dependencies into extra features

  • Debian packaging using FPM

    • Read designated package version from setup.py
    • Use virtualenv-tools for relocating virtualenvs
    • Add systemd service configuration file

Miscellaneous

  • Documentation refactoring and improvements

2015-11-26 0.5.1

  • overhaul configuration files, activate “hydro2motion” channel with vendor LST
  • fix hydro2motion re. database authentication
  • lst: improve documentation

2015-11-26 0.5.0

Vendor LST

  • add sattracker application
  • fix WAMP serialization error when publishing binary data (e.g. “char 0x9c”) by using MsgPack serialization
  • augment c source file before compilation re. #include "mbed.h" vs. #include "stdint.h"
  • parse transformation rules from source code annotation
  • apply transformation rules before publishing to software bus
  • fix grafana dashboard update when having no panels
  • nasty hack to get proper struct initializer data from CParser results
  • show “average” column in Grafana
  • flexible compiler detection re. Linux vs. Mac OSX (MacPorts)
  • improve error handling when using interactive commands
  • explicitly convert values to float when evaluating SymPy expressions
  • influxdb: prevent float<->integer casting errors by converting all numerical values to float
  • upgrade to python influxdb-2.10.0
  • rename etc/lst-h2m.ini to etc/lst.ini
  • generalize h2m-message and sattracker-message into lst-message
  • specify configuration file via KOTORI_CONFIG environment variable
  • add “lst-message list-channels” command
  • wording: change “application” to “channel”
  • refactor configuration mechanics

2015-11-20 0.4.0

Proof-of-concept for vendor LST

  • add struct definitions of h2m project
  • add basic udp message sender in c++ based on h2m struct definitions
  • add infrastructure for parsing schema mappings from c/c++ header files based on pyclibrary
  • instantiate structs from compiled c/c++ header files/libraries
  • introduce struct registry for bookkeeping and runtime dispatching
  • decouple lst/h2m specific struct registry behavior based on ID attribute
  • add initial docs about lst/h2m spikes
  • properly tweak “h2m_structs.h” to be grokked by patched pyclibrary
  • make message receiving actually work in dry-dock, improve pretty-printing
  • add command line entrypoint “h2m-message” with “decode” and “info” actions
  • implement “h2m-message send”
  • lst main application component: receive, decode and store binary messages
  • automatic Grafana dashboard- and panel creation

General improvements

  • add release and documentation infrastructure through Makefile targets
  • fix panel generation for vendor hiveeyes
  • use nanosecond time precision with InfluxDB
  • lst: honour struct field order in Grafana
  • add more details to Grafana dashboard panels
  • improve error messages “h2m-message send/decode”
  • generalize c library adapter, multi-project capabilities for vendor lst

2015-11-06 0.3.2

Proof-of-concept for vendor Hiveeyes

  • upgrade foundation libraries: Twisted, Autobahn, Crossbar
  • receive messages via MQTT and store data points into InfluxDB
  • storage: add support for InfluxDB 0.9
  • storage: minor tweaks to enable influxdb database authentication
  • receive telemetry data from BERadio
  • grafana datasource- and dashboard automation
  • Sort “collect_fields” result before passing to grafana manager

Vendor Hydro2Motion

  • refactor hydro2motion code

User interface

  • add frontend foundation based on Pyramid web framework
  • add jQuery, Bootstrap, Fontawesome, html5shiv and respond.js
  • add material design for bootstrap
  • add prototype html based on SB Admin 2 bootstrap template
  • add modernizr and underscore
  • add foundation for page transitions from codrops
  • http: cache really static resources longer than volatile ones
  • ui: add pages with page transitions, about content, etc.

General improvements

  • refactor project layout
  • use configuration file instead of hardcoded configuration values
  • improve logging

2015-05-21 0.2.2

  • hydro2motion: production improvements from May 2015 in Rotterdam

2015-05-01 0.2.1

Vendor Hydro2Motion

  • ui: set map position to Munich
  • ui: add lat long conversion
  • backend: use InfluxDB on localhost
  • backend: process complete Fuelcell telemetry data package

2015-04-24 0.2.0

Proof-of-concept for vendor Hydro2Motion

  • ui: add d3 and rickshaw
  • ui: add timeseries prototype
  • ui: add cbuffer.js
  • ui: use ringbuffer for telemetry data
  • backend: more convenient default setting: listen on all interfaces
  • sensors: add temp sensor
  • backend: store telemetry data to sqlite database
  • middleware: reduce lag because of debug messages
  • middleware: disable heartbeat
  • backend: add mongodb adapter
  • ui: add leaflet map
  • ui: fix image baseurl for leaflet.js
  • ui: add marker to leaflet widget
  • ui: be graceful to old wire format for telemetry data
  • backend: store latitude and longitude into databases
  • ui: mapview: let the marker follow the position (map.panTo), but disable it
  • backend: add database adapter for InfluxDB and some documentation along the lines
  • improve documentation

2015-03-18 0.1.1

  • ui/backend: add persistent configuration store
  • ui: add bootstrap-editable css
  • namespace refactoring from ilaundry.node to kotori.node
  • upgrade javascript libraries to autobahn 0.10.1, add crossbar configuration
  • partial upgrade to autobahn 0.10.1
  • backend: add udp adapter

2014-01-21 0.1.0

  • node: reactivate heartbeat
  • node: mplayer user-agent hack for correctly spelling umlauts
  • ui: indicate motion activity from node
  • ui: indicate node online/offline state
  • ui: indicate privacy mode
  • ui: button for toggling operator presence
  • ether: refactored node registration, send hostname along
  • ui: layout refactoring, display more details

2014-01-13 0.0.4

  • ui: introduce Bootstrap, jQuery, underscore, etc.
  • ui: reflect multinode capabilities

2014-01-13 0.0.3

  • modularized into three components: master, node, web
  • single-daemon mode
  • first feature set on top of Adafruit_BBIO.GPIO

2014-01-05 0.0.2

  • Multiple nodes for real [NodeRegistry]

2014-01-05 0.0.1

Proof-of-concept for vendor iLaundry

  • Two daemons: master service and node service
  • Communication infrastructure on top of Autobahn using PubSub
  • Text-to-speech on top of Google Translate TTS
  • Basic HTML Dashboard GUI for sending text messages