Ingesting weather data from WeeWX using MQTT#

Introduction#

I’m writing a thesis about automatic acquisition of meteorological data from the Davis Vantage Pro2 station and i’m using Kotori and WeeWX for that, based on Kotori’s WeeWX integration. I have a question about how Kotori ingests data to InfluxDB. Is it through HTTP GET requests? What does Kotori do to the JSON that comes from MQTT?

  • Firstly, I utilized the MQTT Plugin for WeeWX to publish data to the MQTT broker with a specific topic.

  • Then, I subscribed Kotori to that topic by specifying it in the configuration file weewx.ini.

  • By subscribing to that topic, I now have the JSON payload and now I want to ingest that data into InfluxDB accordingly. I don’t want to publish again data into the broker like outlined within MQTT data acquisition.

Data ingest using MQTT#

Question#

Are you sure the data is ingested into InfluxDB through MQTT without using HTTP POST requests?

Answer#

You are on the right path, that is exactly how it works. By publishing measurement data to the MQTT bus and configuring Kotori appropriately, it will converge the data into the InfluxDB timeseries database without further ado. Using HTTP POST is a different path for ingesting data, we will just use MQTT here.

Application configuration#

Question#

What does this application option in the configuration snippet mean?

; -----------------------------
; Data acquisition through MQTT
; -----------------------------
[weewx]
enable      = true
type        = application
realm       = weewx
mqtt_topics = weewx/#
application = kotori.daq.application.mqttkit:mqttkit_application

Answer#

The core of each data acquisition application is its configuration object. application designates the machinery should invoke the entrypoint function mqttkit_application within the Python module kotori.daq.application.mqttkit for spawing an application.

For further details, see also mqttkit_application and the documentation section about the Application configuration.

Exporting data#

Question#

I have a question about the export of data from InfluxDB in the weewx.ini configuration file. My question is about the source, target and transform options within the configuration snippet referenced below.

  • The source option defines a HTTP GET request to the database.

  • The target option is used to define the InfluxDB database and measurement where the timeseries data is stored.

  • Maybe you can shed some more light onto the transform option.

  • Is this snippet used to get the JSON payload published in order to put it into InfluxDB?

  • Why is the export necessary if I can see data in Grafana already?

; ----------------------------------------------------------------------
; Data export
; https://getkotori.org/docs/handbook/export/
; https://getkotori.org/docs/handbook/forwarders/http-api-export.html
; ----------------------------------------------------------------------
[weewx.data-export]
enable          = true

type            = application
application     = kotori.io.protocol.forwarder:boot

realm           = weewx
source          = http:/api/{realm:weewx}/{network:.*}/{gateway:.*}/{node:.*}/{slot:(data|event)}.{suffix} [GET]
target          = influxdb:/{database}?measurement={measurement}
transform       = kotori.daq.strategy.wan:WanBusStrategy.topology_to_storage,
                  kotori.io.protocol.influx:QueryTransformer.transform

Answer#

Is this snippet used to get the JSON payload published in order to put it into InfluxDB?

It is rather about “retreiving” the data from InfluxDB. This snippet implements the data export functionality.

Maybe you can shed some more light onto the transform option.

This is how you read this configuration snippet:

The source option defines a HTTP GET endpoint which, when requested, will run an InfluxDB query against the thing defined by target after being transformed through the machinery defined by transform.

All the heavy lifting is under the hood and implemented by the designated software components like WanBusStrategy and QueryTransformer. The configuration object just ties things together.

The transform setting is for defining components which take the request data and build a sensible InfluxDB query and appropriate request from it in order to pull data from InfluxDB.

Conclusion#

We hope this will shed more light onto the whole thing how data is flowing from WeeWX over MQTT through the Kotori into InfluxDB and how it is finally displayed within Grafana.

All of these mechanisms and subsystems have been designed to a) get you started with data acquisition and graphing instantly and b) will give you the freedom to establish any number of data acquisition channels without having to provision them explicitly beforehand. The motto here is: Just throw a bunch of JSON at the system.

Feel free to ask for more specific things which might still be unclear.

Resources#