Modify InfluxDB subscription to send data to multiple subscribers (asynchronous data replication)

Written by Claudio Kuenzler - 0 comments

Published on June 28th 2021 - Listed in Influx Database

---

To replicate data from an InfluxDB server to another server, one of the possibilities (in the open source/community edition) is to configure a "subscription". Another blog article describes how to create an InfluxDB asynchronous replication using subscription service.

Multiple subscribers

The current subscriptions can be shown on the master server:

root@influxmaster:~# influx -username admin -password secret
Connected to http://localhost:8086 version 1.6.4
InfluxDB shell version: 1.6.4
> SHOW SUBSCRIPTIONS
name: icinga
retention_policy name               mode destinations
---------------- ----               ---- ------------
autogen          icinga-replication ALL  [http://icinga:pass@influx02:8086]

name: mtr
retention_policy name            mode destinations
---------------- ----            ---- ------------
autogen          mtr-replication ALL  [http://mtr:pass@influx02:8086]

But what if data needs to be replicated across multiple nodes (subscribers)? 

The CREATE SUBSCRIPTION command syntax allows to define multiple subscribers as last input, separated with a comma:

CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ALL|ANY 'http://influx02:8086', 'http://influx03:8086'

Note: Very important to focus on the keywords "DESTINATIONS ALL" or "DESTINATIONS ANY". The ALL keyword stands for data replication to all defined subscribers. ANY stands for round-robin data replication across all defined subscribers!

The problem is that there is no "edit" function available for the subscriptions. Only the SHOW, CREATE and DROP commands are available. This means, the existing replication needs to be removed and then re-created with the additional subscriber.

Drop and re-create subscription

From the SHOW SUBSCRIPTIONS command we know the replication name "icinga-replication" and the source database "icinga". This subscription can now be dropped and then re-created with the additional subscriber:

> DROP SUBSCRIPTION "icinga-replication" ON "icinga"."autogen"
> CREATE SUBSCRIPTION "icinga-replication" ON "icinga"."autogen" DESTINATIONS ALL 'http://icinga:pass@influx02:8086', 'http://icinga:pass@influx03:8086'

The new subscriber should show up:

> SHOW SUBSCRIPTIONS
name: icinga
retention_policy name               mode destinations
---------------- ----               ---- ------------
autogen          icinga-replication ALL  [http://icinga:pass@influx02:8086 http://icinga:pass@influx03:8086]

Data loss during subscription re-creation

If your primary InfluxDB server continuously receives a lot of data, the data received during the DROP and CREATE SUBSCRIPTION commands will be lost. The subscription replication is a fire-and-forget kind of replication. When a subscriber does not respond, the data is not queued and then resent at a later time. The subscription service doesn't "catch up" to a certain moment in time either, such as a classical Master-Slave replication (e.g. MySQL).

To make sure all the InfluxDB servers hold the same data, stop all InfluxDB servers and manually sync /var/lib/influxdb from the master server to the others, then start InfluxDB and the subscription service replicated new incoming data.

Caution: No subscription replication in InfluxDB 2.0!

Unfortunately InfluxDB 2.0 does not offer subscription replication anymore. From the InfluxDB 1.x to 2.0 notes:

InfluxDB 2.0 has no subscriptions API and does not support Kapacitor stream tasks. To continue using stream tasks, write data directly to both InfluxDB and Kapacitor. Use Telegraf and its InfluxDB output plugin to write to Kapacitor and the InfluxDB v2 output plugin to write to InfluxDB v2.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.