[influxdb] Kapacitor subscription to influxdb traffic v/s InfluxdbRelay push to kapacitor

[k***@gmail.com]

I am setting up high availability influxdb setup as mentioned on influxdb site.

loadbalancer-->relay-->influxdbs(pri/sec) and kapacitor

Though there are obvious advantages to sending real time traffic via influxdbrelay to Kapacitor together with influxdbs,it takes away flexibility to spin up multiple kapacitors in the environment without updating relay configuration.

Our usecase has several teams who want to control their own alerting configuration in kapacitor so we are looking for spinning up multiple kapacitors in the environment (one controlled by each team). I have couple questions around that:

1) Do you forsee any issues with having multiple kapacitors in the environment talking to same influxdb servers (if they subscribe themselves to influxdb servers instead of relay pushing traffic to kapacitors for which it would need to be updated).

2) If i have two influxdb servers which are identical (primary and secondary), where should my kapacitor point to? (load balancer DNS or influxdb-primary and influxdb-secondary in the list [host1, host2]?

3) How many tasks can one kapacitor handle ( I can throw as many resources (cpu/ram) on it as i want)?


Thanks
Kirtimaan

[k***@gmail.com]

Thank you! your explanation is to the point and very helpful.

[j***@gmail.com]

Hi Nathan

While reading this discussion, I got a question on your statement
"I would look at option 2. This is where my questions comes in. If you are running multiple relays, which I recommend you do, then it should not be too difficult to update the config on each relay one at a time, making use of the load balancer. Thus enabling you to add/remove Kapacitor instances dynamically without downtime. "

My Questions here are
1. If Relays are configured with Load balancer to sends the data points to Kapacitor, the load balancer will send to only kapacitor instance right? The load balance will not distribute the data points to multiple kapacitors instances which are behind the load balancer.

2. How to configure the relays to send the data points to Kapacitor and what is the configuration required in Kapacitor?

Thanks
Jayanna

Kirtimaan,
1) Do you want multiple Kapacitor instances because of redundancy/isolation or because you are worried about capacity?
2) Are you running multiple relays? If so how easy is it to reconfigure them?
3) How often do you expect you will need a new Kapacitor instance? Meaning, once every team is using it, are new teams created frequently or would the need to create new Kapacitor instances flatten out?
I don't need the answer to these question, but they should point you in the right direction.
1) I don't forsee any issues having multiple Kapacitor instances connecting to the same InfluxDB instance. The Kapacitor instances will be isolated and unaware of each other, InfluxDB will handle the subscriptions well. We have tested this use case.
2) This is the tricky part. Since each InfluxDB server is receiving all the data you can't have Kapacitor subscribe to both InfluxDB instances without duplicating all your data to Kapacitor. As a result if you are going to use subscriptions you have to pick only one InfluxDB server, which means your HA Kapacitor story now has two points of failure, Kapacitor itself and the InfluxDB server. Obviously not ideal
3) This depends on the workload each task adds, but we have tested Kapacitor for use cases of 1000+ tasks. In other words as long as you have the cpu/ram you need, which is sounds like you do, a single Kapacitor instance should be efficient at handling many tasks.
1. Use a single Kapacitor instance
2. Run multiple Kapacitor instances behind the relay.
Option 1 is the simplest but you risk one team creating a heavy task and breaking alerting for all other teams. If this is a serious concern then I would look at option 2. This is where my questions comes in. If you are running multiple relays, which I recommend you do, then it should not be too difficult to update the config on each relay one at a time, making use of the load balancer. Thus enabling you to add/remove Kapacitor instances dynamically without downtime.  Depending on the frequency of adding and removing Kapacitor instances  and the importance of isolation/redundancy you can invest in making that process as flexible subscriptions.
Does that help give you a direction? Do you have any further questions?
As a final note, if you do run into performance issues please let us know. Kapacitor is still a young product and when new use cases pop up, we want to learn how we can improve Kapacitor for those types of workloads. That said what you are doing here seems reasonable and within the scope of what we have tested.
Thanks
loadbalancer-->relay-->influxdbs(pri/sec) and kapacitor
Though there are obvious advantages to sending real time traffic via influxdbrelay to Kapacitor together with influxdbs,it takes away flexibility to spin up multiple kapacitors in the environment without updating relay configuration.
1) Do you forsee any issues with having multiple kapacitors in the environment talking to same influxdb servers (if they subscribe themselves to influxdb servers instead of relay pushing traffic to kapacitors for which it would need to be updated).
2) If i have two influxdb servers which are identical (primary and secondary), where should my kapacitor point to? (load balancer DNS or influxdb-primary and influxdb-secondary in the list [host1, host2]?
3) How many tasks can one kapacitor handle ( I can throw as many resources (cpu/ram) on it as i want)?
Thanks
Kirtimaan

[k***@gmail.com]

What should be the setting for chronograf in above scenario ...ie.,

client -> load balancer -> relays -> influxdb, kapacitor (can have multiple instances of each)

Kapacitor should be configured to disable subscriptions and use the load balancer when querying InfluxDB.

I am using latest version of chronograf

[k***@gmail.com]

Reason for asking for config is that even after using loadbalancer's name in kapacitor configuration, i see kapacitor wrote alert only in one of the two influxdbs and not the other when i have all /write/* going to relays.

Load balancer config (single load balancer for all influxdb stack):
/query/* --> influxdb1 and influxdb2 (one of the two will be used for query)
/write/* --> relays (sends to influxdb1/2/kapacitor)
/kapacitor/* --> kapacitor
/chronograf/* --> chronograf

My kapacitor config is:
[[influxdb]]
enabled = true
name = "influxdb-cluster"
default = true
urls = ["https://<loadbalancer>"]
disable-subscriptions = true
username = "<user>"
password = "<passwd>"

So i am wondering if my chronograf or kapacitor setting is issue?
In chronograf i am using loadbalancer name as kapacitor endpoint
and the same loadbalancer name as influxdb endpoint
hoping that all API paths will fit in above loadbalancer routing configuration.

[k***@gmail.com]

I was using wrong path /write/* for routin gtraffic when it should be /write* at load balancer.