[influxdb] kapacitor sending ok alerts as state change on start up

[Archie Archbold]

Hey all. Pretty new to TICK but I have a problem that I can't wrap my head
around.

I am monitoring multiple servers all sending data to one influxdb database
and using the 'host' tag to separate the servers in the DB

My 'disk' measurement is taking in mulitiple disk paths from the servers
(HOSTS) which each have a respective 'PATH' tag.

So basically each server is assigned a HOST tag and each HOST has multiple
PATH tags.

EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of a
HOST's PATH if that path is within the alerting Lambda.
PROBLEM: When I start the kapacitor service, it looks like it's sensing a
state change any time it sees another host/path with a opposite status.

This is a simplified example of the alerts I am getting:

Host: host1 Path: /path1 Status: UP
Host: host1 Path: /path2 Status: DOWN
Host: host1 Path: /path3 Status: UP
Host: host2 Path: /path1 Status: DOWN
Host: host2 Path: /path2 Status: UP

These alerts happen once for each host/path combination and then the
service performs as expected, alerting properly when lambda is achieved.

The result of this is that I receive a slew of up/down alerts every time I
restart the kapacitor service

Here is my current tick:
var data = stream
|from()
.measurement('disk')
.groupBy('host','path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{ index
.Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email($DISK_WARN_GRP)

And the corresponding DOT

ID: disk_alert_warn

Error:

Template:

Type: stream

Status: enabled

Executing: true

Created: 17 Feb 17 22:27 UTC

Modified: 17 Feb 17 22:27 UTC

LastEnabled: 17 Feb 17 22:27 UTC

Databases Retention Policies: ["main"."autogen"]

TICKscript:

var data = stream

|from()

.measurement('disk')

.groupBy('host', 'path')

|alert()

.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{ index
.Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')

.warn(lambda: "used_percent" >= 80)

.id('DISK SPACE WARNING')

.email()


DOT:

digraph disk_alert_warn {

graph [throughput="38.00 points/s"];


stream0 [avg_exec_time_ns="0s" ];

stream0 -> from1 [processed="284"];


from1 [avg_exec_time_ns="3.9µs" ];

from1 -> alert2 [processed="284"];


alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs"
crits_triggered="0" infos_triggered="0" oks_triggered="7"
warns_triggered="7" ];

}

As you can see, I get 7 oks triggered (for host/path groups that are not in
alert range) and 7 warns triggered (for the 7 host/path groups that are
within the alert range) upon start up.
Then it behaves as normal.

I understand that it should be alerting for the 7 host/path groups that are
over 80 but why follow it with an alert about the ok groups?

MORE INFO: When I raise the lambda to 90% (out of range for all host/paths)
I get no alerts at all (which is expected)

Thanks to anyone who can help me understand this

[Archie Archbold]

Thanks so much for the reply. I do want the recovery alerts, but the
problem is that when I start kapacitor, the task sees *any* server/path in
an up status as a recovery of a *different* server/path's down status. So
if 7 server/paths are in a down status at the time of start-up I get 7 down
alerts (expected) but they are immediately followed by 7 recovery messages
from different server/paths. Please let me know if I am not being clear
enough.

If you want to ignore the OK alerts use the `.noRecoveries` property of
the alert node. This will suppress the OK alerts.

Post by Archie Archbold
Hey all. Pretty new to TICK but I have a problem that I can't wrap my
head around.
I am monitoring multiple servers all sending data to one influxdb
database and using the 'host' tag to separate the servers in the DB
My 'disk' measurement is taking in mulitiple disk paths from the servers
(HOSTS) which each have a respective 'PATH' tag.
So basically each server is assigned a HOST tag and each HOST has
multiple PATH tags.
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of a
HOST's PATH if that path is within the alerting Lambda.
PROBLEM: When I start the kapacitor service, it looks like it's sensing a
state change any time it sees another host/path with a opposite status.
Host: host1 Path: /path1 Status: UP
Host: host1 Path: /path2 Status: DOWN
Host: host1 Path: /path3 Status: UP
Host: host2 Path: /path1 Status: DOWN
Host: host2 Path: /path2 Status: UP
These alerts happen once for each host/path combination and then the
service performs as expected, alerting properly when lambda is achieved.
The result of this is that I receive a slew of up/down alerts every time
I restart the kapacitor service
var data = stream
|from()
.measurement('disk')
.groupBy('host','path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
|from()
.measurement('disk')
.groupBy('host', 'path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs"
crits_triggered="0" infos_triggered="0" oks_triggered="7"
warns_triggered="7" ];
}
As you can see, I get 7 oks triggered (for host/path groups that are not
in alert range) and 7 warns triggered (for the 7 host/path groups that are
within the alert range) upon start up.
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups that
are over 80 but why follow it with an alert about the ok groups?
MORE INFO: When I raise the lambda to 90% (out of range for all
host/paths) I get no alerts at all (which is expected)
Thanks to anyone who can help me understand this

[Archie Archbold]

This behavior is not limited to just slack notifications. If email is
configured it sends out same number of notifications.

I will put the logs together and post.

Hmm, I see. Could you put a log node before the alert node and share the
logs along with the logs for triggered alerts after startup?
Also this may be a bug with the most recent alerting system. Do you get
the same behavior if you configure slack directly in the TICKscript instead
of via the topic handler?
On Wednesday, February 22, 2017 at 12:23:53 PM UTC-7, Archie Archbold

Post by Archie Archbold
Thanks so much for the reply. I do want the recovery alerts, but the
problem is that when I start kapacitor, the task sees *any* server/path
in an up status as a recovery of a *different* server/path's down
status. So if 7 server/paths are in a down status at the time of start-up I
get 7 down alerts (expected) but they are immediately followed by 7
recovery messages from different server/paths. Please let me know if I am
not being clear enough.
On Wednesday, February 22, 2017 at 11:10:09 AM UTC-8,

If you want to ignore the OK alerts use the `.noRecoveries` property of
the alert node. This will suppress the OK alerts.

Post by Archie Archbold
Hey all. Pretty new to TICK but I have a problem that I can't wrap my
head around.
I am monitoring multiple servers all sending data to one influxdb
database and using the 'host' tag to separate the servers in the DB
My 'disk' measurement is taking in mulitiple disk paths from the
servers (HOSTS) which each have a respective 'PATH' tag.
So basically each server is assigned a HOST tag and each HOST has
multiple PATH tags.
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of a
HOST's PATH if that path is within the alerting Lambda.
PROBLEM: When I start the kapacitor service, it looks like it's sensing
a state change any time it sees another host/path with a opposite status.
Host: host1 Path: /path1 Status: UP
Host: host1 Path: /path2 Status: DOWN
Host: host1 Path: /path3 Status: UP
Host: host2 Path: /path1 Status: DOWN
Host: host2 Path: /path2 Status: UP
These alerts happen once for each host/path combination and then the
service performs as expected, alerting properly when lambda is achieved.
The result of this is that I receive a slew of up/down alerts every
time I restart the kapacitor service
var data = stream
|from()
.measurement('disk')
.groupBy('host','path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
|from()
.measurement('disk')
.groupBy('host', 'path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs"
crits_triggered="0" infos_triggered="0" oks_triggered="7"
warns_triggered="7" ];
}
As you can see, I get 7 oks triggered (for host/path groups that are
not in alert range) and 7 warns triggered (for the 7 host/path groups that
are within the alert range) upon start up.
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups that
are over 80 but why follow it with an alert about the ok groups?
MORE INFO: When I raise the lambda to 90% (out of range for all
host/paths) I get no alerts at all (which is expected)
Thanks to anyone who can help me understand this

[Archie Archbold]

kap |
'##:::'##::::'###::::'########:::::'###:::::'######::'####:'########::'#######::'########::
kap | ##::'##::::'## ##::: ##.... ##:::'## ##:::'##...
##:. ##::... ##..::'##.... ##: ##.... ##:
kap | ##:'##::::'##:. ##:: ##:::: ##::'##:. ##::
##:::..::: ##::::: ##:::: ##:::: ##: ##:::: ##:
kap | #####::::'##:::. ##: ########::'##:::. ##:
##:::::::: ##::::: ##:::: ##:::: ##: ########::
kap | ##. ##::: #########: ##.....::: #########:
##:::::::: ##::::: ##:::: ##:::: ##: ##.. ##:::
kap | ##:. ##:: ##.... ##: ##:::::::: ##.... ##: ##:::
##:: ##::::: ##:::: ##:::: ##: ##::. ##::
kap | ##::. ##: ##:::: ##: ##:::::::: ##:::: ##:.
######::'####:::: ##::::. #######:: ##:::. ##:
kap |
..::::..::..:::::..::..:::::::::..:::::..:::......:::....:::::..::::::.......:::..:::::..::
kap |
kap | 2017/02/22 22:20:21 Using configuration at:
/etc/kapacitor/kapacitor.conf
kap | [httpd] ::1 - - [22/Feb/2017:22:20:31 +0000] "PATCH
/kapacitor/v1/tasks/disk_alert_dead HTTP/1.1" 200 788 "-" "KapacitorClient"
1f51cf6b-f94d-11e6-805d-000000000000 16607
kap | [httpd] ::1 - - [22/Feb/2017:22:20:31 +0000] "GET
/kapacitor/v1/tasks?dot-view=attributes&fields=link&limit=100&offset=0&pattern=disk_alert_warn&replay-id=&script-format=formatted
HTTP/1.1" 200 131 "-" "KapacitorClient"
1f553781-f94d-11e6-805e-000000000000 5815
kap | [task_master:main] 2017/02/22 22:20:31 I! Started
task: disk_alert_warn
kap | [httpd] ::1 - - [22/Feb/2017:22:20:31 +0000] "PATCH
/kapacitor/v1/tasks/disk_alert_warn HTTP/1.1" 200 754 "-" "KapacitorClient"
1f5640c5-f94d-11e6-805f-000000000000 18441
kap | [httpd] ::1 - - [22/Feb/2017:22:20:31 +0000] "GET
/kapacitor/v1/tasks?dot-view=attributes&fields=link&limit=100&offset=0&pattern=http_status_alert_crit&replay-id=&script-format=formatted
HTTP/1.1" 200 137 "-" "KapacitorClient"
1f59f50c-f94d-11e6-8060-000000000000 754
kap | [task_master:main] 2017/02/22 22:20:31 I! Started
task: http_status_alert_crit
kap | [httpd] ::1 - - [22/Feb/2017:22:20:31 +0000] "PATCH
/kapacitor/v1/tasks/http_status_alert_crit HTTP/1.1" 200 1207 "-"
"KapacitorClient" 1f5a3816-f94d-11e6-8061-000000000000 49713
kap | [httpd] ::1 - - [22/Feb/2017:22:20:31 +0000] "GET
/kapacitor/v1/tasks?dot-view=attributes&fields=link&limit=100&offset=0&pattern=http_status_alert_dead&replay-id=&script-format=formatted
HTTP/1.1" 200 137 "-" "KapacitorClient"
1f62be2e-f94d-11e6-8062-000000000000 716
kap | [task_master:main] 2017/02/22 22:20:31 I! Started
task: http_status_alert_dead
kap | [httpd] ::1 - - [22/Feb/2017:22:20:31 +0000] "PATCH
/kapacitor/v1/tasks/http_status_alert_dead HTTP/1.1" 200 729 "-"
"KapacitorClient" 1f62fb2e-f94d-11e6-8063-000000000000 17563
kap | [httpd] 172.18.0.2 - - [22/Feb/2017:22:20:31 +0000]
"POST /write?consistency=&db=main&precision=ns&rp=autogen HTTP/1.1" 204 0
"-" "InfluxDBClient" 1f90e7a5-f94d-11e6-8064-000000000000 439
kap | [disk_alert_warn:log2] 2017/02/22 22:20:31 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyx,path=/","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-root","fstype":"ext3","host":"jira.hostname.xyx","path":"/"},"Fields":{"free":15286812672,"inodes_free":1373362,"inodes_total":1495552,"inodes_used":122190,"total":24210894848,"used":7703310336,"used_percent":33.50704271273119},"Time":"2017-02-22T22:20:30Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:31 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyx,path=/boot","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"sda1","fstype":"ext3","host":"jira.hostname.xyx","path":"/boot"},"Fields":{"free":377835520,"inodes_free":32715,"inodes_total":32768,"inodes_used":53,"total":528424960,"used":123748352,"used_percent":24.671517348946978},"Time":"2017-02-22T22:20:30Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:31 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyx,path=/home","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-home","fstype":"ext3","host":"jira.hostname.xyx","path":"/home"},"Fields":{"free":12615372800,"inodes_free":884437,"inodes_total":884736,"inodes_used":299,"total":14268055552,"used":928071680,"used_percent":6.85255277097721},"Time":"2017-02-22T22:20:30Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:31 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyx,path=/tmp","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-temp","fstype":"ext3","host":"jira.hostname.xyx","path":"/tmp"},"Fields":{"free":880750592,"inodes_free":65424,"inodes_total":65536,"inodes_used":112,"total":1056858112,"used":122421248,"used_percent":12.20341751219811},"Time":"2017-02-22T22:20:30Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:31 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyx,path=/var","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-var","fstype":"ext3","host":"jira.hostname.xyx","path":"/var"},"Fields":{"free":26661912576,"inodes_free":4707430,"inodes_total":4792320,"inodes_used":84890,"total":77218316288,"used":46634835968,"used_percent":63.62469945035165},"Time":"2017-02-22T22:20:30Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:31 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyx,path=/var/log","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-varlog","fstype":"ext3","host":"jira.hostname.xyx","path":"/var/log"},"Fields":{"free":206135296,"inodes_free":65441,"inodes_total":65536,"inodes_used":95,"total":1056858112,"used":797036544,"used_percent":79.45164648959843},"Time":"2017-02-22T22:20:30Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:31 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyx,path=/var/log/audit","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-audit","fstype":"ext3","host":"jira.hostname.xyx","path":"/var/log/audit"},"Fields":{"free":1905217536,"inodes_free":131056,"inodes_total":131072,"inodes_used":16,"total":2113748992,"used":101158912,"used_percent":5.041870985917793},"Time":"2017-02-22T22:20:30Z"}
kap |
kap | [httpd] 172.18.0.2 - - [22/Feb/2017:22:20:31 +0000]
"POST /write?consistency=&db=main&precision=ns&rp=autogen HTTP/1.1" 204 0
"-" "InfluxDBClient" 1f90ff3b-f94d-11e6-8065-000000000000 381

.
.
.
kap DOWN | [disk_alert_warn:log2] 2017/02/22 22:20:45 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=calendar.hostname.xyz,path=/","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-root","fstype":"ext3","host":"calendar.hostname.xyz","path":"/"},"Fields":{"free":610889728,"inodes_free":146007,"inodes_total":235712,"inodes_used":89705,"total":3798114304,"used":2994286592,"used_percent":83.05520524444141},"Time":"2017-02-22T22:20:48Z"}
kap |
kap UP | [disk_alert_warn:log2] 2017/02/22 22:20:45 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=calendar.hostname.xyz,path=/boot","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"sda1","fstype":"ext3","host":"calendar.hostname.xyz","path":"/boot"},"Fields":{"free":389566464,"inodes_free":32717,"inodes_total":32768,"inodes_used":51,"total":528424960,"used":112017408,"used_percent":22.332737205713027},"Time":"2017-02-22T22:20:48Z"}
.
.
.
kap DOWN | [disk_alert_warn:log2] 2017/02/22 22:20:46 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=mavericks.hostname.xyz,path=/home","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/VolGroup01-LogVol02","fstype":"ext3","host":"mavericks.hostname.xyz","path":"/home"},"Fields":{"free":151598505984,"inodes_free":133300363,"inodes_total":133562368,"inodes_used":262005,"total":1076941541376,"used":870637568000,"used_percent":85.16991232825805},"Time":"2017-02-22T22:20:45Z"}
kap |
kap | [httpd] 172.18.0.2 - - [22/Feb/2017:22:20:46 +0000]
"POST /write?consistency=&db=main&precision=ns&rp=autogen HTTP/1.1" 204 0
"-" "InfluxDBClient" 280b1ecb-f94d-11e6-80ab-000000000000 265
kap | [disk_alert_warn:log2] 2017/02/22 22:20:46 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=mavericks.hostname.xyz,path=/","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/VolGroup00-LogVol00","fstype":"ext3","host":"mavericks.hostname.xyz","path":"/"},"Fields":{"free":33492471808,"inodes_free":11372969,"inodes_total":11555808,"inodes_used":182839,"total":45829500928,"used":9971441664,"used_percent":22.941886423604558},"Time":"2017-02-22T22:20:45Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:46 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=mavericks.hostname.xyz,path=/boot","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"sda1","fstype":"ext3","host":"mavericks.hostname.xyz","path":"/boot"},"Fields":{"free":55656448,"inodes_free":26045,"inodes_total":26104,"inodes_used":59,"total":103512064,"used":42511360,"used_percent":43.304786840101386},"Time":"2017-02-22T22:20:45Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:46 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=mavericks.hostname.xyz,path=/home","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/VolGroup01-LogVol02","fstype":"ext3","host":"mavericks.hostname.xyz","path":"/home"},"Fields":{"free":151598505984,"inodes_free":133300363,"inodes_total":133562368,"inodes_used":262005,"total":1076941541376,"used":870637568000,"used_percent":85.16991232825805},"Time":"2017-02-22T22:20:45Z"}
kap |
kap | [httpd] 172.18.0.2 - - [22/Feb/2017:22:20:46 +0000]
"POST /write?consistency=&db=main&precision=ns&rp=autogen HTTP/1.1" 204 0
"-" "InfluxDBClient" 28399894-f94d-11e6-80ac-000000000000 357
kap UP | [disk_alert_warn:log2] 2017/02/22 22:20:46 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=fusion.hostname.xyz,path=/","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-root","fstype":"ext3","host":"fusion.hostname.xyz","path":"/"},"Fields":{"free":1118441472,"inodes_free":154449,"inodes_total":245760,"inodes_used":91311,"total":3963305984,"used":2643537920,"used_percent":70.26986712424818},"Time":"2017-02-22T22:20:46Z"}
.
.
.
kap DOWN | [disk_alert_warn:log2] 2017/02/22 22:20:46 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=wiki.hostname.xyz,path=/home","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vg_wikidev-lv_home","fstype":"ext4","host":"wiki.hostname.xyz","path":"/home"},"Fields":{"free":73774489600,"inodes_free":27882718,"inodes_total":28819456,"inodes_used":936738,"total":464559513600,"used":367198433280,"used_percent":83.27006358617717},"Time":"2017-02-22T22:20:46Z"}
kap |
kap | [httpd] 172.18.0.2 - - [22/Feb/2017:22:20:46 +0000]
"POST /write?consistency=&db=main&precision=ns&rp=autogen HTTP/1.1" 204 0
"-" "InfluxDBClient" 284241d0-f94d-11e6-80af-000000000000 352
kap | [disk_alert_warn:log2] 2017/02/22 22:20:46 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=wiki.hostname.xyz,path=/","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vg_wikidev-lv_root","fstype":"ext4","host":"wiki.hostname.xyz","path":"/"},"Fields":{"free":57767124992,"inodes_free":6351092,"inodes_total":6553600,"inodes_used":202508,"total":105555197952,"used":42420019200,"used_percent":42.34078088772119},"Time":"2017-02-22T22:20:46Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:46 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=wiki.hostname.xyz,path=/boot","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"sda1","fstype":"ext4","host":"wiki.hostname.xyz","path":"/boot"},"Fields":{"free":362619904,"inodes_free":127963,"inodes_total":128016,"inodes_used":53,"total":499355648,"used":110521344,"used_percent":23.359059153515187},"Time":"2017-02-22T22:20:46Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:46 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=wiki.hostname.xyz,path=/home","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vg_wikidev-lv_home","fstype":"ext4","host":"wiki.hostname.xyz","path":"/home"},"Fields":{"free":73774489600,"inodes_free":27882718,"inodes_total":28819456,"inodes_used":936738,"total":464559513600,"used":367198433280,"used_percent":83.27006358617717},"Time":"2017-02-22T22:20:46Z"}
kap |
kap | [httpd] 172.18.0.2 - - [22/Feb/2017:22:20:46 +0000]
"POST /write?consistency=&db=main&precision=ns&rp=autogen HTTP/1.1" 204 0
"-" "InfluxDBClient" 285f27f4-f94d-11e6-80b0-000000000000 1608
kap UP | [disk_alert_warn:log2] 2017/02/22 22:20:46 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=name-chat.hostname.xyz,path=/","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"rootfs","fstype":"rootfs","host":"name-chat.hostname.xyz","path":"/"},"Fields":{"free":10452328448,"inodes_free":10470655,"inodes_total":10484736,"inodes_used":14081,"total":10725883904,"used":273555456,"used_percent":2.5504234284876333},"Time":"2017-02-22T22:20:47Z"}
.
.
.
kap DOWN | [disk_alert_warn:log2] 2017/02/22 22:20:48 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=spork.hostname.xyz,path=/home","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-home","fstype":"ext3","host":"spork.hostname.xyz","path":"/home"},"Fields":{"free":110653030400,"inodes_free":42103075,"inodes_total":43679744,"inodes_used":1576669,"total":704386506752,"used":557976289280,"used_percent":83.45076604583276},"Time":"2017-02-22T22:21:00Z"}
kap |
kap UP | [disk_alert_warn:log2] 2017/02/22 22:20:48 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=spork.hostname.xyz,path=/tmp","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-temp","fstype":"ext3","host":"spork.hostname.xyz","path":"/tmp"},"Fields":{"free":881258496,"inodes_free":65490,"inodes_total":65536,"inodes_used":46,"total":1056858112,"used":121913344,"used_percent":12.152787701855745},"Time":"2017-02-22T22:21:00Z"}
kap |
.
.
.
kap DOWN | [disk_alert_warn:log2] 2017/02/22 22:20:50 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=name-fusion.hostname.xyz,path=/var","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-var","fstype":"ext3","host":"name-fusion.hostname.xyz","path":"/var"},"Fields":{"free":1289412608,"inodes_free":636979,"inodes_total":655360,"inodes_used":18381,"total":10568916992,"used":8742641664,"used_percent":87.14707304167185},"Time":"2017-02-22T22:21:10Z"}
kap |
kap UP | [disk_alert_warn:log2] 2017/02/22 22:20:50 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=name-fusion.hostname.xyz,path=/var/log","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-varlog","fstype":"ext3","host":"name-fusion.hostname.xyz","path":"/var/log"},"Fields":{"free":799748096,"inodes_free":65316,"inodes_total":65536,"inodes_used":220,"total":1056858112,"used":203423744,"used_percent":20.278055651960887},"Time":"2017-02-22T22:21:10Z"}
.
.
.
kap DOWN | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=chat.hostname.xyz,path=/rootfs/etc","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"fstype":"xfs","host":"chat.hostname.xyz","path":"/rootfs/etc"},"Fields":{"free":13547388928,"inodes_free":52926628,"inodes_total":53033344,"inodes_used":106716,"total":122295418880,"used":108748029952,"used_percent":88.9224068635857},"Time":"2017-02-22T22:21:00Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=chat.hostname.xyz,path=/etc/resolv.conf","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"fstype":"xfs","host":"chat.hostname.xyz","path":"/etc/resolv.conf"},"Fields":{"free":13547388928,"inodes_free":52926628,"inodes_total":53033344,"inodes_used":106716,"total":122295418880,"used":108748029952,"used_percent":88.9224068635857},"Time":"2017-02-22T22:21:00Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=chat.hostname.xyz,path=/etc/hostname","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"fstype":"xfs","host":"chat.hostname.xyz","path":"/etc/hostname"},"Fields":{"free":13547388928,"inodes_free":52926628,"inodes_total":53033344,"inodes_used":106716,"total":122295418880,"used":108748029952,"used_percent":88.9224068635857},"Time":"2017-02-22T22:21:00Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=chat.hostname.xyz,path=/etc/hosts","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"fstype":"xfs","host":"chat.hostname.xyz","path":"/etc/hosts"},"Fields":{"free":13547388928,"inodes_free":52926628,"inodes_total":53033344,"inodes_used":106716,"total":122295418880,"used":108748029952,"used_percent":88.9224068635857},"Time":"2017-02-22T22:21:00Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=chat.hostname.xyz,path=/etc/telegraf/telegraf.conf","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"fstype":"xfs","host":"chat.hostname.xyz","path":"/etc/telegraf/telegraf.conf"},"Fields":{"free":13547388928,"inodes_free":52926628,"inodes_total":53033344,"inodes_used":106716,"total":122295418880,"used":108748029952,"used_percent":88.9224068635857},"Time":"2017-02-22T22:21:00Z"}
kap |
kap | [httpd] 172.18.0.2 - - [22/Feb/2017:22:20:51 +0000]
"POST /write?consistency=&db=main&precision=ns&rp=autogen HTTP/1.1" 204 0
"-" "InfluxDBClient" 2b7cc1e0-f94d-11e6-80d4-000000000000 470
kap | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyz,path=/","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-root","fstype":"ext3","host":"jira.hostname.xyz","path":"/"},"Fields":{"free":15286771712,"inodes_free":1373362,"inodes_total":1495552,"inodes_used":122190,"total":24210894848,"used":7703351296,"used_percent":33.50722087619724},"Time":"2017-02-22T22:20:50Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyz,path=/boot","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"sda1","fstype":"ext3","host":"jira.hostname.xyz","path":"/boot"},"Fields":{"free":377835520,"inodes_free":32715,"inodes_total":32768,"inodes_used":53,"total":528424960,"used":123748352,"used_percent":24.671517348946978},"Time":"2017-02-22T22:20:50Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyz,path=/home","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-home","fstype":"ext3","host":"jira.hostname.xyz","path":"/home"},"Fields":{"free":12615372800,"inodes_free":884437,"inodes_total":884736,"inodes_used":299,"total":14268055552,"used":928071680,"used_percent":6.85255277097721},"Time":"2017-02-22T22:20:50Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyz,path=/tmp","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-temp","fstype":"ext3","host":"jira.hostname.xyz","path":"/tmp"},"Fields":{"free":880750592,"inodes_free":65424,"inodes_total":65536,"inodes_used":112,"total":1056858112,"used":122421248,"used_percent":12.20341751219811},"Time":"2017-02-22T22:20:50Z"}
kap |
kap | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyz,path=/var","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-var","fstype":"ext3","host":"jira.hostname.xyz","path":"/var"},"Fields":{"free":26661879808,"inodes_free":4707430,"inodes_total":4792320,"inodes_used":84890,"total":77218316288,"used":46634868736,"used_percent":63.62474415629107},"Time":"2017-02-22T22:20:50Z"}
kap |
kap DOWN | [disk_alert_warn:log2] 2017/02/22 22:20:51 I! LOGGER
{"Name":"disk","Database":"main","RetentionPolicy":"autogen","Group":"host=jira.hostname.xyz,path=/var/log","Dimensions":{"ByName":false,"TagNames":["host","path"]},"Tags":{"device":"mapper/vgroup1-varlog","fstype":"ext3","host":"jira.hostname.xyz","path":"/var/log"},"Fields":{"free":206041088,"inodes_free":65441,"inodes_total":65536,"inodes_used":95,"total":1056858112,"used":797130752,"used_percent":79.4610375028071},"Time":"2017-02-22T22:20:50Z"}












Here is my log. I edited down to just the entries that fired alerts as
there was a lot to sift through. I marked which entries fired the DOWN
alert and the lines that triggered recoveries. The last two alerts fired
DOWN but never fired recoveries.

Hmm, I see. Could you put a log node before the alert node and share the
logs along with the logs for triggered alerts after startup?
Also this may be a bug with the most recent alerting system. Do you get
the same behavior if you configure slack directly in the TICKscript instead
of via the topic handler?
On Wednesday, February 22, 2017 at 12:23:53 PM UTC-7, Archie Archbold

Post by Archie Archbold
Thanks so much for the reply. I do want the recovery alerts, but the
problem is that when I start kapacitor, the task sees *any* server/path
in an up status as a recovery of a *different* server/path's down
status. So if 7 server/paths are in a down status at the time of start-up I
get 7 down alerts (expected) but they are immediately followed by 7
recovery messages from different server/paths. Please let me know if I am
not being clear enough.
On Wednesday, February 22, 2017 at 11:10:09 AM UTC-8,

If you want to ignore the OK alerts use the `.noRecoveries` property of
the alert node. This will suppress the OK alerts.

Post by Archie Archbold
Hey all. Pretty new to TICK but I have a problem that I can't wrap my
head around.
I am monitoring multiple servers all sending data to one influxdb
database and using the 'host' tag to separate the servers in the DB
My 'disk' measurement is taking in mulitiple disk paths from the
servers (HOSTS) which each have a respective 'PATH' tag.
So basically each server is assigned a HOST tag and each HOST has
multiple PATH tags.
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of a
HOST's PATH if that path is within the alerting Lambda.
PROBLEM: When I start the kapacitor service, it looks like it's sensing
a state change any time it sees another host/path with a opposite status.
Host: host1 Path: /path1 Status: UP
Host: host1 Path: /path2 Status: DOWN
Host: host1 Path: /path3 Status: UP
Host: host2 Path: /path1 Status: DOWN
Host: host2 Path: /path2 Status: UP
These alerts happen once for each host/path combination and then the
service performs as expected, alerting properly when lambda is achieved.
The result of this is that I receive a slew of up/down alerts every
time I restart the kapacitor service
var data = stream
|from()
.measurement('disk')
.groupBy('host','path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
|from()
.measurement('disk')
.groupBy('host', 'path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs"
crits_triggered="0" infos_triggered="0" oks_triggered="7"
warns_triggered="7" ];
}
As you can see, I get 7 oks triggered (for host/path groups that are
not in alert range) and 7 warns triggered (for the 7 host/path groups that
are within the alert range) upon start up.
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups that
are over 80 but why follow it with an alert about the ok groups?
MORE INFO: When I raise the lambda to 90% (out of range for all
host/paths) I get no alerts at all (which is expected)
Thanks to anyone who can help me understand this

[Archie Archbold]

Here are the logs from the alerts triggered:
{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:mavericks.sd.spawar.navy.mil Path: /home USED PERCENT:
85.16991232825805","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=mavericks.sd.spawar.navy.mil,path=/home\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/VolGroup01-LogVol02\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;mavericks.sd.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/home\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:151598505984,\u0026#34;inodes_free\u0026#34;:133300363,\u0026#34;inodes_total\u0026#34;:133562368,\u0026#34;inodes_used\u0026#34;:262005,\u0026#34;total\u0026#34;:1076941541376,\u0026#34;used\u0026#34;:870637568000,\u0026#34;used_percent\u0026#34;:85.16991232825805},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:05Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:mavericks.sd.spawar.navy.mil Path: /home USED PERCENT:
85.16991232825805\u0026#34;}\n","time":"2017-02-22T23:27:05Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/VolGroup01-LogVol02","fstype":"ext3","host":"mavericks.sd.spawar.navy.mil","path":"/home"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:05Z",151598505984,133300363,133562368,262005,1076941541376,870637568000,85.16991232825805]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:fusion.spawar.navy.mil Path: / USED PERCENT:
70.27302461097585","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=fusion.spawar.navy.mil,path=/\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-root\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;fusion.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:1118322688,\u0026#34;inodes_free\u0026#34;:154449,\u0026#34;inodes_total\u0026#34;:245760,\u0026#34;inodes_used\u0026#34;:91311,\u0026#34;total\u0026#34;:3963305984,\u0026#34;used\u0026#34;:2643656704,\u0026#34;used_percent\u0026#34;:70.27302461097585},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:06Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:fusion.spawar.navy.mil Path: / USED PERCENT:
70.27302461097585\u0026#34;}\n","time":"2017-02-22T23:27:06Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-root","fstype":"ext3","host":"fusion.spawar.navy.mil","path":"/"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:06Z",1118322688,154449,245760,91311,3963305984,2643656704,70.27302461097585]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:wiki.spawar.navy.mil Path: /home USED PERCENT:
83.31287638084197","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=wiki.spawar.navy.mil,path=/home\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vg_wikidev-lv_home\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext4\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;wiki.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/home\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:73585696768,\u0026#34;inodes_free\u0026#34;:27882405,\u0026#34;inodes_total\u0026#34;:28819456,\u0026#34;inodes_used\u0026#34;:937051,\u0026#34;total\u0026#34;:464559513600,\u0026#34;used\u0026#34;:367387226112,\u0026#34;used_percent\u0026#34;:83.31287638084197},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:06Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:wiki.spawar.navy.mil Path: /home USED PERCENT:
83.31287638084197\u0026#34;}\n","time":"2017-02-22T23:27:06Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vg_wikidev-lv_home","fstype":"ext4","host":"wiki.spawar.navy.mil","path":"/home"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:06Z",73585696768,27882405,28819456,937051,464559513600,367387226112,83.31287638084197]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:navsea-chat.sd.spawar.navy.mil Path: / USED PERCENT:
2.5504234284876333","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=navsea-chat.sd.spawar.navy.mil,path=/\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;rootfs\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;rootfs\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;navsea-chat.sd.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:10452328448,\u0026#34;inodes_free\u0026#34;:10470655,\u0026#34;inodes_total\u0026#34;:10484736,\u0026#34;inodes_used\u0026#34;:14081,\u0026#34;total\u0026#34;:10725883904,\u0026#34;used\u0026#34;:273555456,\u0026#34;used_percent\u0026#34;:2.5504234284876333},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:07Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:navsea-chat.sd.spawar.navy.mil Path: / USED PERCENT:
2.5504234284876333\u0026#34;}\n","time":"2017-02-22T23:27:07Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"rootfs","fstype":"rootfs","host":"navsea-chat.sd.spawar.navy.mil","path":"/"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:07Z",10452328448,10470655,10484736,14081,10725883904,273555456,2.5504234284876333]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:spork.spawar.navy.mil Path: /home USED PERCENT:
83.46179768411558","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=spork.spawar.navy.mil,path=/home\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-home\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;spork.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/home\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:110579269632,\u0026#34;inodes_free\u0026#34;:42103018,\u0026#34;inodes_total\u0026#34;:43679744,\u0026#34;inodes_used\u0026#34;:1576726,\u0026#34;total\u0026#34;:704386506752,\u0026#34;used\u0026#34;:558050050048,\u0026#34;used_percent\u0026#34;:83.46179768411558},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:20Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:spork.spawar.navy.mil Path: /home USED PERCENT:
83.46179768411558\u0026#34;}\n","time":"2017-02-22T23:27:20Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-home","fstype":"ext3","host":"spork.spawar.navy.mil","path":"/home"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:20Z",110579269632,42103018,43679744,1576726,704386506752,558050050048,83.46179768411558]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:spork.spawar.navy.mil Path: /tmp USED PERCENT:
12.152787701855745","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=spork.spawar.navy.mil,path=/tmp\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-temp\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;spork.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/tmp\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:881258496,\u0026#34;inodes_free\u0026#34;:65490,\u0026#34;inodes_total\u0026#34;:65536,\u0026#34;inodes_used\u0026#34;:46,\u0026#34;total\u0026#34;:1056858112,\u0026#34;used\u0026#34;:121913344,\u0026#34;used_percent\u0026#34;:12.152787701855745},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:20Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:spork.spawar.navy.mil Path: /tmp USED PERCENT:
12.152787701855745\u0026#34;}\n","time":"2017-02-22T23:27:20Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-temp","fstype":"ext3","host":"spork.spawar.navy.mil","path":"/tmp"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:20Z",881258496,65490,65536,46,1056858112,121913344,12.152787701855745]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:navsea-fusion.sd.spawar.navy.mil Path: /var USED PERCENT:
87.1589951462336","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=navsea-fusion.sd.spawar.navy.mil,path=/var\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-var\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;navsea-fusion.sd.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/var\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:1288216576,\u0026#34;inodes_free\u0026#34;:636548,\u0026#34;inodes_total\u0026#34;:655360,\u0026#34;inodes_used\u0026#34;:18812,\u0026#34;total\u0026#34;:10568916992,\u0026#34;used\u0026#34;:8743837696,\u0026#34;used_percent\u0026#34;:87.1589951462336},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:30Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:navsea-fusion.sd.spawar.navy.mil Path: /var USED
PERCENT:
87.1589951462336\u0026#34;}\n","time":"2017-02-22T23:27:30Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-var","fstype":"ext3","host":"navsea-fusion.sd.spawar.navy.mil","path":"/var"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:30Z",1288216576,636548,655360,18812,10568916992,8743837696,87.1589951462336]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:navsea-fusion.sd.spawar.navy.mil Path: /var/log USED PERCENT:
20.560194353142926","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=navsea-fusion.sd.spawar.navy.mil,path=/var/log\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-varlog\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;navsea-fusion.sd.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/var/log\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:796917760,\u0026#34;inodes_free\u0026#34;:65316,\u0026#34;inodes_total\u0026#34;:65536,\u0026#34;inodes_used\u0026#34;:220,\u0026#34;total\u0026#34;:1056858112,\u0026#34;used\u0026#34;:206254080,\u0026#34;used_percent\u0026#34;:20.560194353142926},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:30Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:navsea-fusion.sd.spawar.navy.mil Path: /var/log USED
PERCENT:
20.560194353142926\u0026#34;}\n","time":"2017-02-22T23:27:30Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-varlog","fstype":"ext3","host":"navsea-fusion.sd.spawar.navy.mil","path":"/var/log"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:30Z",796917760,65316,65536,220,1056858112,206254080,20.560194353142926]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:chat.spawar.navy.mil Path: /rootfs/etc USED PERCENT:
88.9189638171997","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=chat.spawar.navy.mil,path=/rootfs/etc\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;fstype\u0026#34;:\u0026#34;xfs\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;chat.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/rootfs/etc\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:13551599616,\u0026#34;inodes_free\u0026#34;:52943076,\u0026#34;inodes_total\u0026#34;:53049792,\u0026#34;inodes_used\u0026#34;:106716,\u0026#34;total\u0026#34;:122295418880,\u0026#34;used\u0026#34;:108743819264,\u0026#34;used_percent\u0026#34;:88.9189638171997},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:20Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:chat.spawar.navy.mil Path: /rootfs/etc USED PERCENT:
88.9189638171997\u0026#34;}\n","time":"2017-02-22T23:27:20Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"fstype":"xfs","host":"chat.spawar.navy.mil","path":"/rootfs/etc"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:20Z",13551599616,52943076,53049792,106716,122295418880,108743819264,88.9189638171997]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:jira.spawar.navy.mil Path: / USED PERCENT:
33.5491961887984","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=jira.spawar.navy.mil,path=/\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-root\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;jira.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:15277121536,\u0026#34;inodes_free\u0026#34;:1373361,\u0026#34;inodes_total\u0026#34;:1495552,\u0026#34;inodes_used\u0026#34;:122191,\u0026#34;total\u0026#34;:24210894848,\u0026#34;used\u0026#34;:7713001472,\u0026#34;used_percent\u0026#34;:33.5491961887984},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:10Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:jira.spawar.navy.mil Path: / USED PERCENT:
33.5491961887984\u0026#34;}\n","time":"2017-02-22T23:27:10Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-root","fstype":"ext3","host":"jira.spawar.navy.mil","path":"/"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:10Z",15277121536,1373361,1495552,122191,24210894848,7713001472,33.5491961887984]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:jira.spawar.navy.mil Path: /var/log USED PERCENT:
82.31917195761795","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=jira.spawar.navy.mil,path=/var/log\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-varlog\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;jira.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/var/log\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:177369088,\u0026#34;inodes_free\u0026#34;:65441,\u0026#34;inodes_total\u0026#34;:65536,\u0026#34;inodes_used\u0026#34;:95,\u0026#34;total\u0026#34;:1056858112,\u0026#34;used\u0026#34;:825802752,\u0026#34;used_percent\u0026#34;:82.31917195761795},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:10Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:jira.spawar.navy.mil Path: /var/log USED PERCENT:
82.31917195761795\u0026#34;}\n","time":"2017-02-22T23:27:10Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-varlog","fstype":"ext3","host":"jira.spawar.navy.mil","path":"/var/log"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:10Z",177369088,65441,65536,95,1056858112,825802752,82.31917195761795]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:jira.spawar.navy.mil Path: /var/log/audit USED PERCENT:
5.053099187894773","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=jira.spawar.navy.mil,path=/var/log/audit\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-audit\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;jira.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/var/log/audit\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:1904992256,\u0026#34;inodes_free\u0026#34;:131056,\u0026#34;inodes_total\u0026#34;:131072,\u0026#34;inodes_used\u0026#34;:16,\u0026#34;total\u0026#34;:2113748992,\u0026#34;used\u0026#34;:101384192,\u0026#34;used_percent\u0026#34;:5.053099187894773},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:10Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:jira.spawar.navy.mil Path: /var/log/audit USED
PERCENT:
5.053099187894773\u0026#34;}\n","time":"2017-02-22T23:27:10Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-audit","fstype":"ext3","host":"jira.spawar.navy.mil","path":"/var/log/audit"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:10Z",1904992256,131056,131072,16,2113748992,101384192,5.053099187894773]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:calendar.spawar.navy.mil Path: / USED PERCENT:
83.05497801561063","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=calendar.spawar.navy.mil,path=/\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-root\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;calendar.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:610897920,\u0026#34;inodes_free\u0026#34;:146007,\u0026#34;inodes_total\u0026#34;:235712,\u0026#34;inodes_used\u0026#34;:89705,\u0026#34;total\u0026#34;:3798114304,\u0026#34;used\u0026#34;:2994278400,\u0026#34;used_percent\u0026#34;:83.05497801561063},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:18Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:calendar.spawar.navy.mil Path: / USED PERCENT:
83.05497801561063\u0026#34;}\n","time":"2017-02-22T23:27:18Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-root","fstype":"ext3","host":"calendar.spawar.navy.mil","path":"/"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:18Z",610897920,146007,235712,89705,3798114304,2994278400,83.05497801561063]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:calendar.spawar.navy.mil Path: /boot USED PERCENT:
22.332737205713027","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=calendar.spawar.navy.mil,path=/boot\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;sda1\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;calendar.spawar.navy.mil\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/boot\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:389566464,\u0026#34;inodes_free\u0026#34;:32717,\u0026#34;inodes_total\u0026#34;:32768,\u0026#34;inodes_used\u0026#34;:51,\u0026#34;total\u0026#34;:528424960,\u0026#34;used\u0026#34;:112017408,\u0026#34;used_percent\u0026#34;:22.332737205713027},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:18Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:calendar.spawar.navy.mil Path: /boot USED PERCENT:
22.332737205713027\u0026#34;}\n","time":"2017-02-22T23:27:18Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"sda1","fstype":"ext3","host":"calendar.spawar.navy.mil","path":"/boot"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:18Z",389566464,32717,32768,51,528424960,112017408,22.332737205713027]]}],"Messages":null,"Err":null}}

Hmm, I see. Could you put a log node before the alert node and share the
logs along with the logs for triggered alerts after startup?
Also this may be a bug with the most recent alerting system. Do you get
the same behavior if you configure slack directly in the TICKscript instead
of via the topic handler?
On Wednesday, February 22, 2017 at 12:23:53 PM UTC-7, Archie Archbold

Post by Archie Archbold
Thanks so much for the reply. I do want the recovery alerts, but the
problem is that when I start kapacitor, the task sees *any* server/path
in an up status as a recovery of a *different* server/path's down
status. So if 7 server/paths are in a down status at the time of start-up I
get 7 down alerts (expected) but they are immediately followed by 7
recovery messages from different server/paths. Please let me know if I am
not being clear enough.
On Wednesday, February 22, 2017 at 11:10:09 AM UTC-8,

If you want to ignore the OK alerts use the `.noRecoveries` property of
the alert node. This will suppress the OK alerts.

Post by Archie Archbold
Hey all. Pretty new to TICK but I have a problem that I can't wrap my
head around.
I am monitoring multiple servers all sending data to one influxdb
database and using the 'host' tag to separate the servers in the DB
My 'disk' measurement is taking in mulitiple disk paths from the
servers (HOSTS) which each have a respective 'PATH' tag.
So basically each server is assigned a HOST tag and each HOST has
multiple PATH tags.
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of a
HOST's PATH if that path is within the alerting Lambda.
PROBLEM: When I start the kapacitor service, it looks like it's sensing
a state change any time it sees another host/path with a opposite status.
Host: host1 Path: /path1 Status: UP
Host: host1 Path: /path2 Status: DOWN
Host: host1 Path: /path3 Status: UP
Host: host2 Path: /path1 Status: DOWN
Host: host2 Path: /path2 Status: UP
These alerts happen once for each host/path combination and then the
service performs as expected, alerting properly when lambda is achieved.
The result of this is that I receive a slew of up/down alerts every
time I restart the kapacitor service
var data = stream
|from()
.measurement('disk')
.groupBy('host','path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
|from()
.measurement('disk')
.groupBy('host', 'path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs"
crits_triggered="0" infos_triggered="0" oks_triggered="7"
warns_triggered="7" ];
}
As you can see, I get 7 oks triggered (for host/path groups that are
not in alert range) and 7 warns triggered (for the 7 host/path groups that
are within the alert range) upon start up.
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups that
are over 80 but why follow it with an alert about the ok groups?
MORE INFO: When I raise the lambda to 90% (out of range for all
host/paths) I get no alerts at all (which is expected)
Thanks to anyone who can help me understand this

[Archie Archbold]

Here are the logs for the triggered alerts:
{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:mavericks.hostname.xyz Path: /home USED PERCENT:
85.16991232825805","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=mavericks.hostname.xyz,path=/home\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/VolGroup01-LogVol02\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;mavericks.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/home\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:151598505984,\u0026#34;inodes_free\u0026#34;:133300363,\u0026#34;inodes_total\u0026#34;:133562368,\u0026#34;inodes_used\u0026#34;:262005,\u0026#34;total\u0026#34;:1076941541376,\u0026#34;used\u0026#34;:870637568000,\u0026#34;used_percent\u0026#34;:85.16991232825805},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:05Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:mavericks.hostname.xyz Path: /home USED PERCENT:
85.16991232825805\u0026#34;}\n","time":"2017-02-22T23:27:05Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/VolGroup01-LogVol02","fstype":"ext3","host":"mavericks.hostname.xyz","path":"/home"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:05Z",151598505984,133300363,133562368,262005,1076941541376,870637568000,85.16991232825805]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:fusion.hostname.xyz Path: / USED PERCENT:
70.27302461097585","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=fusion.hostname.xyz,path=/\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-root\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;fusion.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:1118322688,\u0026#34;inodes_free\u0026#34;:154449,\u0026#34;inodes_total\u0026#34;:245760,\u0026#34;inodes_used\u0026#34;:91311,\u0026#34;total\u0026#34;:3963305984,\u0026#34;used\u0026#34;:2643656704,\u0026#34;used_percent\u0026#34;:70.27302461097585},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:06Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:fusion.hostname.xyz Path: / USED PERCENT:
70.27302461097585\u0026#34;}\n","time":"2017-02-22T23:27:06Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-root","fstype":"ext3","host":"fusion.hostname.xyz","path":"/"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:06Z",1118322688,154449,245760,91311,3963305984,2643656704,70.27302461097585]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:wiki.hostname.xyz Path: /home USED PERCENT:
83.31287638084197","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=wiki.hostname.xyz,path=/home\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vg_wikidev-lv_home\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext4\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;wiki.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/home\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:73585696768,\u0026#34;inodes_free\u0026#34;:27882405,\u0026#34;inodes_total\u0026#34;:28819456,\u0026#34;inodes_used\u0026#34;:937051,\u0026#34;total\u0026#34;:464559513600,\u0026#34;used\u0026#34;:367387226112,\u0026#34;used_percent\u0026#34;:83.31287638084197},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:06Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:wiki.hostname.xyz Path: /home USED PERCENT:
83.31287638084197\u0026#34;}\n","time":"2017-02-22T23:27:06Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vg_wikidev-lv_home","fstype":"ext4","host":"wiki.hostname.xyz","path":"/home"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:06Z",73585696768,27882405,28819456,937051,464559513600,367387226112,83.31287638084197]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:name-chat.hostname.xyz Path: / USED PERCENT:
2.5504234284876333","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=name-chat.hostname.xyz,path=/\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;rootfs\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;rootfs\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;name-chat.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:10452328448,\u0026#34;inodes_free\u0026#34;:10470655,\u0026#34;inodes_total\u0026#34;:10484736,\u0026#34;inodes_used\u0026#34;:14081,\u0026#34;total\u0026#34;:10725883904,\u0026#34;used\u0026#34;:273555456,\u0026#34;used_percent\u0026#34;:2.5504234284876333},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:07Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:name-chat.hostname.xyz Path: / USED PERCENT:
2.5504234284876333\u0026#34;}\n","time":"2017-02-22T23:27:07Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"rootfs","fstype":"rootfs","host":"name-chat.hostname.xyz","path":"/"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:07Z",10452328448,10470655,10484736,14081,10725883904,273555456,2.5504234284876333]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:spork.hostname.xyz Path: /home USED PERCENT:
83.46179768411558","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=spork.hostname.xyz,path=/home\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-home\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;spork.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/home\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:110579269632,\u0026#34;inodes_free\u0026#34;:42103018,\u0026#34;inodes_total\u0026#34;:43679744,\u0026#34;inodes_used\u0026#34;:1576726,\u0026#34;total\u0026#34;:704386506752,\u0026#34;used\u0026#34;:558050050048,\u0026#34;used_percent\u0026#34;:83.46179768411558},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:20Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:spork.hostname.xyz Path: /home USED PERCENT:
83.46179768411558\u0026#34;}\n","time":"2017-02-22T23:27:20Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-home","fstype":"ext3","host":"spork.hostname.xyz","path":"/home"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:20Z",110579269632,42103018,43679744,1576726,704386506752,558050050048,83.46179768411558]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:spork.hostname.xyz Path: /tmp USED PERCENT:
12.152787701855745","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=spork.hostname.xyz,path=/tmp\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-temp\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;spork.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/tmp\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:881258496,\u0026#34;inodes_free\u0026#34;:65490,\u0026#34;inodes_total\u0026#34;:65536,\u0026#34;inodes_used\u0026#34;:46,\u0026#34;total\u0026#34;:1056858112,\u0026#34;used\u0026#34;:121913344,\u0026#34;used_percent\u0026#34;:12.152787701855745},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:20Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:spork.hostname.xyz Path: /tmp USED PERCENT:
12.152787701855745\u0026#34;}\n","time":"2017-02-22T23:27:20Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-temp","fstype":"ext3","host":"spork.hostname.xyz","path":"/tmp"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:20Z",881258496,65490,65536,46,1056858112,121913344,12.152787701855745]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:name-fusion.hostname.xyz Path: /var USED PERCENT:
87.1589951462336","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=name-fusion.hostname.xyz,path=/var\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-var\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;name-fusion.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/var\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:1288216576,\u0026#34;inodes_free\u0026#34;:636548,\u0026#34;inodes_total\u0026#34;:655360,\u0026#34;inodes_used\u0026#34;:18812,\u0026#34;total\u0026#34;:10568916992,\u0026#34;used\u0026#34;:8743837696,\u0026#34;used_percent\u0026#34;:87.1589951462336},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:30Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:name-fusion.hostname.xyz Path: /var USED PERCENT:
87.1589951462336\u0026#34;}\n","time":"2017-02-22T23:27:30Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-var","fstype":"ext3","host":"name-fusion.hostname.xyz","path":"/var"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:30Z",1288216576,636548,655360,18812,10568916992,8743837696,87.1589951462336]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:name-fusion.hostname.xyz Path: /var/log USED PERCENT:
20.560194353142926","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=name-fusion.hostname.xyz,path=/var/log\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-varlog\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;name-fusion.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/var/log\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:796917760,\u0026#34;inodes_free\u0026#34;:65316,\u0026#34;inodes_total\u0026#34;:65536,\u0026#34;inodes_used\u0026#34;:220,\u0026#34;total\u0026#34;:1056858112,\u0026#34;used\u0026#34;:206254080,\u0026#34;used_percent\u0026#34;:20.560194353142926},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:30Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:name-fusion.hostname.xyz Path: /var/log USED PERCENT:
20.560194353142926\u0026#34;}\n","time":"2017-02-22T23:27:30Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-varlog","fstype":"ext3","host":"name-fusion.hostname.xyz","path":"/var/log"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:30Z",796917760,65316,65536,220,1056858112,206254080,20.560194353142926]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:chat.hostname.xyz Path: /rootfs/etc USED PERCENT:
88.9189638171997","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=chat.hostname.xyz,path=/rootfs/etc\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;fstype\u0026#34;:\u0026#34;xfs\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;chat.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/rootfs/etc\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:13551599616,\u0026#34;inodes_free\u0026#34;:52943076,\u0026#34;inodes_total\u0026#34;:53049792,\u0026#34;inodes_used\u0026#34;:106716,\u0026#34;total\u0026#34;:122295418880,\u0026#34;used\u0026#34;:108743819264,\u0026#34;used_percent\u0026#34;:88.9189638171997},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:20Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:chat.hostname.xyz Path: /rootfs/etc USED PERCENT:
88.9189638171997\u0026#34;}\n","time":"2017-02-22T23:27:20Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"fstype":"xfs","host":"chat.hostname.xyz","path":"/rootfs/etc"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:20Z",13551599616,52943076,53049792,106716,122295418880,108743819264,88.9189638171997]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:jira.hostname.xyz Path: / USED PERCENT:
33.5491961887984","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=jira.hostname.xyz,path=/\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-root\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;jira.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:15277121536,\u0026#34;inodes_free\u0026#34;:1373361,\u0026#34;inodes_total\u0026#34;:1495552,\u0026#34;inodes_used\u0026#34;:122191,\u0026#34;total\u0026#34;:24210894848,\u0026#34;used\u0026#34;:7713001472,\u0026#34;used_percent\u0026#34;:33.5491961887984},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:10Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:jira.hostname.xyz Path: / USED PERCENT:
33.5491961887984\u0026#34;}\n","time":"2017-02-22T23:27:10Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-root","fstype":"ext3","host":"jira.hostname.xyz","path":"/"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:10Z",15277121536,1373361,1495552,122191,24210894848,7713001472,33.5491961887984]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:jira.hostname.xyz Path: /var/log USED PERCENT:
82.31917195761795","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=jira.hostname.xyz,path=/var/log\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-varlog\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;jira.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/var/log\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:177369088,\u0026#34;inodes_free\u0026#34;:65441,\u0026#34;inodes_total\u0026#34;:65536,\u0026#34;inodes_used\u0026#34;:95,\u0026#34;total\u0026#34;:1056858112,\u0026#34;used\u0026#34;:825802752,\u0026#34;used_percent\u0026#34;:82.31917195761795},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:10Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:jira.hostname.xyz Path: /var/log USED PERCENT:
82.31917195761795\u0026#34;}\n","time":"2017-02-22T23:27:10Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-varlog","fstype":"ext3","host":"jira.hostname.xyz","path":"/var/log"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:10Z",177369088,65441,65536,95,1056858112,825802752,82.31917195761795]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:jira.hostname.xyz Path: /var/log/audit USED PERCENT:
5.053099187894773","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=jira.hostname.xyz,path=/var/log/audit\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-audit\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;jira.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/var/log/audit\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:1904992256,\u0026#34;inodes_free\u0026#34;:131056,\u0026#34;inodes_total\u0026#34;:131072,\u0026#34;inodes_used\u0026#34;:16,\u0026#34;total\u0026#34;:2113748992,\u0026#34;used\u0026#34;:101384192,\u0026#34;used_percent\u0026#34;:5.053099187894773},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:10Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:jira.hostname.xyz Path: /var/log/audit USED PERCENT:
5.053099187894773\u0026#34;}\n","time":"2017-02-22T23:27:10Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-audit","fstype":"ext3","host":"jira.hostname.xyz","path":"/var/log/audit"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:10Z",1904992256,131056,131072,16,2113748992,101384192,5.053099187894773]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:calendar.hostname.xyz Path: / USED PERCENT:
83.05497801561063","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=calendar.hostname.xyz,path=/\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;mapper/vgroup1-root\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;calendar.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:610897920,\u0026#34;inodes_free\u0026#34;:146007,\u0026#34;inodes_total\u0026#34;:235712,\u0026#34;inodes_used\u0026#34;:89705,\u0026#34;total\u0026#34;:3798114304,\u0026#34;used\u0026#34;:2994278400,\u0026#34;used_percent\u0026#34;:83.05497801561063},\u0026#34;Level\u0026#34;:\u0026#34;WARNING\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:18Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:calendar.hostname.xyz Path: / USED PERCENT:
83.05497801561063\u0026#34;}\n","time":"2017-02-22T23:27:18Z","duration":0,"level":"WARNING","data":{"Series":[{"name":"disk","tags":{"device":"mapper/vgroup1-root","fstype":"ext3","host":"calendar.hostname.xyz","path":"/"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:18Z",610897920,146007,235712,89705,3798114304,2994278400,83.05497801561063]]}],"Messages":null,"Err":null}}


{"id":"DISK SPACE WARNING","message":"DISK SPACE WARNING
Server:calendar.hostname.xyz Path: /boot USED PERCENT:
22.332737205713027","details":"{\u0026#34;Name\u0026#34;:\u0026#34;disk\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;disk_alert_warn\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=calendar.hostname.xyz,path=/boot\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;device\u0026#34;:\u0026#34;sda1\u0026#34;,\u0026#34;fstype\u0026#34;:\u0026#34;ext3\u0026#34;,\u0026#34;host\u0026#34;:\u0026#34;calendar.hostname.xyz\u0026#34;,\u0026#34;path\u0026#34;:\u0026#34;/boot\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;DISK
SPACE
WARNING\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;free\u0026#34;:389566464,\u0026#34;inodes_free\u0026#34;:32717,\u0026#34;inodes_total\u0026#34;:32768,\u0026#34;inodes_used\u0026#34;:51,\u0026#34;total\u0026#34;:528424960,\u0026#34;used\u0026#34;:112017408,\u0026#34;used_percent\u0026#34;:22.332737205713027},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2017-02-22T23:27:18Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;DISK
SPACE WARNING Server:calendar.hostname.xyz Path: /boot USED PERCENT:
22.332737205713027\u0026#34;}\n","time":"2017-02-22T23:27:18Z","duration":9223372036854775807,"level":"OK","data":{"Series":[{"name":"disk","tags":{"device":"sda1","fstype":"ext3","host":"calendar.hostname.xyz","path":"/boot"},"columns":["time","free","inodes_free","inodes_total","inodes_used","total","used","used_percent"],"values":[["2017-02-22T23:27:18Z",389566464,32717,32768,51,528424960,112017408,22.332737205713027]]}],"Messages":null,"Err":null}}

Hmm, I see. Could you put a log node before the alert node and share the
logs along with the logs for triggered alerts after startup?
Also this may be a bug with the most recent alerting system. Do you get
the same behavior if you configure slack directly in the TICKscript instead
of via the topic handler?
On Wednesday, February 22, 2017 at 12:23:53 PM UTC-7, Archie Archbold

Post by Archie Archbold
Thanks so much for the reply. I do want the recovery alerts, but the
problem is that when I start kapacitor, the task sees *any* server/path
in an up status as a recovery of a *different* server/path's down
status. So if 7 server/paths are in a down status at the time of start-up I
get 7 down alerts (expected) but they are immediately followed by 7
recovery messages from different server/paths. Please let me know if I am
not being clear enough.
On Wednesday, February 22, 2017 at 11:10:09 AM UTC-8,

If you want to ignore the OK alerts use the `.noRecoveries` property of
the alert node. This will suppress the OK alerts.

Post by Archie Archbold
Hey all. Pretty new to TICK but I have a problem that I can't wrap my
head around.
I am monitoring multiple servers all sending data to one influxdb
database and using the 'host' tag to separate the servers in the DB
My 'disk' measurement is taking in mulitiple disk paths from the
servers (HOSTS) which each have a respective 'PATH' tag.
So basically each server is assigned a HOST tag and each HOST has
multiple PATH tags.
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of a
HOST's PATH if that path is within the alerting Lambda.
PROBLEM: When I start the kapacitor service, it looks like it's sensing
a state change any time it sees another host/path with a opposite status.
Host: host1 Path: /path1 Status: UP
Host: host1 Path: /path2 Status: DOWN
Host: host1 Path: /path3 Status: UP
Host: host2 Path: /path1 Status: DOWN
Host: host2 Path: /path2 Status: UP
These alerts happen once for each host/path combination and then the
service performs as expected, alerting properly when lambda is achieved.
The result of this is that I receive a slew of up/down alerts every
time I restart the kapacitor service
var data = stream
|from()
.measurement('disk')
.groupBy('host','path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
|from()
.measurement('disk')
.groupBy('host', 'path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs"
crits_triggered="0" infos_triggered="0" oks_triggered="7"
warns_triggered="7" ];
}
As you can see, I get 7 oks triggered (for host/path groups that are
not in alert range) and 7 warns triggered (for the 7 host/path groups that
are within the alert range) upon start up.
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups that
are over 80 but why follow it with an alert about the ok groups?
MORE INFO: When I raise the lambda to 90% (out of range for all
host/paths) I get no alerts at all (which is expected)
Thanks to anyone who can help me understand this

[Archie Archbold]

Interestingly enough, when I add the .noRecoveries() property to the alert
node I only get one DOWN alert even though there are 7 servers that are
within the alert range

If you want to ignore the OK alerts use the `.noRecoveries` property of
the alert node. This will suppress the OK alerts.

Post by Archie Archbold
Hey all. Pretty new to TICK but I have a problem that I can't wrap my
head around.
I am monitoring multiple servers all sending data to one influxdb
database and using the 'host' tag to separate the servers in the DB
My 'disk' measurement is taking in mulitiple disk paths from the servers
(HOSTS) which each have a respective 'PATH' tag.
So basically each server is assigned a HOST tag and each HOST has
multiple PATH tags.
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of a
HOST's PATH if that path is within the alerting Lambda.
PROBLEM: When I start the kapacitor service, it looks like it's sensing a
state change any time it sees another host/path with a opposite status.
Host: host1 Path: /path1 Status: UP
Host: host1 Path: /path2 Status: DOWN
Host: host1 Path: /path3 Status: UP
Host: host2 Path: /path1 Status: DOWN
Host: host2 Path: /path2 Status: UP
These alerts happen once for each host/path combination and then the
service performs as expected, alerting properly when lambda is achieved.
The result of this is that I receive a slew of up/down alerts every time
I restart the kapacitor service
var data = stream
|from()
.measurement('disk')
.groupBy('host','path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
|from()
.measurement('disk')
.groupBy('host', 'path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{
index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs"
crits_triggered="0" infos_triggered="0" oks_triggered="7"
warns_triggered="7" ];
}
As you can see, I get 7 oks triggered (for host/path groups that are not
in alert range) and 7 warns triggered (for the 7 host/path groups that are
within the alert range) upon start up.
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups that
are over 80 but why follow it with an alert about the ok groups?
MORE INFO: When I raise the lambda to 90% (out of range for all
host/paths) I get no alerts at all (which is expected)
Thanks to anyone who can help me understand this

[j***@gmail.com]

Any updates on this?

We're having this same problem. Restart Kapacitor or re-define the task, and we get spammed alerts saying everything is OK (even from hosts which never entered a non-OK state).

Our TICK is pretty simple (and very similar to OP):

stream
|from()
.database('telegraf')
.measurement('disk')
.groupBy('host', 'device')
|alert()
.warn(lambda: "used_percent" >= 80)
.warnReset(lambda: "used_percent" < 80)
.crit(lambda: "used_percent" >= 90)
.critReset(lambda: "used_percent" < 90)
.stateChangesOnly()

Interestingly enough, when I add the .noRecoveries() property to the alert node I only get one DOWN alert even though there are 7 servers that are within the alert range 
If you want to ignore the OK alerts use the `.noRecoveries` property of the alert node. This will suppress the OK alerts.
Hey all. Pretty new to TICK but I have a problem that I can't wrap my head around.
I am monitoring multiple servers all sending data to one influxdb database and using the 'host' tag to separate the servers in the DB
My 'disk' measurement  is taking in mulitiple disk paths from the servers (HOSTS) which each have a respective 'PATH' tag.
So basically each server is assigned a HOST tag and each HOST has multiple PATH tags.
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of a HOST's PATH if that path is within the alerting Lambda. 
PROBLEM: When I start the kapacitor service, it looks like it's sensing a state change any time it sees another host/path with a opposite status.
Host: host1  Path: /path1  Status: UP
Host: host1  Path: /path2  Status: DOWN
Host: host1  Path: /path3  Status: UP
Host: host2  Path: /path1 Status: DOWN
Host: host2  Path: /path2  Status: UP
These alerts happen once for each host/path combination and then the service performs as expected, alerting properly when lambda is achieved.
The result of this is that I receive a slew of up/down alerts every time I restart the kapacitor service
var data = stream
    |from()
        .measurement('disk')
        .groupBy('host','path')       
    |alert()
        .message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{ index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
        .warn(lambda: "used_percent" >= 80)
     .id('DISK SPACE WARNING')
        .email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Error: 
Template: 
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
    |from()
        .measurement('disk')
        .groupBy('host', 'path')
    |alert()
        .message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{ index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
        .warn(lambda: "used_percent" >= 80)
        .id('DISK SPACE WARNING')
        .email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs" crits_triggered="0" infos_triggered="0" oks_triggered="7" warns_triggered="7" ];
}
As you can see, I get 7 oks triggered (for host/path groups that are not in alert range) and 7 warns triggered (for the 7 host/path groups that are within the alert range) upon start up.
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups that are over 80 but why follow it with an alert about the ok groups?
MORE INFO: When I raise the lambda to 90% (out of range for all host/paths) I get no alerts at all (which is expected)
Thanks to anyone who can help me understand this

[j***@gmail.com]

From some limited testing, it seems like the problem is that if a particular host (say, 'host1') has a WARN/CRITICAL for a particular (host1, device1) grouping, as well as an OK for a (host1, device2) grouping, that alerts will be generated for both device1 and device2, even though only device1 is in an alert state.

I've tested this hypothesis on a host that has no groupings in an alert state, and one with just a single grouping in an alert state. The host with no groupings in alert state receives no alerts.

Can anyone make sense of this?

I'm using Kapacitor 1.3.1 BTW

Post by j***@gmail.com
Any updates on this?
We're having this same problem. Restart Kapacitor or re-define the task, and we get spammed alerts saying everything is OK (even from hosts which never entered a non-OK state).
stream
|from()
.database('telegraf')
.measurement('disk')
.groupBy('host', 'device')
|alert()
.warn(lambda: "used_percent" >= 80)
.warnReset(lambda: "used_percent" < 80)
.crit(lambda: "used_percent" >= 90)
.critReset(lambda: "used_percent" < 90)
.stateChangesOnly()

Interestingly enough, when I add the .noRecoveries() property to the alert node I only get one DOWN alert even though there are 7 servers that are within the alert range 
If you want to ignore the OK alerts use the `.noRecoveries` property of the alert node. This will suppress the OK alerts.
Hey all. Pretty new to TICK but I have a problem that I can't wrap my head around.
I am monitoring multiple servers all sending data to one influxdb database and using the 'host' tag to separate the servers in the DB
My 'disk' measurement  is taking in mulitiple disk paths from the servers (HOSTS) which each have a respective 'PATH' tag.
So basically each server is assigned a HOST tag and each HOST has multiple PATH tags.
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of a HOST's PATH if that path is within the alerting Lambda. 
PROBLEM: When I start the kapacitor service, it looks like it's sensing a state change any time it sees another host/path with a opposite status.
Host: host1  Path: /path1  Status: UP
Host: host1  Path: /path2  Status: DOWN
Host: host1  Path: /path3  Status: UP
Host: host2  Path: /path1 Status: DOWN
Host: host2  Path: /path2  Status: UP
These alerts happen once for each host/path combination and then the service performs as expected, alerting properly when lambda is achieved.
The result of this is that I receive a slew of up/down alerts every time I restart the kapacitor service
var data = stream
    |from()
        .measurement('disk')
        .groupBy('host','path')       
    |alert()
        .message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{ index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
        .warn(lambda: "used_percent" >= 80)
     .id('DISK SPACE WARNING')
        .email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Error: 
Template: 
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
    |from()
        .measurement('disk')
        .groupBy('host', 'path')
    |alert()
        .message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{ index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
        .warn(lambda: "used_percent" >= 80)
        .id('DISK SPACE WARNING')
        .email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs" crits_triggered="0" infos_triggered="0" oks_triggered="7" warns_triggered="7" ];
}
As you can see, I get 7 oks triggered (for host/path groups that are not in alert range) and 7 warns triggered (for the 7 host/path groups that are within the alert range) upon start up.
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups that are over 80 but why follow it with an alert about the ok groups?
MORE INFO: When I raise the lambda to 90% (out of range for all host/paths) I get no alerts at all (which is expected)
Thanks to anyone who can help me understand this

[j***@gmail.com]

In case anyone else has this problem, the solution involves setting the alert ID to something unique.

AFAIK, this isn't documented very well, but each alert has an associated identifier which is used internally to track the alert state. If you use groupBy to split out a metric stream into multiple alerts, but use the same ID, then a change in the alert condition for any metric stream will cause the alert to change state.

So for example, in the case of disk usage, you would do something like:

|groupBy('host', 'path')
|alert()
.id('{{ .Tags.host }}/{{ .Tags.path }}/disk_usage')

Post by j***@gmail.com
From some limited testing, it seems like the problem is that if a particular host (say, 'host1') has a WARN/CRITICAL for a particular (host1, device1) grouping, as well as an OK for a (host1, device2) grouping, that alerts will be generated for both device1 and device2, even though only device1 is in an alert state.
I've tested this hypothesis on a host that has no groupings in an alert state, and one with just a single grouping in an alert state. The host with no groupings in alert state receives no alerts.
Can anyone make sense of this?
I'm using Kapacitor 1.3.1 BTW

Post by j***@gmail.com
Any updates on this?
We're having this same problem. Restart Kapacitor or re-define the task, and we get spammed alerts saying everything is OK (even from hosts which never entered a non-OK state).
stream
|from()
.database('telegraf')
.measurement('disk')
.groupBy('host', 'device')
|alert()
.warn(lambda: "used_percent" >= 80)
.warnReset(lambda: "used_percent" < 80)
.crit(lambda: "used_percent" >= 90)
.critReset(lambda: "used_percent" < 90)
.stateChangesOnly()

Interestingly enough, when I add the .noRecoveries() property to the alert node I only get one DOWN alert even though there are 7 servers that are within the alert range 
If you want to ignore the OK alerts use the `.noRecoveries` property of the alert node. This will suppress the OK alerts.
Hey all. Pretty new to TICK but I have a problem that I can't wrap my head around.
I am monitoring multiple servers all sending data to one influxdb database and using the 'host' tag to separate the servers in the DB
My 'disk' measurement  is taking in mulitiple disk paths from the servers (HOSTS) which each have a respective 'PATH' tag.
So basically each server is assigned a HOST tag and each HOST has multiple PATH tags.
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of a HOST's PATH if that path is within the alerting Lambda. 
PROBLEM: When I start the kapacitor service, it looks like it's sensing a state change any time it sees another host/path with a opposite status.
Host: host1  Path: /path1  Status: UP
Host: host1  Path: /path2  Status: DOWN
Host: host1  Path: /path3  Status: UP
Host: host2  Path: /path1 Status: DOWN
Host: host2  Path: /path2  Status: UP
These alerts happen once for each host/path combination and then the service performs as expected, alerting properly when lambda is achieved.
The result of this is that I receive a slew of up/down alerts every time I restart the kapacitor service
var data = stream
    |from()
        .measurement('disk')
        .groupBy('host','path')       
    |alert()
        .message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{ index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
        .warn(lambda: "used_percent" >= 80)
     .id('DISK SPACE WARNING')
        .email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Error: 
Template: 
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
    |from()
        .measurement('disk')
        .groupBy('host', 'path')
    |alert()
        .message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{ index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')
        .warn(lambda: "used_percent" >= 80)
        .id('DISK SPACE WARNING')
        .email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs" crits_triggered="0" infos_triggered="0" oks_triggered="7" warns_triggered="7" ];
}
As you can see, I get 7 oks triggered (for host/path groups that are not in alert range) and 7 warns triggered (for the 7 host/path groups that are within the alert range) upon start up.
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups that are over 80 but why follow it with an alert about the ok groups?
MORE INFO: When I raise the lambda to 90% (out of range for all host/paths) I get no alerts at all (which is expected)
Thanks to anyone who can help me understand this

[Archie Archbold]

great. I'll give it a try and post back results.

Post by j***@gmail.com
In case anyone else has this problem, the solution involves setting the
alert ID to something unique.
AFAIK, this isn't documented very well, but each alert has an associated
identifier which is used internally to track the alert state. If you use
groupBy to split out a metric stream into multiple alerts, but use the same
ID, then a change in the alert condition for any metric stream will cause
the alert to change state.
|groupBy('host', 'path')
|alert()
.id('{{ .Tags.host }}/{{ .Tags.path }}/disk_usage')

Post by j***@gmail.com
From some limited testing, it seems like the problem is that if a

particular host (say, 'host1') has a WARN/CRITICAL for a particular (host1,
device1) grouping, as well as an OK for a (host1, device2) grouping, that
alerts will be generated for both device1 and device2, even though only
device1 is in an alert state.

Post by j***@gmail.com
I've tested this hypothesis on a host that has no groupings in an alert

state, and one with just a single grouping in an alert state. The host with
no groupings in alert state receives no alerts.

Post by j***@gmail.com
Can anyone make sense of this?
I'm using Kapacitor 1.3.1 BTW

Post by j***@gmail.com
Any updates on this?
We're having this same problem. Restart Kapacitor or re-define the

task, and we get spammed alerts saying everything is OK (even from hosts
which never entered a non-OK state).

Post by j***@gmail.com

Post by j***@gmail.com
stream
|from()
.database('telegraf')
.measurement('disk')
.groupBy('host', 'device')
|alert()
.warn(lambda: "used_percent" >= 80)
.warnReset(lambda: "used_percent" < 80)
.crit(lambda: "used_percent" >= 90)
.critReset(lambda: "used_percent" < 90)
.stateChangesOnly()
On Wednesday, February 22, 2017 at 7:14:23 PM UTC-5, Archie Archbold

Post by Archie Archbold
Interestingly enough, when I add the .noRecoveries() property to the

alert node I only get one DOWN alert even though there are 7 servers that
are within the alert range

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
On Wednesday, February 22, 2017 at 11:10:09 AM UTC-8,
If you want to ignore the OK alerts use the `.noRecoveries` property

of the alert node. This will suppress the OK alerts.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
On Friday, February 17, 2017 at 3:33:16 PM UTC-7, Archie Archbold
Hey all. Pretty new to TICK but I have a problem that I can't wrap

my head around.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
I am monitoring multiple servers all sending data to one influxdb

database and using the 'host' tag to separate the servers in the DB

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
My 'disk' measurement is taking in mulitiple disk paths from the

servers (HOSTS) which each have a respective 'PATH' tag.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
So basically each server is assigned a HOST tag and each HOST has

multiple PATH tags.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of

a HOST's PATH if that path is within the alerting Lambda.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
PROBLEM: When I start the kapacitor service, it looks like it's

sensing a state change any time it sees another host/path with a opposite
status.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
Host: host1 Path: /path1 Status: UP
Host: host1 Path: /path2 Status: DOWN
Host: host1 Path: /path3 Status: UP
Host: host2 Path: /path1 Status: DOWN
Host: host2 Path: /path2 Status: UP
These alerts happen once for each host/path combination and then the

service performs as expected, alerting properly when lambda is achieved.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
The result of this is that I receive a slew of up/down alerts every

time I restart the kapacitor service

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
var data = stream
|from()
.measurement('disk')
.groupBy('host','path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{

index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
|from()
.measurement('disk')
.groupBy('host', 'path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{

index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs"

crits_triggered="0" infos_triggered="0" oks_triggered="7"
warns_triggered="7" ];

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
}
As you can see, I get 7 oks triggered (for host/path groups that are

not in alert range) and 7 warns triggered (for the 7 host/path groups that
are within the alert range) upon start up.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups

that are over 80 but why follow it with an alert about the ok groups?

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
MORE INFO: When I raise the lambda to 90% (out of range for all

host/paths) I get no alerts at all (which is expected)

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
Thanks to anyone who can help me understand this

[Archie Archbold]

That worked perfectly. In looking at the documentation again, your
explanation makes total sense. Thank you so much for your time and help.

Post by j***@gmail.com
In case anyone else has this problem, the solution involves setting the
alert ID to something unique.
AFAIK, this isn't documented very well, but each alert has an associated
identifier which is used internally to track the alert state. If you use
groupBy to split out a metric stream into multiple alerts, but use the same
ID, then a change in the alert condition for any metric stream will cause
the alert to change state.
|groupBy('host', 'path')
|alert()
.id('{{ .Tags.host }}/{{ .Tags.path }}/disk_usage')

Post by j***@gmail.com
From some limited testing, it seems like the problem is that if a

particular host (say, 'host1') has a WARN/CRITICAL for a particular (host1,
device1) grouping, as well as an OK for a (host1, device2) grouping, that
alerts will be generated for both device1 and device2, even though only
device1 is in an alert state.

Post by j***@gmail.com
I've tested this hypothesis on a host that has no groupings in an alert

state, and one with just a single grouping in an alert state. The host with
no groupings in alert state receives no alerts.

Post by j***@gmail.com
Can anyone make sense of this?
I'm using Kapacitor 1.3.1 BTW

Post by j***@gmail.com
Any updates on this?
We're having this same problem. Restart Kapacitor or re-define the

task, and we get spammed alerts saying everything is OK (even from hosts
which never entered a non-OK state).

Post by j***@gmail.com

Post by j***@gmail.com
stream
|from()
.database('telegraf')
.measurement('disk')
.groupBy('host', 'device')
|alert()
.warn(lambda: "used_percent" >= 80)
.warnReset(lambda: "used_percent" < 80)
.crit(lambda: "used_percent" >= 90)
.critReset(lambda: "used_percent" < 90)
.stateChangesOnly()
On Wednesday, February 22, 2017 at 7:14:23 PM UTC-5, Archie Archbold

Post by Archie Archbold
Interestingly enough, when I add the .noRecoveries() property to the

alert node I only get one DOWN alert even though there are 7 servers that
are within the alert range

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
On Wednesday, February 22, 2017 at 11:10:09 AM UTC-8,
If you want to ignore the OK alerts use the `.noRecoveries` property

of the alert node. This will suppress the OK alerts.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
On Friday, February 17, 2017 at 3:33:16 PM UTC-7, Archie Archbold
Hey all. Pretty new to TICK but I have a problem that I can't wrap

my head around.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
I am monitoring multiple servers all sending data to one influxdb

database and using the 'host' tag to separate the servers in the DB

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
My 'disk' measurement is taking in mulitiple disk paths from the

servers (HOSTS) which each have a respective 'PATH' tag.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
So basically each server is assigned a HOST tag and each HOST has

multiple PATH tags.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
EXPECTED FUNCTIONALITY: kapacitor should alert upon state change of

a HOST's PATH if that path is within the alerting Lambda.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
PROBLEM: When I start the kapacitor service, it looks like it's

sensing a state change any time it sees another host/path with a opposite
status.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
Host: host1 Path: /path1 Status: UP
Host: host1 Path: /path2 Status: DOWN
Host: host1 Path: /path3 Status: UP
Host: host2 Path: /path1 Status: DOWN
Host: host2 Path: /path2 Status: UP
These alerts happen once for each host/path combination and then the

service performs as expected, alerting properly when lambda is achieved.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
The result of this is that I receive a slew of up/down alerts every

time I restart the kapacitor service

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
var data = stream
|from()
.measurement('disk')
.groupBy('host','path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{

index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email($DISK_WARN_GRP)
And the corresponding DOT
ID: disk_alert_warn
Type: stream
Status: enabled
Executing: true
Created: 17 Feb 17 22:27 UTC
Modified: 17 Feb 17 22:27 UTC
LastEnabled: 17 Feb 17 22:27 UTC
Databases Retention Policies: ["main"."autogen"]
var data = stream
|from()
.measurement('disk')
.groupBy('host', 'path')
|alert()
.message('{{ .ID }} Server:{{ index .Tags "host" }} Path: {{

index .Tags "path" }} USED PERCENT: {{ index .Fields "used_percent" }}')

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
.warn(lambda: "used_percent" >= 80)
.id('DISK SPACE WARNING')
.email()
digraph disk_alert_warn {
graph [throughput="38.00 points/s"];
stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="284"];
from1 [avg_exec_time_ns="3.9µs" ];
from1 -> alert2 [processed="284"];
alert2 [alerts_triggered="14" avg_exec_time_ns="72.33µs"

crits_triggered="0" infos_triggered="0" oks_triggered="7"
warns_triggered="7" ];

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
}
As you can see, I get 7 oks triggered (for host/path groups that are

not in alert range) and 7 warns triggered (for the 7 host/path groups that
are within the alert range) upon start up.

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
Then it behaves as normal.
I understand that it should be alerting for the 7 host/path groups

that are over 80 but why follow it with an alert about the ok groups?

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
MORE INFO: When I raise the lambda to 90% (out of range for all

host/paths) I get no alerts at all (which is expected)

Post by j***@gmail.com

Post by j***@gmail.com

Post by Archie Archbold
Thanks to anyone who can help me understand this