[influxdb] Anomaly Detection for data whose pattern repeats daily

[a***@gmail.com]

Hello All,

I have been playing with Kapacitor, trying various ways to find anomaly like Morgoth, joining today's data with yesterday's and comparing the std deviation.

I see Morgoth works great in the case when data pattern range is vast(which is in my case, it can vary from few hundreds to thousands).

Only problem i find with Morgoth is that it doesn't take into account the time of day in which the event occurred. Lets say at night time sum(count) is generally around 100 but in day time its 10,000. Above script will not give an alert if the sum(count) falls to 100 at the day time, as that is a common sum(count) for it. (during the night time).

Can suggest some better fingerprinting algorithm for Morgoth which takes time into account.

Or some other method for anomaly detection for data:
1) Whose range is vast. (I can't just compare std. dev at all times)
2) There is a common pattern expected everyday. ( but we need to find this NOT at the day end)

Here is my existing TICKScript :

var data = batch
|query('''
SELECT sum("count")
FROM "db"."autogen".events
''')
.groupBy('domain_name', 'domain_hash')
.period(15m)
.cron('*/15 * * * *')
.offset(2h)

data
|deadman(0.0, 5m)
.id('deadman')
.message('Deadman alert: less than 1 datapoint in 5m')
.log('/tmp/no_events.log')

data
@morgoth()
.field('sum')
.anomalousField('anomalous')
.scoreField('anomalyScore')
.errorTolerance(0.01)
.minSupport(0.03)
.sigma(3.0)
|alert()
// Trigger a critical alert when the window is marked as anomalous.
.crit(lambda: "anomalous")
.log('/tmp/total_event_alert.log')

Thanks, any help is appreciated.