[influxdb] Automatically downsample all data

[o***@gmail.com]

Hi,

In my environment I'm planning to monitor up to 1.600 hosts (cpu, memory, network, disk, etc...) and I'm a little bit worried about the ammount of space required to store all such ammount of data.

I have though that an automatic downsampling of data would be a nice approach. After reading a lot, it looks one can make use of the "continuous query" feature. Nevertheless, it just applies to one measurement.

For example "CPU USAGE"

The first month: 1 point every 20 seconds
The second month: 1 point every minute
The third mohnt: 1 point every hour
and so on...

(Im pretty sure this is not the best configuration but it is good enought for make understand myself).

Is there any global configuration in order to achieve this objective without doing it point per point?

Thanks in dvance,
Óscar

[r***@gmail.com]

Post by o***@gmail.com
Hi,
In my environment I'm planning to monitor up to 1.600 hosts (cpu, memory, network, disk, etc...) and I'm a little bit worried about the ammount of space required to store all such ammount of data.
I have though that an automatic downsampling of data would be a nice approach. After reading a lot, it looks one can make use of the "continuous query" feature. Nevertheless, it just applies to one measurement.
For example "CPU USAGE"
The first month: 1 point every 20 seconds
The second month: 1 point every minute
The third mohnt: 1 point every hour
and so on...
(Im pretty sure this is not the best configuration but it is good enought for make understand myself).
Is there any global configuration in order to achieve this objective without doing it point per point?
Thanks in dvance,
Óscar

I'm Having the same issue. Also looking for a simple way to set the downsampling automatically for all measurements for a single database.

[Jan Vokas]

#meetoo

Post by o***@gmail.com

Post by o***@gmail.com
Hi,
In my environment I'm planning to monitor up to 1.600 hosts (cpu,

memory, network, disk, etc...) and I'm a little bit worried about the
ammount of space required to store all such ammount of data.

Post by o***@gmail.com
I have though that an automatic downsampling of data would be a nice

approach. After reading a lot, it looks one can make use of the "continuous
query" feature. Nevertheless, it just applies to one measurement.

Post by o***@gmail.com
For example "CPU USAGE"
The first month: 1 point every 20 seconds
The second month: 1 point every minute
The third mohnt: 1 point every hour
and so on...
(Im pretty sure this is not the best configuration but it is good

enought for make understand myself).

Post by o***@gmail.com
Is there any global configuration in order to achieve this objective

without doing it point per point?

Post by o***@gmail.com
Thanks in dvance,
Óscar

I'm Having the same issue. Also looking for a simple way to set the
downsampling automatically for all measurements for a single database.

[Jan Vokas]

#metoo

Post by o***@gmail.com

Post by o***@gmail.com
Hi,
In my environment I'm planning to monitor up to 1.600 hosts (cpu,

memory, network, disk, etc...) and I'm a little bit worried about the
ammount of space required to store all such ammount of data.

Post by o***@gmail.com
I have though that an automatic downsampling of data would be a nice

approach. After reading a lot, it looks one can make use of the "continuous
query" feature. Nevertheless, it just applies to one measurement.

Post by o***@gmail.com
For example "CPU USAGE"
The first month: 1 point every 20 seconds
The second month: 1 point every minute
The third mohnt: 1 point every hour
and so on...
(Im pretty sure this is not the best configuration but it is good

enought for make understand myself).

Post by o***@gmail.com
Is there any global configuration in order to achieve this objective

without doing it point per point?

Post by o***@gmail.com
Thanks in dvance,
Óscar

I'm Having the same issue. Also looking for a simple way to set the
downsampling automatically for all measurements for a single database.

[Gino Lisignoli]

From memory I did this with a continuous query similar to example 3 here:
https://docs.influxdata.com/influxdb/v1.4/query_language/continuous_queries/#examples-of-basic-syntax

CREATE CONTINUOUS QUERY "cq_basic_br" ON "transportation"
BEGIN
SELECT mean(*) INTO "downsampled_transportation"."autogen".:MEASUREMENT
FROM /.*/ GROUP BY time(30m),*
END

The above example will take all measurements and values and put them into a
new database called downsampled_transportation (with a retention policy
that lasts forever). With a series of retention policies and queries you
should be able to achieve what you are after.

Post by Jan Vokas
#metoo

Post by o***@gmail.com

Post by o***@gmail.com
Hi,
In my environment I'm planning to monitor up to 1.600 hosts (cpu,

memory, network, disk, etc...) and I'm a little bit worried about the
ammount of space required to store all such ammount of data.

Post by o***@gmail.com
I have though that an automatic downsampling of data would be a nice

approach. After reading a lot, it looks one can make use of the "continuous
query" feature. Nevertheless, it just applies to one measurement.

Post by o***@gmail.com
For example "CPU USAGE"
The first month: 1 point every 20 seconds
The second month: 1 point every minute
The third mohnt: 1 point every hour
and so on...
(Im pretty sure this is not the best configuration but it is good

enought for make understand myself).

Post by o***@gmail.com
Is there any global configuration in order to achieve this objective

without doing it point per point?

Post by o***@gmail.com
Thanks in dvance,
Óscar

I'm Having the same issue. Also looking for a simple way to set the
downsampling automatically for all measurements for a single database.

--
Remember to include the version number!
---
You received this message because you are subscribed to the Google Groups
"InfluxData" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/influxdb.
To view this discussion on the web visit https://groups.google.com/d/
msgid/influxdb/db157831-7cac-4d40-8955-7de0533666b4%40googlegroups.com
<https://groups.google.com/d/msgid/influxdb/db157831-7cac-4d40-8955-7de0533666b4%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[r***@gmail.com]

@Geno Is this CQ only working when the "INTO" is another database? or can is also be an other Retention policy in the same database. I tried it, and i'm not getting any data into the RP.



__________________________________________________________
name: telegraf
name query
---- -----
cq_four_week CREATE CONTINUOUS QUERY cq_four_week ON telegraf BEGIN SELECT mean(*) INTO telegraf.four_weeks.:MEASUREMENT FROM telegraf.autogen./.*/ GROUP BY time(5m), * END
cq_a_year CREATE CONTINUOUS QUERY cq_a_year ON telegraf BEGIN SELECT mean(*) INTO telegraf.a_year.:MEASUREMENT FROM telegraf.autogen./.*/ GROUP BY time(30m), * END
__________________________________________________________

[Gino Lisignoli]

Yep, you can have a single database with multiple retention policies. For
example:

create RETENTION POLICY "1hour" on "telegraf" DURATION 1h REPLICATION 1
DEFAULT
create RETENTION POLICY "1day" on "telegraf" DURATION 1d REPLICATION 1
create RETENTION POLICY "1week" on "telegraf" DURATION 1w REPLICATION 1
create DATABASE another_telegraf

CREATE CONTINUOUS QUERY "1hour-to-1day-telegraf" on "telegraf" BEGIN SELECT
mean(*) INTO "telegraf"."1day".:MEASUREMENT FROM /.*/ GROUP BY time(1m) END
CREATE CONTINUOUS QUERY "1hour-to-1hour-telegraf" on "telegraf" BEGIN
SELECT mean(*) INTO "telegraf"."1hour".:MEASUREMENT FROM /.*/ GROUP BY
time(1h) END
CREATE CONTINUOUS QUERY "1hour-to-1week-telegraf" on "telegraf" BEGIN
SELECT mean(*) INTO "telegraf"."1week".:MEASUREMENT FROM /.*/ GROUP BY
time(1w) END
CREATE CONTINUOUS QUERY "1hour-to-1day-another_telegraf" on "telegraf"
BEGIN SELECT mean(*) INTO "another_telegraf"."autogen".:MEASUREMENT FROM
/.*/ GROUP BY time(1m) END

This will take data from the 'telegraf' database and put it into another
retention policy on the same database. But the other continuous query
"1hour-1d-another_telegraf" will take data from the "telegraf" database and
put it into the "autogen" retention policy (or any policy you specify) on
the "another_telegraf" database.

Post by r***@gmail.com
@Geno Is this CQ only working when the "INTO" is another database? or can
is also be an other Retention policy in the same database. I tried it, and
i'm not getting any data into the RP.
__________________________________________________________
name: telegraf
name query
---- -----
cq_four_week CREATE CONTINUOUS QUERY cq_four_week ON telegraf BEGIN SELECT
mean(*) INTO telegraf.four_weeks.:MEASUREMENT FROM telegraf.autogen./.*/
GROUP BY time(5m), * END
cq_a_year CREATE CONTINUOUS QUERY cq_a_year ON telegraf BEGIN SELECT
mean(*) INTO telegraf.a_year.:MEASUREMENT FROM telegraf.autogen./.*/ GROUP
BY time(30m), * END
__________________________________________________________
--
Remember to include the version number!
---
You received this message because you are subscribed to the Google Groups
"InfluxData" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/influxdb.
To view this discussion on the web visit https://groups.google.com/d/
msgid/influxdb/beb456d0-d674-447d-9d1c-3cad0acf2bf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.