[influxdb] Telegraf Logparser plugin - * not working on Windows

[z***@gmail.com]

Hi All,

I'm having some issues using the Telegraf logparser plugin to parse all log files in a directory on Windows. I'm using the latest 1.01 Windows binary.

If I use an * in the file path, it doesn't find any files to read. But it works if I give the complete path to the file. Am I doing anything wrong in the conf file?

telegraf.conf input:

[[inputs.logparser]]
## files to tail.
files = ["D:\\temp\\*.log"]
## Read file from beginning.
from_beginning = true
## Override the default measurement name, which would be "logparser_grok"
name_override = "log"
## For parsing logstash-style "grok" patterns:
[inputs.logparser.grok]
patterns = ["%{CUSTOM_LOG}"]
custom_patterns = '''
CUSTOM_LOG %{TIMESTAMP_ISO8601:ts:ts-"2006-01-02 15:04:05"} %{IPORHOST:serverhost} %{WORD:method:tag} %{URIPATH:page:tag} %{NOTSPACE:querystring} %{NUMBER:port:drop} %{NOTSPACE:username:drop} %{IPORHOST:clienthost} %{NOTSPACE:useragent:drop} %{NOTSPACE:cookie:drop} %{NOTSPACE:referer:drop} %{IPORHOST:hostname} %{NUMBER:response:tag} %{NUMBER:subresponse} %{NUMBER:scstatus:drop} %{NUMBER:scbytes:drop} %{NUMBER:csbytes:drop} %{NUMBER:timetaken:int}
'''

--test output:

* Plugin: logparser, Collection 1

Thank you

[k***@gmail.com]

Hi,

You can try this for the file path on windows and it works for me. The wildcard does now work on Windows, may be someone has to fix that but if you can combine all the *.log into single.log file then you can provide path as below

files = ["D:/temp/single.log"]

Hope this works for you. Enjoy.

Post by z***@gmail.com
Hi All,
I'm having some issues using the Telegraf logparser plugin to parse all log files in a directory on Windows. I'm using the latest 1.01 Windows binary.
If I use an * in the file path, it doesn't find any files to read. But it works if I give the complete path to the file. Am I doing anything wrong in the conf file?
[[inputs.logparser]]
## files to tail.
files = ["D:\\temp\\*.log"]
## Read file from beginning.
from_beginning = true
## Override the default measurement name, which would be "logparser_grok"
name_override = "log"
[inputs.logparser.grok]
patterns = ["%{CUSTOM_LOG}"]
custom_patterns = '''
CUSTOM_LOG %{TIMESTAMP_ISO8601:ts:ts-"2006-01-02 15:04:05"} %{IPORHOST:serverhost} %{WORD:method:tag} %{URIPATH:page:tag} %{NOTSPACE:querystring} %{NUMBER:port:drop} %{NOTSPACE:username:drop} %{IPORHOST:clienthost} %{NOTSPACE:useragent:drop} %{NOTSPACE:cookie:drop} %{NOTSPACE:referer:drop} %{IPORHOST:hostname} %{NUMBER:response:tag} %{NUMBER:subresponse} %{NUMBER:scstatus:drop} %{NUMBER:scbytes:drop} %{NUMBER:csbytes:drop} %{NUMBER:timetaken:int}
'''
* Plugin: logparser, Collection 1
Thank you

[s***@gmail.com]

Hi,

I am trying to parse java logs from telegraf on windows machine.log file is not able to read by telegraf.please help me with correct working configuration. here is my configuration:
[[inputs.logparser]]
## files to tail.
files = ["C:/Users/hp/logs/application_logs.log"]
## Read file from beginning.
from_beginning = true
## Override the default measurement name, which would be "logparser_grok"
name_override = "my_log"
## For parsing logstash-style "grok" patterns:
[inputs.logparser.grok]
patterns = ["%{TOMCATLOG}"]
custom_patterns = '''
JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*
#Space is an allowed character to match special cases like 'Native Method' or 'Unknown Source'
JAVAFILE (?:[A-Za-z0-9_. -]+)
#Allow special <init>, <clinit> methods
JAVAMETHOD (?:(<(?:cl)?init>)|[a-zA-Z$_][a-zA-Z$_0-9]*)
#Line number is optional in special cases 'Native method' or 'Unknown source'
JAVASTACKTRACEPART %{SPACE}at %{JAVACLASS:class}\.%{JAVAMETHOD:method}\(%{JAVAFILE:file}(?::%{NUMBER:line})?\)
# Java Logs
JAVATHREAD (?:[A-Z]{2}-Processor[\d]+)
JAVACLASS (?:[a-zA-Z0-9-]+\.)+[A-Za-z0-9$]+
JAVAFILE (?:[A-Za-z0-9_.-]+)
JAVALOGMESSAGE (.*)
# MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM
CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
# yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800
TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}
CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}
# 2014-01-09 20:03:28,269 -0800 | ERROR | com.example.service.ExampleService - something compeletely unexpected happened...
TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}
'''

Post by k***@gmail.com
Hi,
You can try this for the file path on windows and it works for me. The wildcard does now work on Windows, may be someone has to fix that but if you can combine all the *.log into single.log file then you can provide path as below
files = ["D:/temp/single.log"]
Hope this works for you. Enjoy.

Post by z***@gmail.com
Hi All,
I'm having some issues using the Telegraf logparser plugin to parse all log files in a directory on Windows. I'm using the latest 1.01 Windows binary.
If I use an * in the file path, it doesn't find any files to read. But it works if I give the complete path to the file. Am I doing anything wrong in the conf file?
[[inputs.logparser]]
## files to tail.
files = ["D:\\temp\\*.log"]
## Read file from beginning.
from_beginning = true
## Override the default measurement name, which would be "logparser_grok"
name_override = "log"
[inputs.logparser.grok]
patterns = ["%{CUSTOM_LOG}"]
custom_patterns = '''
CUSTOM_LOG %{TIMESTAMP_ISO8601:ts:ts-"2006-01-02 15:04:05"} %{IPORHOST:serverhost} %{WORD:method:tag} %{URIPATH:page:tag} %{NOTSPACE:querystring} %{NUMBER:port:drop} %{NOTSPACE:username:drop} %{IPORHOST:clienthost} %{NOTSPACE:useragent:drop} %{NOTSPACE:cookie:drop} %{NOTSPACE:referer:drop} %{IPORHOST:hostname} %{NUMBER:response:tag} %{NUMBER:subresponse} %{NUMBER:scstatus:drop} %{NUMBER:scbytes:drop} %{NUMBER:csbytes:drop} %{NUMBER:timetaken:int}
'''
* Plugin: logparser, Collection 1
Thank you