New to logstash, really enjoying it.
Trying to parse a CSV file containing a timestamp. Would like to parse the timestamp and use it as the @timestamp field.
Sample of my CSV input
input {
stdin {}
}
filter {
# filter the input by csv (i.e. comma-separated-value)
csv {
columns => [
"Job ID",
"Server Name",
"Status Code",
"Job Type",
"Client Name",
"Start Time",
"End Time"
]
}
# parse the start time to create a real date
date {
# Examples of times in this log file
# "May 29, 2015 10:00:01 PM"
# "May 9, 2015 4:47:23 AM"
match => [ "End Time",
"MMM dd, YYYY HH:mm:ss aa",
"MMM d, YYYY HH:mm:ss aa" ]
}
}
# send the output to stdout, using the rubydebug codec
# rubydedug uses the Ruby Awesome Print library
output {
stdout { codec => rubydebug }
}
Sample of my input
108628,anmuswcnbu01,1,Backup,anmuswcrfax01.na.jnj.com,"May 29, 2015 10:00:01 PM","May 30, 2015 6:21:29 AM"
108629,anmuswcnbu01,1,Backup,anmuswcapps01.na.jnj.com,"May 29, 2015 10:00:01 PM","May 9, 2015 10:51:39 pm"
108630,anmuswcnbu01,1,Backup,anmuswcapps03.na.jnj.com,"May 29, 2015 10:00:01 PM","May 29, 2015 9:31:19 PM"
Sample of my output
Logstash startup completed
{
"message" => [
[0] "108628,anmuswcnbu01,1,Backup,anmuswcrfax01.na.jnj.com,\"May 29, 2015 10:00:01 PM\",\"May 30, 2015 6:21:29 AM\"\r"
],
"@version" => "1",
"@timestamp" => "2015-05-30T06:21:29.000Z",
"host" => "ip-172-31-34-14",
"Job ID" => "108628",
"Server Name" => "anmuswcnbu01",
"Status Code" => "1",
"Job Type" => "Backup",
"Client Name" => "anmuswcrfax01.na.jnj.com",
"Start Time" => "May 29, 2015 10:00:01 PM",
"End Time" => "May 30, 2015 6:21:29 AM"
}
{
"message" => [
[0] "108629,anmuswcnbu01,1,Backup,anmuswcapps01.na.jnj.com,\"May 29, 2015 10:00:01 PM\",\"May 9, 2015 10:51:39 pm\"\r"
],
"@version" => "1",
"@timestamp" => "2015-05-09T10:51:39.000Z",
"host" => "ip-172-31-34-14",
"Job ID" => "108629",
"Server Name" => "anmuswcnbu01",
"Status Code" => "1",
"Job Type" => "Backup",
"Client Name" => "anmuswcapps01.na.jnj.com",
"Start Time" => "May 29, 2015 10:00:01 PM",
"End Time" => "May 9, 2015 10:51:39 pm"
}
{
"message" => [
[0] "108630,anmuswcnbu01,1,Backup,anmuswcapps03.na.jnj.com,\"May 29, 2015 10:00:01 PM\",\"May 29, 2015 9:31:19 PM\"\r"
],
"@version" => "1",
"@timestamp" => "2015-05-29T09:31:19.000Z",
"host" => "ip-172-31-34-14",
"Job ID" => "108630",
"Server Name" => "anmuswcnbu01",
"Status Code" => "1",
"Job Type" => "Backup",
"Client Name" => "anmuswcapps03.na.jnj.com",
"Start Time" => "May 29, 2015 10:00:01 PM",
"End Time" => "May 29, 2015 9:31:19 PM"
}
Logstash shutdown completed
For example, in that last (3rd) row of data, rather than:
"@timestamp" => "2015-05-29T09:31:19.000Z",
I feel like I should be getting
"@timestamp" => "2015-05-29T21:31:19.000Z",
As best I can tell, the date filter seems to be ignoring my "half-day" syntax
match => [ "End Time",
"MMM dd, YYYY HH:mm:ss aa",
"MMM d, YYYY HH:mm:ss aa" ]
New to logstash, so wondering if I'm doing something wrong ?
-Chad
The
date
filter uses a format compatible with Joda-Time.Quoting part of Joda's symbol table:
It's easy to overlook, but halfday hours in your case are
KK
instead ofHH
.