Parsing POSTFIX logs and sending to ElasticSearch

Question

I am using Postfix and I need to parse the mail logs after the mails are sent. Right now, I have been trying to get things to work using Fluentd using the tail plugin to parse Postfix logs.

The problem that I am facing is that I cannot parse them by using the multiline option of the tail plugin. I can parse them individually using regular expressions but not together. This may be because of the fact that the logs are asynchronous in nature and each process logs into the file whenever it has information to be logged. I am planning to write a ruby script for this purpose but this might take a lot of time.

Is there a better solution than writing the script/custom plugin for Fluentd, considering I don't know Ruby/Perl?

Logs for which I do not care to extract information from:

Jan  5 09:02:48 localhost postfix/pickup[5501]: 1733D16A90A: uid=0 from=<root>
Jan  5 09:02:51 localhost postfix/qmgr[2596]: 1733D16A90A: removed

Logs for which I care to extract information from:

Jan  5 09:02:48 localhost postfix/cleanup[5978]: 1733D16A90A: message-id=<[email protected]>
Jan  5 09:02:48 localhost postfix/qmgr[2596]: 1733D16A90A: from=<[email protected]>, size=460, nrcpt=1 (queue active)
Jan  5 09:02:51 localhost postfix/smtp[5980]: 1733D16A90A: to=<[email protected]>, relay=gmail-smtp-in.l.google.com[74.125.130.27]:25, delay=3.4, delays=0.05/0.01/1.9/1.5, dsn=2.0.0, status=sent (250 2.0.0 OK 1483624971 s11si76004239pgc.259 - gsmtp)

The example logs need to be sent to ElasticSearch as a JSON File, which will then save them for further processing.

Answer 1

If you have only to track if a mail is sent. You could use Postfix queue id
(1733D16A90A in your log) as index key in elasticsearch. Then you have to parse each line with grok and update the status for your entry. Take a look at this example for grok patterns :

https://github.com/whyscream/postfix-grok-patterns/blob/master/50-filter-postfix.conf

Answer 2

I have answered a similar question here for parsing Postfix logs with Logstash, and although I hate linking to another answer, the concepts remain the same and should get you started, but do not involve the complexity of the configuration as whyscream provided.

The same concepts with grok apply for Fluent as well.

Your best bet is the file input plugin for reading the file. This will ensure that you do not have to worry about multiline processing, and can instead focus on each line of necessary information.

input {
    file {
        path => "/var/log/maillog"
        type => "postfix"   # You can define a type however you like.
    }
}

Ultimately you should try to avoid writing a custom script, if possible. The work required to do so has been taken on by the communities of Logstash, Fluent, and other similar tools. Managing regular expressions for thousands of granular logs is an annoyance, at best.

Answer 3

I needed exactly this a few years ago so I wrote log-ship-elastic-postfix. From my description:

Turns these:

Jul 26 04:18:34 mx12 postfix/pickup[20280]: 3mfHGL1r9gzyQP: uid=1208 from=<system>
Jul 26 04:18:34 mx12 postfix/cleanup[20659]: 3mfHGL1r9gzyQP: message-id=<[email protected]>
Jul 26 04:18:34 mx12 postfix/qmgr[28761]: 3mfHGL1r9gzyQP: from=<system>, size=813, nrcpt=1 (queue active)
Jul 26 04:18:34 mx12 postfix/smtp[20662]: 3mfHGL1r9gzyQP: to=<system>, relay=127.0.0.2[127.0.0.2]:25, delay=0.53, delays=0.13/0/0.23/0.16, dsn=2.0.0, status=sent (250 Queued! (#2.0.0))
Jul 26 04:18:34 mx12 postfix/qmgr[28761]: 3mfHGL1r9gzyQP: removed

into this:

{
    "id": "3mfHGL1r9gzyQP",
    "host": "mx12",
    "events": [
      {
        "date": "2015-07-26T04:18:34-04:00",
        "action": "queued"
      },
      {
        "to": "system",
        "relay": "127.0.0.2[127.0.0.2]:25",
        "dsn": "2.0.0",
        "status": "sent (250 Queued! (#2.0.0))",
        "date": "2015-07-26T04:18:34-04:00"
      },
      {
        "date": "2015-07-26T04:18:34-04:00",
        "action": "removed"
      }
    ],
    "date": "2015-07-26T04:18:34-04:00",
    "isFinal": true,
    "uid": "1208",
    "message-id": "[email protected]",
    "from": "system",
    "size": "813",
    "nrcpt": "1",
    "delay": "0.53",
    "delays": "0.13/0/0.23/0.16"
}

It runs under node.js and has billions of postfix log entries passed through it.