Message missing in NXLOG log shipping

Question

I have following SQL logs in ERRORLOG file,

2014-12-19 14:27:21.76 spid52      Starting up database 'MyDatabase'.
2014-12-19 14:27:22.06 spid52      Setting database option COMPATIBILITY_LEVEL to 110 for database 'MyDatabase'.
2014-12-19 14:27:22.06 spid52      Setting database option ANSI_NULL_DEFAULT to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.06 spid52      Setting database option ANSI_NULLS to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option ANSI_PADDING to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option ANSI_WARNINGS to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option ARITHABORT to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option AUTO_CLOSE to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option AUTO_CREATE_STATISTICS to ON for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option AUTO_SHRINK to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option AUTO_UPDATE_STATISTICS to ON for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option CURSOR_CLOSE_ON_COMMIT to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option CURSOR_DEFAULT to GLOBAL for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option CONCAT_NULL_YIELDS_NULL to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option NUMERIC_ROUNDABORT to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.09 spid52      Setting database option QUOTED_IDENTIFIER to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option RECURSIVE_TRIGGERS to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option DISABLE_BROKER to ON for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option AUTO_UPDATE_STATISTICS_ASYNC to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option DATE_CORRELATION_OPTIMIZATION to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option PARAMETERIZATION to SIMPLE for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option READ_COMMITTED_SNAPSHOT to OFF for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option READ_WRITE to ON for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option RECOVERY to FULL for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option MULTI_USER to ON for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option PAGE_VERIFY to CHECKSUM for database 'MyDatabase'.
2014-12-19 14:27:22.10 spid52      Setting database option target_recovery_time to 0 for database 'MyDatabase'.

I have used the following code in nxlog to ship the logs to Logstash..

<Input sql-ERlogs>
    Module      im_file
    File 'C:\Program Files\Microsoft SQL Server\MSSQL11.SQL\MSSQL\Log\ER*'
    ReadFromLast TRUE
    Exec        to_json();
</Input>

I am receiving following messages only in kibana..

{"message":"{\"EventReceivedTime\":\"2014-12-19 14:52:50\",\"SourceModuleName\":\"sql-ERlogs\",\"SourceModuleType\":\"im_file\"}\r","@version":"1","@timestamp":"2014-12-19T09:21:11.709Z","host":"192.168.1.9:1426","type":"sqllogs"}

here I have the time only from this log.. Why there is no messages from this log, I mean why Starting up database 'MyDatabase'. not there. Is there any wrong in my log shipping way.

Update After update the @b0ti answer I am get the following output

{"EventReceivedTime":"2014-12-19 15:50:36","SourceModuleName":"sql-ERlogs","SourceModuleType":"im_file","Message":"2\u00000\u00001\u00004\u0000-\u00001\u00002\u0000-\u00001\u00009\u0000 \u00001\u00005\u0000:\u00005\u00000\u0000:\u00003\u00006\u0000.\u00003\u00000\u0000 \u0000s\u0000p\u0000i\u0000d\u00005\u00002\u0000 \u0000 \u0000 \u0000 \u0000 \u0000 \u0000S\u0000e\u0000t\u0000t\u0000i\u0000n\u0000g\u0000 \u0000d\u0000a\u0000t\u0000a\u0000b\u0000a\u0000s\u0000e\u0000 \u0000o\u0000p\u0000t\u0000i\u0000o\u0000n\u0000 \u0000R\u0000E\u0000C\u0000O\u0000V\u0000E\u0000R\u0000Y\u0000 \u0000t\u0000o\u0000 \u0000F\u0000U\u0000L\u0000L\u0000 \u0000f\u0000o\u0000r\u0000 \u0000d\u0000a\u0000t\u0000a\u0000b\u0000a\u0000s\u0000e\u0000 \u0000'\u0000S\u0000a\u0000m\u0000p\u0000l\u0000e\u0000'\u0000.\u0000"}

Why the message is comes as Unicode characters, how can I change it as message?

Answer 1

By adding extension for charconv I got the solution.

Code for extension,

<Extension charconv>
    Module xm_charconv
    AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2
</Extension>

And modified the input as follows,

<Input sql-ERlogs>
    Module      im_file
    File 'C:\Program Files\Microsoft SQL Server\MSSQL11.SQL\MSSQL\Log\ER*'
    ReadFromLast TRUE
    Exec        convert_fields("AUTO", "utf-8");
</Input>

Now I got the log as perfectly.

I hope this solution will be helpful for someone who needs solution.

Thanks.

Answer 2

im_file reads data into $raw_event. Calling to_json() rewrites $raw_event with the json of all fields except raw_event, thus your original message stored in $raw_event is lost. What you want is this:

<Input sql-ERlogs>
    Module      im_file
    File 'C:\Program Files\Microsoft SQL Server\MSSQL11.SQL\MSSQL\Log\ER*'
    ReadFromLast TRUE
    Exec        $Message = $raw_event; to_json();
</Input>