NXLOG logs are in unicode characters

Question

I have write the code like as following,

<Extension charconv>
    Module xm_charconv
    AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2
</Extension>

And,

<Input sql-ERlogs>
    Module      im_file
    File 'C:\Program Files\Microsoft SQL Server\MSSQL11.SQL\MSSQL\Log\ER*'
    ReadFromLast TRUE
    Exec        convert_fields("AUTO", "utf-8"); if $raw_event == '' drop();
</Input>

I got the output like following image

If I expand one of the log I got original log like this,

Why is it in Unicode characters before expand?

Answer 1

I had this same issue. Turns out "AUTO" doesn't work reliably. The NXLOG manual on MS SQLSRV hints at what the answer is; you must specify type, such as UTF-16LE.

Check the file contents with a hex editor and compare to MS's chart https://msdn.microsoft.com/en-us/library/windows/desktop/dd374101%28v=vs.85%29.aspx

Exec    $Event = convert($raw_event,"UTF-16LE","UTF-8"); to_json();