Can not use log4net.Ext.Json to make Json log in Sitecore related project

Question

As we know that Sitecore has capsulated log4net as default logging functionality(Sitecore.Logging). My task is to use log4net.Ext.Json Layout to create Json log and feed it from Azure to Datadog APM. Corresponding useful links are as follows: https://docs.datadoghq.com/logs/log_collection/csharp/?tab=log4net https://docs.datadoghq.com/tracing/connect_logs_and_traces/dotnet?tab=log4net https://github.com/DataDog/dd-trace-dotnet/tree/master/samples/AutomaticTraceIdInjection/Log4NetExample

While during last 2 weeks, I failed to achieve it:

Try 1: Add log4net.ext.Json(ver. 2.0.8.3) NuGet Package in Visual Studio. Since Sitecore has capsulated log4net(ver. 2.0.9), so I do not need to add log4net NuGet package.

Then I configured config file as follows:

<appender name="JsonFileAppender" type="log4net.Appender.RollingFileAppender">
      <file value="$(dataFolder)/logs/json_log/json_log.json" />
      <appendToFile value="true" />
      <maxSizeRollBackups value="10" />
      <maximumFileSize value="100MB" />
      <!--json formatted log4net logging-->
      <layout type='log4net.Layout.SerializedLayout, log4net.Ext.Json'>
          <decorator type="log4net.Layout.Decorators.StandardTypesDecorator, log4net.Ext.Json" />
          <!--explicit default members-->
          <default />
          <!--remove the default preformatted message member-->
          <remove value="ndc" />
          <remove value="message" />
          <!--add raw message-->
          <member value="message:messageobject" />
          <!--add value='properties' to emit Datadog properties -->
          <member value='properties'/>
          <member value='dd.env' />
          <member value='dd.service' />
          <member value='dd.version' />
          <member value='dd.trace_id' />
          <member value='dd.span_id' />
      </layout>
</appender>
<root>
      <level value='INFO'/>
      <appender-ref ref="JsonFileAppender"/>
</root>

The result is that Json file is created, while no content written into the log file.

Try 2: I downloaded log4net.Ext.Json source code from official gitlab site, and create an empty project and involved all necessary C# source files of the log4net.Ext.Json into my solution. I also renamed the project name to log4netExtJson, which will not make conflict to existing Out-of-the-box log4net.Ext.Json.dll. To make log4net.Core working, I also have to add log4net(ver. 2.0.12) NuGet package. The source code is as follows:

https://gitlab.com/reeloadead/log4net.Ext.Json#installation

And the config file I changed to:

<appender name="JsonFileAppender" type="log4net.Appender.RollingFileAppender">
      <file value="$(dataFolder)/logs/json_log/json_log.json" />
      <appendToFile value="true" />
      <maxSizeRollBackups value="10" />
      <maximumFileSize value="100MB" />
      <!--json formatted log4net logging-->
      <layout type='log4net.Layout.SerializedLayout, log4netExtJson'>
          <decorator type="log4net.Layout.Decorators.StandardTypesDecorator, log4netExtJson" />
          <!--explicit default members-->
          <default />
          <!--remove the default preformatted message member-->
          <remove value="ndc" />
          <remove value="message" />
          <!--add raw message-->
          <member value="message:messageobject" />
          <!--add value='properties' to emit Datadog properties -->
          <member value='properties'/>
          <member value='dd.env' />
          <member value='dd.service' />
          <member value='dd.version' />
          <member value='dd.trace_id' />
          <member value='dd.span_id' />
      </layout>
</appender>
<root>
      <level value='INFO'/>
      <appender-ref ref="JsonFileAppender"/>
</root>

The outcome of this round is that even no json log file is created.

Try 3: I contacted Sitecore Support portal and got response that they did not tested log4net.Ext.Json package, and they need me to try SitecoreLoggingExtensions NuGet package as a workaround. Source codes is as follows:

https://github.com/sitecoreops/sitecore-logging-extensions

The config file I wrote is as follows:

<appender name="JsonFileAppender" type="log4net.Appender.RollingFileAppender">
      <file value="$(dataFolder)/logs/json_log/json_log.json" />
      <appendToFile value="true" />
      <maxSizeRollBackups value="10" />
      <maximumFileSize value="100MB" />
      <!--json formatted log4net logging-->
      <!--SerializedLayout-->
      <layout type='SitecoreLoggingExtensions.JsonLayout, SitecoreLoggingExtensions'>  
          <!--explicit default members-->
          <default />
          <!--add value='properties' to emit Datadog properties -->
          <member value='properties'/>
          <!-- Manual changes: start -->
          <member value='dd.env' />
          <member value='dd.service' />
          <member value='dd.version' />
          <member value='dd.trace_id' />
          <member value='dd.span_id' />
          <!-- Manual changes: end -->
      </layout>
</appender>
<root>
      <!--<level value='INFO'/>-->
      <appender-ref ref="JsonLogFileAppender"/>
</root>

The Json log with content is finally created! But after I made the setting on Azure, there is no connection with Datadog with log file. My guess is that the Json log with the Layout of SitecoreLoggingExtensions is not compatible with Datadog. So this way is not useful for my need.

I wonder if someone has successfully configured log4net.Ext.Json package in Sitecore related project before.

Answer 1

Similar experience: empty files, xml configuration errors, etc. Gave up trying to use the JSON extension, but found another solution.

I use pattern layout, and changed the pattern to be JSON. In the XML configuration you cannot use double quotes, but JsonConvert (used for deserialization) does'nt care:

    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="{'Thread':'%4t', 'Date':'%d{ABSOLUTE}', 'Level':'%-5p', 'Message':%m}%n" />
    </layout>

Implement a simple wrapper for the log4net logger and pass an object (of your choice) as the log message. Serialize it using JsonConvert and write it to the log file:

    public static void Info(LogMessage logMessage)
    {
        var message = JsonConvert.SerializeObject(logMessage);

        logger.Info(message);
    }

My initial intention was to be able to parse custom log files to be able, to generate some statistics, but was tired of using tidious string operations. So JSON was the format of choice.