Configure Livy log4j properties on EMR Cluster

Question

I am using rest Apis provided by livy to submit spark jobs on EMR cluster.I am able to overwrite some of the livy properties in livy-conf file using below json in configuration while creating cluster -

[{'classification': 'livy-conf','Properties': {'livy.server.session.state-retain.sec':'1200s'}}]

And I see there is a log4j.properties folder in livy conf folder. , but how do I overwrite the configuration properties in log4j.properties? I used the below json in software settings while starting cluster but it gives the error - Classification 'log4j.properties' is not valid for parent classification 'null'.

[
    {
        "classification": "livy-conf",
        "properties": {
            "livy.server.session.state-retain.sec": "1200s",
            "livy.server.yarn.poll-interval": "3s"
        }
        
    },
    {
        "classification": "log4j.properties",
        "properties": {
            "log4j.rootCategory": "INFO,file",
            "log4j.appender.file": "org.apache.log4j.FileAppender",
            "log4j.appender.file.append": "true",
            "log4j.appender.file.File": "s3://oneid-idaas-dev-us-east-1/dev/emr/livy_logs/",
            "log4j.appender.file.layout": "org.apache.log4j.PatternLayout",
            "log4j.appender.file.layout.ConversionPattern": ""
        }
    }
]

When I sub configure it under livy-conf classification , it still gives the error - Classification 'log4j.properties' is not valid for parent classification 'livy-conf'.

[
  {
    "classification": "livy-conf",
    "properties": {
      "livy.server.session.state-retain.sec": "1200s",
      "livy.server.yarn.poll-interval": "3s"
    },
    "configurations": [
      {
        "classification": "log4j.properties",
        "properties": {
          "log4j.rootCategory": "INFO,file",
          "log4j.appender.file": "org.apache.log4j.FileAppender",
          "log4j.appender.file.append": "true",
          "log4j.appender.file.File": "s3://oneid-idaas-dev-us-east-1/dev/emr/livy_logs/",
          "log4j.appender.file.layout": "org.apache.log4j.PatternLayout",
          "log4j.appender.file.layout.ConversionPattern": "%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %5p %c{7} - %m%n"
        }
      }
    ]
  }
]

Is it possible to overwrite log4j.properties in livy-conf? How do I do it? Am expecting the stdouts generated from spark jobs to be going into the file I configure here. Thanks is advance.

Answer 1

I think that you got the wrong classification name, the name should be "livy-log4j", reference link here.