Format logs for PyFlink application with different pattern for the Python part

Question

I have a Python application running in PyFlink.

Is there a way of saying "Apply this pattern to the logs produced by the python code, and this other pattern to every other log?"

I'd like to format the log messages that are coming from my Python code to show just the message printed by the app (what the Log4j PatternLayout calls %m%n) but keep log messages printed by other Java classes that are not my code with some extra information (the logger's name that printed the message, the time... basically, this layout %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n)

I'd like the terminal to look something like:

2022-05-19 16:48:31,765 INFO  org.apache.flink.api.SinkFunction [] - Hello from Java

This line is python logging (no date, no priority info, no logname)...

I am using a ConsoleAppender for the root logger, and I could change the format in there to be just %m%n, and that works and it shows just the message in the "Python side"... BUT it'd also apply to the "Java side" as well. I mean: if I do that, I lose the Java extra information. It would look like:

Hello from Java

This line is python logging (no date, no priority info, no logname)...

^{Output from Java "bad" (no date, no logger name...), output from Python "good"}

If I keep the root logger's pattern as %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n, my logs look like:

2022-05-19 16:48:31,765 INFO  org.apache.flink.api.SinkFunction [] - Hello from Java

2022-05-19 16:48:41,470 INFO  /app/borrajax/my_python_file.py:290  [] - This line is python logging (no date, no priority info, no logname)...

^{Output from Java "good", output from Python "bad" (extra information to the left)}

This is my... current config file for log4j (unsuccessful, as many others):

appenders=borrajax_console,console

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.console.ref = ConsoleAppender

# Log all infos to the console
appender.console.name = ConsoleAppender
appender.console.type = CONSOLE
appender.console.layout.type = PatternLayout
appender.console.layout.pattern =%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Console appender JUST FOR PYTHON:
appender.borrajax_console.name=BorrajaxConsoleAppender
appender.borrajax_console.type=CONSOLE
appender.borrajax_console.layout.type = PatternLayout
appender.borrajax_console.layout.pattern =%m%n

logger.app.name = org.apache.flink.streaming.api
logger.app.appenderRef.borrajax_console.ref = BorrajaxConsoleAppender

Answer 1

You could try to set the root logger's pattern as %m%n and Java logger's pattern as %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n, e.g.

appenders=borrajax_console,console

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.console.ref = ConsoleAppender

# Log all infos to the console
appender.console.name = ConsoleAppender
appender.console.type = CONSOLE
appender.console.layout.type = PatternLayout
appender.console.layout.pattern =%m%n

# Console appender JUST FOR JAVA:
appender.borrajax_console.name=BorrajaxConsoleAppender
appender.borrajax_console.type=CONSOLE
appender.borrajax_console.layout.type = PatternLayout
appender.borrajax_console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

logger.app.name = org.apache
logger.app.appenderRef.borrajax_console.ref = BorrajaxConsoleAppender