I have a couple of containers running as a Spark Cluster. When I am submitting my Python application, I made sure to have logging being managed by a the logging module. However, I cannot see any of my logs on the container stdout/stderr. How is this supposed to be configured?
This is how I am configuring my python logger:
logger = logging.getLogger(app_name)
logger.setLevel(logging.DEBUG) # set logger level
logFormatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
consoleHandler = logging.StreamHandler() #set streamhandler to stdout
consoleHandler.setFormatter(logFormatter)
logger.addHandler(consoleHandler)
Had anyone faced the same situation? How to overcome?
Thank you