AWS Elastic Beanstalk logging with python (django)

Question

How do you manage your application logs in AWS elastic beanstalk? Which file you write you application logs to?

I'm using the following Logging configuration in my development environment but this doesn't work when I deploy in AWS.

DEBUG_LOG_DIR = BASE_DIR + "/django_debug.log"
LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    # How to format the output
    'formatters': {
        'standard': {
            'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            'datefmt' : "%d/%b/%Y %H:%M:%S"
        },
    },
    # Log handlers (where to go)
    'handlers': {
        'null': {
            'level':'DEBUG',
            'class':'django.utils.log.NullHandler',
        },
        'log_file': {
            'level':'DEBUG',
            'class':'logging.handlers.RotatingFileHandler',
            'filename': DEBUG_LOG_DIR,
            'maxBytes': 50000,
            'backupCount': 2,
            'formatter': 'standard',
        },
        'console':{
            'level':'INFO',
            'class':'logging.StreamHandler',
            'formatter': 'standard'
        },
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
        },
    },
    # Loggers (where does the log come from)
    'loggers': {
        'repackager': {
            'handlers': ['console', 'log_file'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'django': {
            'handlers':['console'],
            'propagate': True,
            'level':'WARN',
        },
        'django.db.backends': {
            'handlers': ['console', 'log_file'],
            'level': 'WARN',
            'propagate': False,
        },
        '': {
            'handlers': ['console', 'log_file'],
            'level': 'DEBUG',
        },
    }
}

Answer 1

Ok, I figured out a way to do it.

First I connected via ssh to ec2 machine, then I create a folder in /var/log called app_logs with root user:

mkdir /var/log/app_logs

After that I did the follow:

cd /var/log/
chmod g+s app_logs/
setfacl -d -m g::rw app_logs/
chown wsgi:wsgi app_logs/

That ensures that all the files created in this folder will have wsgi as owner and will be writable for the group that the file belongs. I had to do that because I noticed that the log file created by django app had root as owner and owner group but the application runs through wsgi user.

Finally I changed DEBUG_LOG_DIR to /var/log/app_logs/django_debug.log

Answer 2

This answer is for Amazon Linux 2 only. For those who have not migrated yet, please see my old answer for Amazon Linux AMI.

Background

The official AWS Python sample-application for Amazon linux 2 uses the /tmp folder for logging.

However, custom log files added to /tmp are not included automatically when requesting logs from Elastic Beanstalk. To include custom log files, we need to create logging tasks in subfolders of /opt/elasticbeanstalk/tasks on the EC2 instance. See instructions in the documentation.

The sample app (source) accomplishes this using .ebextensions. However, the AWS Linux 2 migration docs suggest we should use .platform hooks instead:

We recommend using platform hooks to run custom code on your environment instances. You can still use commands and container commands in .ebextensions configuration files, but they aren't as easy to work with. For example, writing command scripts inside a YAML file can be cumbersome and difficult to test.

This has the additional advantage that output from platform hooks is collected in a separate log file, viz. /var/log/eb-hooks.log, which makes debugging a bit easier.

DRY logging setup for a basic Django app on Amazon Linux 2

Log level and log path are defined in one place, as Elastic Beanstalk environment properties, e.g. in .ebextensions/options.config:

option_settings:
  aws:elasticbeanstalk:application:environment:
    LOG_LEVEL: INFO
    DJANGO_LOG_FILE_PATH: /tmp/django-app.log
    ...

The DJANGO_LOG_FILE_PATH environment property can now be used in a platform hook to create logging tasks:

.platform/hooks/postdeploy/020_create_logging_tasks.sh

#!/bin/bash
TASKS_DIR=/opt/elasticbeanstalk/tasks
# include all app log files in bundle logs (replaces ".log" by "*")
echo "${DJANGO_LOG_FILE_PATH//.log/*}" > "$TASKS_DIR/bundlelogs.d/01-app-log.conf"
# include current app log file in tail logs
echo $DJANGO_LOG_FILE_PATH > "$TASKS_DIR/taillogs.d/01-app-log.conf"

Note that the platform hooks require execution permission, e.g. chmod +x 020_create_logging_tasks.sh. On windows you can use git as described here.

To prevent permission issues, we use another platform hook to ensure that the log file is always owned by webapp. Note this hook runs before the logging-tasks hook:

.platform/hooks/postdeploy/010_create_log_file.sh

#!/bin/bash

if test -f "$DJANGO_LOG_FILE_PATH";
then
  echo "$DJANGO_LOG_FILE_PATH exists"
else
  # create log file
  touch $DJANGO_LOG_FILE_PATH
fi

# set log file owner (we are currently "root", but the app runs as "webapp")
chown webapp:webapp $DJANGO_LOG_FILE_PATH

We also use the LOG_LEVEL and DJANGO_LOG_FILE_PATH environment properties in our Django settings:

settings.py

...
# basic logging with file rotation ()
log_level = os.getenv('LOG_LEVEL', 'INFO')
handlers = dict(file={'class': 'logging.handlers.TimedRotatingFileHandler',
                      'filename': os.getenv('DJANGO_LOG_FILE_PATH'),
                      'when': 'midnight',
                      'interval': 1,
                      'backupCount': 1,
                      'encoding': 'utf-8'})
loggers = dict(django=dict(level=log_level, handlers=['file']),
               myapp=dict(level=log_level, handlers=['file']))
LOGGING = dict(version=1,
               disable_existing_loggers=False,
               handlers=handlers,
               loggers=loggers)
...

Some notes:

• We normally specify custom formatters as well, but I left those out for clarity.

• The application itself can now be found on the EC2 instance in /var/app/current. Also see extending EB Linux platforms for more details.

• The application now runs as webapp with group webapp.

• eb ssh is your friend. See docs.

• We also use platform hooks to run Django's migrate and collectstatic commands, as described here.

EDIT:

As pointed out by @hax0 in the comments, file permission issues may arise if you try to run manage.py commands on the EC2 instance, using SSH, after deployment.

For example, when using eb ssh, you are logged in as ec2-user, but the log file is owned by the webapp user, and, by default, only the owner has write permission (644). Thus, when running python manage.py as ec2-user, you will get an error saying it cannot configure the log file handler because permission is denied.

A quick & dirty workaround is to change file permissions temporarily, e.g. using

sudo chmod 646 /tmp/django-app.log

Another workaround would be to run manage.py as the webapp user, for example like so:

sudo su - webapp <<'EOF'
source $(find /var/app/venv/*/bin/activate)
export $(/opt/elasticbeanstalk/bin/get-config --output YAML environment | 
         sed -r 's/: /=/' | xargs)
python3 /var/app/current/manage.py showmigrations
EOF

Answer 3

By default in elasticbeanstalk, you can see the django error logs here.

/var/log/httpd/error_log

Answer 4

There's a simple way that doesn't require any beanstalk configuration.

In your django settings under LOGGING set up a handler directed to the file '/opt/python/log/{log_file_name}'. The logs can then be accessed via the beanstalk environment menu under "Logs".

LOGGING = {
    ...,
    'handlers': {
        'logfile': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/opt/python/log/{log_file_name}',
        },
    },
    'loggers': {
        'debugger': {
            'level': 'DEBUG',
            'handlers': ['logfile'],
        'propagate': False,
    },
}

This location is stated in the documentation here:

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-instancelocation

Answer 5

If you're using gunicorn, then you can use this flag:

gunicorn --access-logfile=- 

Answer 6

Here is my solution on 64bit Amazon Linux 2023/4.0.2

commands:
  00_create_dir:
    command: mkdir -p /var/log/app-logs
  01_change_permissions:
    command: chmod g+s /var/log/app-logs
  02_change_owner:
    command: chown webapp:webapp /var/log/app-logs

and the log file setting in Django,

'handlers': {
    'console': {
        'level': 'INFO',
        'class': 'logging.StreamHandler',
    },
    'logfile': {
        'level': 'INFO',
        'class': 'logging.handlers.RotatingFileHandler',
        'filename': "/var/log/app-logs/django.log",
        'formatter': 'standard',
    },
},
'loggers': {
    '': {
        'handlers': ['console', 'logfile'],
        'level': 'INFO',
    },
}

Make sure that the log file has the correct ownership and group configuration, the owner and group should be webapp:wehapp as follows

[ec2-user@ip-171-11-1-8 app-logs]$ ll django.log
-rw-r--r--. 1 webapp webapp 1120 Jul 29 01:08 django.log

Answer 7

-- edit --

This answer was originally written for Amazon Linux AMI, which has now reached end-of-life.

To keeps things clear and separate, I wrote a new answer for Amazon Linux 2.

-- original answer --

Summary

The simplest solution, in my opinion, is to log to the /opt/python/log folder, as suggested by bewestphal and by @thierry-j (under steve-dunlop's answer).

That is also what the official AWS EB Python Sample Application does: see python-v1.zip source

The log file will then be included automatically when you request logs from EB.

This solution works out-of-the-box, without any modification of .ebextensions, as long as you do not call django-admin.py (or other django code) in your .ebextensions.

However, most apps do need to call django-admin.py in .ebextensions, e.g. in order to migrate. That will cause the log file to be created prematurely, with root owner and root group. This leads to permission errors, because the app runs as wsgi:wsgi.

This can be fixed by adding a new command, at the end of your container_commands, to remove the "premature" log file, e.g.:

container_commands:
  ...
  9999_remove_root_log_file:
    command: rm /opt/python/log/django.log
    ignoreErrors: true

Details below.

Background

On a standard pre-configured Amazon Linux/Python platform, which uses Apache with mod_wsgi (see AWS platform docs), the WSGIDaemonProcess for the Django app runs as user wsgi and group wsgi (see /etc/httpd/conf.d/wsgi.conf on your EC2 instance).

In addition, the default folder permissions for the /opt/python/log folder (on my standard EC2 instance) are: drwxrwxr-x 3 root wsgi 4096 Mar 5 14:08 .

That is, the wsgi group has all permissions (rwx), so the Django app (group wsgi) can create log files there.

This works, out-of-the-box, as demonstrated by the official AWS EB Python Sample Application (python-v1.zip).

However, if you do anything in your .ebextensions that causes the logging file-handler to be initialized (like calling django-admin.py), it will break.

Permission issues

Here's how using django-admin.py in .ebextensions breaks your log file permissions:

Elastic Beanstalk container_commands, in .ebextensions, are executed as the root user (see aws docs).

If you call django-admin.py in any of the container_commands, e.g. with collectstatic or migrate, that will cause your logging file handler(s) to be initialized. If the specified log file does not exist yet, at that time, it will be created, with root owner and root group.

That means the Django app, running as part of the wsgi group, will not have permission to write to the log file (which belongs to the root group).

This leads to permission errors, e.g.: PermissionError: [Errno 13] Permission denied: '/opt/python/log/django.log'

How to reproduce

The following snippet illustrates the permissions issue and shows how to fix it.

To reproduce the issue, add these container_commands to a clean project (e.g. following the AWS EB Django tutorial), configure Django settings.py to log to /opt/python/log/django.log, deploy to AWS EB, then check the eb-activity.log to see the output of the container commands.

...

container_commands:
  0100_show_current_user:
    # show that we are running as root user
    command: whoami
  0200_try_to_remove_log_file:
    # we need a clean slate for this example (make sure no log file owned by wsgi is present)
    command: rm /opt/python/log/django.log
    ignoreErrors: true
  0300_break_log_file_permissions:
    # this causes a new log file to be created, owned by root:root (instead of wsgi:wsgi)
    command: django-admin.py
  0400_show_log_file_permissions:
    # prove that a log file was created by root, and show folder permissions
    command: ls -la /opt/python/log
  0500_fix_by_removing_log_file_after_all_django_admin_calls:
    # remove the log file created by django-admin.py, to ensure that a new log file will  
    # be created when the server starts, owned by wsgi:wsgi
    command: rm /opt/python/log/django.log
    ignoreErrors: true

DRY solution

So, there is no need to mess with file/folder permissions explicitly.

If you don't call django code in .ebextensions, logging to /opt/python/log works, out-of-the-box.

If you do call django code in .ebextensions, e.g. django-admin.py collectstatic, simply remove the log file at the end of your container_commands section.

NOTE: If you want to log files to persist between deployments, only remove them if they are owned by root.

Here's a DRY example:

In .ebextensions config:

option_settings:
  # create EB environment property for the log file path
  aws:elasticbeanstalk:application:environment:
    LOG_FILE_PATH: /opt/python/log/django.log
...

container_commands:
  ...
  # django code called here, e.g. "django-admin.py collectstatic"
  ...
  9999_remove_any_existing_django_log_files:
    command: rm $LOG_FILE_PATH      
    ignoreErrors: true

and in settings.py:

...
# get log path from environment variable, with fallback for local development
log_file_path = os.getenv('LOG_FILE_PATH', 'local.log')
# use this as 'filename' for the file handler, as described in the other answers
...

Answer 8

I had a similar issue but on Elastic Beanstalk, so I created a config file (e.g. applogs.config) in .ebextensions folder of the app. This creates the app-logs folder if it is not there already and sets the file permissions and owner so that the app can write its logs there.

commands:
  00_create_dir:
    command: mkdir -p /var/log/app-logs
  01_change_permissions:
    command: chmod g+s /var/log/app-logs
  02_change_owner:
    command: chown wsgi:wsgi /var/log/app-logs

Finally, in your Django settings:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/var/log/app-logs/django.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

Aditionally, if you want your log to be accessible from beanstalk logs using the web, add this to your file in .ebextensions

files:
  "/opt/elasticbeanstalk/tasks/taillogs.d/django.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      /var/log/app-logs/django.log

Answer 9

As a beginner in terms of linux permissions, it took me some time to get it to work. Summarising the above given answers the following finally worked for me:

logging.config

commands:
  00_create_dir:
    command: mkdir -p /var/log/app-logs
  01_change_permissions:
    command: chmod g+s /var/log/app-logs
  02_change_default_owner:
    command: setfacl -d -m g::rw /var/log/app-logs
  03_change_owner:
    command: chown wsgi:wsgi /var/log/app-logs

settings.py

LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
    'file': {
        'level': 'DEBUG',
        'class': 'logging.FileHandler',
        'filename': '/var/log/app-logs/django.log',
    },
},
'loggers': {
    'django': {
        'handlers': ['file'],
        'level': 'DEBUG',
        'propagate': True,
    },
},
}

With this I can see the logs as a separate section using 'eb logs' or within the Beanstalk environment, section "logs".