Problem Description: I am attempting to create a websocket through a web application in an AWS environment (Beanstalk). The components involved are: Django Channels, Daphne, and Redis (Redis ElastiCache). We are using POSTMAN as a client (although we also use Python scripts), and this is our URL: wss://api.?????.com/ws/tunnel/group_test/.
The client can connect to the socket without any issues and receives the welcome message. This is the consumer handler responsible for it:
class ChatConsumer(AsyncWebsocketConsumer):
async def connect(self):
try:
self.room_name = self.scope['url_route']['kwargs']['comm_name']
self.room_group_name = f'chat_{self.room_name}'
# Unirse a un grupo de chat
await self.channel_layer.group_add(
self.room_group_name,
self.channel_name
)
await self.accept()
welcome_message_1 = f'Bienvenido al WebSocket 1. Tu canal es {self.channel_name}'
await self.chat_message({'message': welcome_message_1})
As we mentioned, the client successfully connects, but when sending a message to an existing group (otherwise, it does not produce an error, which in itself is a clue), we receive the following error message from Daphne:
2024-01-16 03:46:40,053 No-IP daphne.server ERROR Exception inside application: unpack(b) received extra data. Traceback (most recent call last):
(full log below)
This is the consumer handler:
async def receive(self, text_data):
try:
text_data_json = json.loads(text_data)
message = text_data_json['message']
# Enviar mensaje al grupo de chat
await self.channel_layer.group_send(
self.room_group_name,
{
"type": "chat_message",
"message": message
}
)
The log error occurs during the await.
Initially, I thought there was a problem with the data I was sending to the operation, and that's why unpack(b) was not performing its job well (in fact, I created a thread that I will have to close regarding this). But now I understand that while this could be a first error, the problem might come upstream, perhaps in the way Redis and Django communicate. I haven't been able to find information in the Django release notes about this.
We work with AWS Redis ElastiCache, which should be well configured, as the connection takes place and the client correctly receives the welcome message. (in addition to the typical write/read variable checks). However, it is possible that in a more complex operation (message broadcasting), they don't communicate well with Daphne…or Django-channels, and this is where I get a bit lost as I don't know the exact way they communicate with each other. The LOG does not provide further information.
QUESTION:
Has anyone ever successfully configured Django channels in AWS Beanstalk environments? Is it possible?
Attempts at Resolution:
PRIOR ISSUE A prior issue that may be a clue is that in the configuration of CHANNELS_LAYERS, I used as backend:
'BACKEND': 'channels_redis.core.RedisChannelLayer'
But for some reason, this caused immediate disconnection after connection (the client connected and disconnected after a second) due to an error in the lua scripts: 2024-01-13 15:48:13,640 No-IP daphne.server ERROR Exception inside application: Lua scripts without any input keys are not supported.
However, when changing the backend to pubsub.RedisPubSubChannelLayer, this problem disappeared and the connection is now persistent.
TESTS PERFORMED All the mentioned tests WORK CORRECTLY in a local test environment, with the same requirements.txt as the AWS environment.
We hardcoded the message sent. Same error reported in the log. We created a Python command to publish a message to the group (in this case, the client is the instance itself, so we rule out permission issues). Same error reported in the log. We created a REST view to receive the message sending request and through this, we invoked the handler for a specific channel. IMPORTANT: With this, we rule out a problem in the group send handler, as we use the same handler that the connection process uses (chat_message), which works correctly in AWS. Same error reported in the log. Context:
Framework and Versions:
daphne==4.0.0 Django==4.2.6 channels==4.0.0 channels-redis==4.1.0 django-redis==5.4.0
Channel layers in settings.py
CHANNEL_LAYERS = {
'default': {
'BACKEND': 'channels_redis.pubsub.RedisPubSubChannelLayer',
'CONFIG': {
"hosts": [os.environ.get('REDIS_CHANNEL_LAYER_URL')],
},
},
} REDIS_CHANNEL_LAYER_URL is defined in .env as : rediss://cacheback-mxxx54.serverless.euw3.cache.amazonaws.com:6379
ENVIRONMENT DETAILS (staging-LQM1lest)
We have a load balancer to redirect traffic to a WSGI server (port 8000 where everything works as expected) and the wss traffic to an ASGI server (Daphne) for asynchronous management. The security groups are configured correctly and allow traffic as expected. The Elastic Redis cache service has been enabled and is working correctly (we persist and retrieve information without any problem).
Initially, as the Backend for CHANNEL_LAYERS, channels_redis.core.RedisChannelLayer was selected, but the connection was not persistent (the client connected and disconnected after a second), which was solved by selecting channels_redis.pubsub.RedisPubSubChannelLayer.
I have minimized any chance of problem, so, under .ebextensions, I only have django.config, which content is this:
option_settings:
aws:elasticbeanstalk:application:environment:
DJANGO_SETTINGS_MODULE: "appProject.settings"
aws:elasticbeanstalk:container:python:
WSGIPath: appProject.wsgi:application
commands:
set_locale:
command: "localedef -i es_ES -f UTF-8 es_ES.UTF-8 && export
LC_ALL=es_ES.UTF-8 && export LANG=es_ES.UTF-8"
LOG Traceback:
2024-01-16 03:46:33,676 No-IP daphne.ws_protocol DEBUG WebSocket ['172.31.9.79', 52020] open and established 2024-01-16 03:46:33,676 No-IP daphne.ws_protocol DEBUG WebSocket ['172.31.9.79', 52020] accepted by application 2024-01-16 03:46:33,676 No-IP daphne.ws_protocol DEBUG Sent WebSocket packet to client for ['172.31.9.79', 52020] 2024-01-16 03:46:33,677 No-IP daphne.ws_protocol DEBUG Sent WebSocket packet to client for ['172.31.9.79', 52020] 2024-01-16 03:46:39,226 No-IP daphne.ws_protocol DEBUG WebSocket incoming frame on ['172.31.9.79', 52020] 2024-01-16 03:46:40,053 No-IP daphne.server ERROR Exception inside application: unpack(b) received extra data. Traceback (most recent call last): File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/routing.py", line 62, in call return await application(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/sessions.py", line 47, in call return await self.inner(dict(scope, cookies=cookies), receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/sessions.py", line 263, in call return await self.inner(wrapper.scope, receive, wrapper.send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/auth.py", line 185, in call return await super().call(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/middleware.py", line 24, in call return await self.inner(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/routing.py", line 116, in call return await application( ^^^^^^^^^^^^^^^^^^ File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/consumer.py", line 94, in app return await consumer(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/consumer.py", line 58, in call await await_many_dispatch( File "/var/app/venv/staging-LQM1lest/lib64/python3.11/site-packages/channels/utils.py", line 57, in await_many_dispatch await task msgpack.exceptions.ExtraData: unpack(b) received extra data. 2024-01-16 03:46:40,289 No-IP daphne.ws_protocol DEBUG WebSocket closed for ['172.31.9.79', 52020]