My Django webapp deploys fine, however my db connections max out after a few queries to Postgres.
Here is my understanding of my problem: Db connections to Postgres are opened for every query but not closed and this results is a db timeout once max connections is reached.
Heroku documentation for this error, but I think that whatever new number of connections are enabled through an upgrade, those connections will also max out fast. (Or am I wrong?)
pg:killall is not a solution to my problem. The solution must not be manual in this application.
I have used Heroku documentation for Concurrency and DB connection in Django
This SO question is related but answers don't solve my problem
Debug error
FATAL: too many connections for role "DB_username"
Error traceback failure point
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Running heroku pg:info --app appname in cmd
=== DATABASE_URL
Plan: Hobby-dev
Status: Available
Connections: 20/20 <<<<<<<<<<<<<<<<<<<<<
PG Version: 12.4
Created: 2020-09-11 16:57 UTC
Data Size: 9.3 MB
Tables: 15
Rows: 85/10000 (In compliance)
Fork/Follow: Unsupported
Rollback: Unsupported
Continuous Protection: Off
Add-on: postgresql-infinite-00894
Here are the things I have tried:
- setting dj_database_url.config(conn_max_age==0,...)
- Using Pgbouncer to pool connections
- Disabling server side cursors
Settings.py
# PRODUCTION SETTINGS
MIDDLEWARE.append('whitenoise.middleware.WhiteNoiseMiddleware')
DATABASE_URL = os.environ['DATABASE_URL']
DISABLE_SERVER_SIDE_CURSORS = True
SECRET_KEY = os.environ.get('SECRET_KEY')
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
#HEROKU SETTINGS
DATABASES['default'] = dj_database_url.config(conn_max_age=0, ssl_require=True)
django_heroku.settings(locals())
Installed pgbouncer buildpack
Procfile
web: bin/start-pgbouncer
web: daphne appname.asgi:application --port $PORT --bind 0.0.0.0 -v2
chatworker: python manage.py runworker --settings=appname.settings -v2
This is the only Procfile that worked for me, I've seen people on other forums complain that you can't do 2 web dynos, this works for me, pgbouncer launches successfully at deployment, daphne works too. Using web and web2 crashes app. That being said I am open to trying any and all suggestions.
asgi.py in case you're curious
import os
import django
from channels.routing import get_default_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "Stonks.settings")
django.setup()
application = get_default_application()
The documentation around this is very confusing and other SO/code forums haven't suggested anything I haven't tried. Clarifying whether this is a Django code issue, Heroku config issue, or Daphne issue would be tremendously helpful as well.
consumers.py sample
class StatsConsumer(AsyncConsumer):
"""Creates your stats websocket for each connected user"""
async def websocket_connect(self, event):
"""Connects to HTML and JS websocket for your stats"""
await self.send({
"type": "websocket.accept"
})
player = self.scope['user']
player_bids_list = await self.get_bids(player)
result_dict = await self.find_stats(player_bids_list)
await self.send({
"type": "websocket.send",
"text": json.dumps(result_dict),
})
async def websocket_receive(self, event):
"""Is called when yourStats.html pings server for updated data"""
player = self.scope['user']
player_bids_list = await self.get_bids(player)
result_dict = await self.find_stats(player_bids_list)
await self.send({
"type": "websocket.send",
"text": json.dumps(result_dict),
})
async def websocket_disconnect(self, event):
"""Is called when websocket disconnects"""
print("disconnected", event)
@database_sync_to_async
def get_profile(self, user):
"""fetches user profile funds asynchronously"""
return Profile.objects.get(user=user).funds
@database_sync_to_async
def get_bids(self, user):
"""fetches the connected user's bids and returns all bids based on id, asynchronously"""
userid = User.objects.get(username=user).id
result = Bid.objects.filter(username_id=userid)
returnlist = []
for i in result:
returnlist.append(i.amount)
return returnlist