I have a production web app running on NextJS 13 using Prisma 5 for ORM, connecting to a MySQL 8 server. Been running reliably for the last 3 years with periods of heavy traffic. Recently, after a marketing announcement, we started receiving heavy traffic for a few days. Traffic does hit the DB for writes, as the marketing call to action involves filling out a form which gets stored in our DB. There were several periods where Prisma would begin failing with this error, and could not recover until a manual reboot:
Timed out fetching a new connection from the connection pool. More info: http://pris.ly/d/connection-pool (Current connection pool timeout: 10, connection limit: 5)
This is a pretty common Prisma error that can have many different causes. I have properly tuned the connection pool at various times in the past. After some troubleshooting the last few days and tinkering and no resolution, I realized that this error was happening almost exactly every 8 hours, breaking the web app until a reboot. I also realized that 8 hours is the default idle connection timeout for MySQL, and this is also true of our hosted DB provider. I know that the Prisma connection pool keeps DB connections open so that it can reuse them for multiple queries for efficiency and to use up fewer total connections to the DB.
Is it possible that the Prisma connection pool is failing catastrophically after the MySQL server kills these "idle" pooled connections after the 8 hour timeout period? I cannot find any documentation about this or any reports of similar behavior.