Is there a way to optimize Large data set to load fast in Django

Question

I have a model with over 6 millions object and anytime I am accessing the object page in my admin page its load very slow it even runs to time out sometimes and also the same thing to my view so I would like to get help if there is any recommendations method I can use for the issue. The project is hosted in AWS elastic beanstalk.

The database configuration is attached to the Post

Thanks in advance

Database configuration The database object

I have increased the database resources but still the same.

From db.t2.medium To db.t3.medium

Answer 1

One of the initial and must have practice would be to use pagination in your API that gets admin-page data from your model that contain large number of records

Answer 2

Slow Pagination in Django Admin

The Django Admin interface can suffer from performance issues with large datasets. This is because it often relies on a COUNT query to determine the total number of records in a table, which can be slow for millions of entries.

Identifying the Bottleneck

To pinpoint the issue, you can:

1. Run your project locally: Set up a local development environment (with a copy of your large table data) and run your Django application.
2. Install Django Debug Toolbar: This helpful tool provides insights into database queries and their execution times. Django Debug Toolbar: https://django-debug-toolbar.readthedocs.io/
3. Inspect Query Performance: Use the Debug Toolbar to see how long the COUNT query takes to execute.

Optimizing Pagination

If you're experiencing slow performance due to the COUNT query, consider using a cursor-based pagination approach. This avoids the need to count all records upfront and can significantly improve performance for large tables.

I personally use the django-admin-cursor-paginator package (https://pypi.org/project/django-admin-cursor-paginator/) in my project for a table of over 30k records.