Django's query result caching

Question

I am creating a website using Django 1.7 and GeoDjango. I've hit the point when i need to optimize website speed.

One of the bottlenecks is query execution. There are some queries which run slowly even when optimized. So i'd like to cache query results and store them in Redis.

The problem that i am getting is that i cannot cache some query results. Particularly the ones containing geometry types and distance calculations. I hit "TypeError: can't pickle Binary objects" error.

What is the recommended/right way of caching Django/GeoDjango QuerySets ?

Answer 1

Turns out the main problems in storing querysets are that:

1. QuerySets are lazy
2. To evaluate them you need to serialize them [link]
3. Not all QuerySets can be serialized because Python's serializer (Pickle) has it's own limitations [link]

The best solution i found is to cache query results in template.

So in my template "sample.html" i write something like:

{% cache 600 slow_query_results %}
<!-- result of page generation -->
{% endcache %}

And in view i do:

from django.core.cache import cache
from django.core.cache.utils import make_template_fragment_key
...
slow_query_results_key = make_template_fragment_key('slow_query_results')
if not cache.get(slow_query_results_key):
    # return calculated result
    slow_query_results = perform_some_slow_query()

This method is fine because data stored in cache is in expected text form. So there should be no problems/exceptions while storing data.

The main drawbacks are that:

1. Cache might contain repetitive similar data. This could happen when you cache html fragment containing language translation strings and so on. So under some circumstances you'll have to use language as a parameter for generating cache. And if you have translations for 2 languages you'll have 2 caches of the same data.

2. You'll have to invalidate cache in situations when you make changes to your html. This could become a real pain if the html in the block of code you are caching is continuously changing.

I personally think that problem 1) is not a big deal. The problem 2) could be avoided by good planning of site structure and knowing that you can do massive invalidation of cache keys in Redis. [link] This is possible because cache is stored in the following key format: ":1:template.cache.slow_query.8a5b358dfc28a6bc1b3397e398d28b66"

So it should be possible to delete all cache keys related to some caching block.