Django snippets: Manager method for limiting GenericForeignKey queries

from django.db import models

class GFKManager(models.Manager):
    def relate(self, qs):
        model_map = {}
        item_map = {}
        for item in qs:
            model_map.setdefault(item.content_type, {}) \
                [item.object_id] = item.id
            item_map[item.id] = item
        for ct, items_ in model_map.items():
            for o in ct.model_class().objects.select_related() \
                    .filter(id__in=items_.keys()).all():
                item_map[items_[o.id]].content_object = o
        return qs

This is a simple manager that offers one additional method called relate, which fetches generic foreign keys (as referenced by content_type and object_id fields) without requiring one additional query for each contained element.

Basically, when working with generic foreign keys (and esp. in the usecase of having something like a tumblelog where you use an additional model just to have a single sorting point of multiple other models), don't do something like result = StreamItem.objects.select_related() but just fetch the content type with result = StreamItem.objects.select_related('content_type'), otherwise you will end up with first one query for the list of StreamItems but then also with one additional query for each item contained in this resultset.

When you now combine the latter call with result = StreamItem.gfkmanager.relate(result), you will just get the one query for the item list + one query for each content type contained in this list (if the models have already been cached).

For further details, please read this post on my blog.

Author:: zerok
Posted:: August 15, 2008
Language:: Python
Tags:: foreignkey generic manager query tuning
Score:: 7 (after 7 ratings)

Tools

Comments

carljm (on September 22, 2008):

This is really nice. A few issues I've observed with it:

1) Referring to item.content_type causes n queries to the ContentType table which bypass its lookup cache. This could easily be addressed by storing the content type id instead and using ContentType.objects.get_for_id, which goes through the cache.

2) This snippet makes assumptions about the names of the content_type and object_id fields used for the GenericForeignKey. It would be more robust to pull this information from the GFK itself.

3) This would be easier to use if it were a method on a queryset of the model with a GFK, so you could just say "StreamItem.objects.filter(...).fetch_generic_relations()" and have it done in one command.

I've posted a version that addresses these issues here.

Django snippets

Manager method for limiting GenericForeignKey queries

Tools

Comments

Snippets

About