Django Admin Performance Tips

For when you have millions of rows.

Deferring the loading of fields

With each iteration of Django the admin site becomes more configurable. Django 1.2.x included additional functions in the ORM that allow developers the opportunity to tune the performance of their admin sites namely the defer(*fields) method and the ModelAdmin.queryset(self, request) method.

If a model contains many fields or fields that contain a lot of data (e.g. TextFields) and that data is not needed in that model's change list page there is an opportunity to make massive performance improvements by using the defer(*fields) method. The defer(*fields) method essentially lazy loads those fields, which greatly reduces the amount of data that must be brought back from the database, thereby improving the response time of the page.

There are three steps that must be taken in order to ensure that during this implementation the performance of your main site is not negatively impacted:

  1. Create a custom model manager for the model in question if it does not already have one, and set the model to use the new manager
  2. Override the get_query_set method in the manager to use the defer(*fields) method
  3. Override the ModelAdmin.queryset(self, request) method for the model in question to use the new model manager

This example uses a model that could be used to view logs that are recorded to a database. First create the class. 

class Log(models.Model):
    id = models.IntegerField(primary_key=True, db_column='ID') 
    date = models.DateTimeField(db_column='Date')
    thread = models.CharField(max_length=96, db_column='Thread') 
    context = models.CharField(max_length=1536, db_column='Context') 
    level = models.CharField(max_length=1536, db_column='Level') 
    host = models.CharField(max_length=765, db_column='Host', blank=True)
    logger = models.CharField(max_length=1536, db_column='Logger') 
    args = models.TextField(max_length=1533, db_column='Args', blank=True) 
    message = models.TextField(max_length=12000, db_column='Message')
    exception = models.TextField(max_length=6000, db_column='Exception', blank=True) 
    
    objects = models.Manager()
    admin = LogManager() #Custom manager
    
    
    class Meta:
        db_table = u'log'

Next create the custom manager. Here we are deferring the loading of all of the fields of type TextField until they are needed. This means that when the change list page for the Log model is viewed these fields will not be loaded from the database. 

class LogManager(models.Manager):
    def get_query_set(self):
        return super(LogManager, self).get_query_set().defer("message", "context", "exception")

Finally, the model for the admin is created. 

class LogAdmin(admin.ModelAdmin):
    def queryset(self, request):
        qs = Log.admin #Use the admin manager regardless of what the default one is
        
        # we need this from the superclass method
        ordering = self.ordering or ()
        if ordering:
            qs = qs.order_by(*ordering)
        return qs

Adding Indexes

Another area where performance improvements can be made is adding database indexes on fields that are in the ModelAdmin.list_filter variable. A quick look at the queries triggered by the loading of the change list for the Log model in the Django Debug Toolbar shows that each entry in the ModelAdmin.list_filter variable causes a SELECT DISTINCT on that column to be run. Depending on the size of the table this could be a very costly operation.

Adding an index can greatly improve the load time of these queries, but caution should be taken as adding too many indexes can impede INSERT throughput into that table. Below is a list of references on how to create indexes in various databases:

References:

blog comments powered by Disqus