How pagination affects performance

I am working on a kind of medium-size website where we have many posts (when I say many I talk about millions of posts).

Because posts are shown to users according to their preferences; the list of posts changes for every user. We used a Digg style pagination with the paginators provided by django, but the problem was that those paginators perform “SELECT COUNT(*) …” queries and when you have complex querysets and many rows; that queries are really expensive, making the complete application very slow.

Since we knew in this case users don’t care about the total number of objects or pages there was no need to use that “counts” so we decided to use django endless pagination http://code.google.com/p/django-endless-pagination/ Now our database is not exhausted anymore and the site is very quick now. Also, the installation and configuration was so easy that just took some minutes.

I’m really glad these guys built endless pagination, you should definitely give it a try.

Getting Random objects from a Queryset in Django

When providing related or suggested info to the user in a website, it’s a common practice to obtain a random set of items. To do this in django the “usual” way is:

Book.objects.all().order_by('?')[:10]


The above code, sorts all books in a random order and then picks the first 10 objects. This approach is not really efficient en MySQL. Since the “ORDER BY ?” clause is really expensive. So, a good way to obtain random elemets (actually a random slice) is the following:

import random
count = Book.objects.all().count()
slice = random.random() * (count - 10)
Book.objects.all()[slice: slice+10]


The above code selects a random position for starting a slice (the next items are going to be consecutive). So it’s not really random.

Another approach is to get the random ids from Python:

from random import sample
count = Book.objecs.all().count()
rand_ids = sample(xrange(1, count), 10)
Book.objects.filter(id__in=rand_ids)

This last solution works fine for consecutive ids, where there are no missing elements. But you can always re call it until you have all the items you want.

Django render_to_response

It is important to see how django doesn’t automatically appends the context variables to the request object.

Say you want to access MEDIA_URL in your templates, using the following code will not work since the context is not created and it is not passed to the render_to_response function

def my_view(request):
    return render_to_response('my_template.html',
                              my_data_dictionary)

The first solution is to build the context and pass it to the render_to_response function, like this:

def my_view(request):
    return render_to_response('my_template.html',
                              my_data_dictionary,
                              context_instance=RequestContext(request))

But, this is kind of repetitive and boring. So a django snippet was proposed for that in http://www.djangosnippets.org/snippets/3/

from django.shortcuts import render_to_response
from django.template import RequestContext

def render_response(req, *args, **kwargs):
    kwargs['context_instance'] = RequestContext(req)
    return render_to_response(*args, **kwargs)

That works, But django just implemented almost the same thing in generic views. So, the solution I prefer is using the direct_to_template generic view

def my_view(request):
    # View code here...
    return direct_to_tempalte(request,
                                        template_name='my_template.html',
                                        extra_context=some_dict)

New Django Rpx

Rpx is a great sig in center. With RPX users can login using their facebook, openid, google, myspace, aol, yahoo, blogger, flickr and some other types of accounts.

Some days ago I looked at django-rpx wich is pretty good but doesn´t cover som interesting rpx like mapping. Mapping allows a user to have multiple openids so if you don´t remeber what account to use for a specific site just register all your ids in the site and login with any of them. So I decide to create a brach of django-rpx callend django-newrpx

With this application you can use rpx sing very easy. You can add openids to your accoutns, and delete them. You can also retrive the list of all openids registered for a user in the rpx servers.

Give it a try

Django JQuery Autocomplete for Model Selection

I’ve been working on a very cool snippet, inspired by Django Autocomplete Widget :
http://www.djangosnippets.org/snippets/233/

The idea is to create a Widget containing the client side code (I mean html + jquery), and a custom field called ModelAutocompleteField that accepts any model as a contructor parameter and its clean method returns the instance selected by the user. So making it work would be as simple as …

from myapp.models import MyModel
from utils.fields import ModelAutocompleteField
class TestForm(forms.Form):
    customer = ModelAutocompleteField(model=MyModel, 
                               lookup_url='/ajax/model_list/')

Now this is working pretty cool but I would like to make some enhancements before publishing like adding the ability of attaching javascript or jquery functions to the item selection event.

MVC en .Net – Primeras Impresiones

Bueno hace mucho tiempo que queria iniciar un blog y que mejor manera de iniciarlo que hablando de nuestro nunca bien ponderado .Net. Nunca he sido fanático ni seguidor de las tecnologías de Microsoft pero tampoco las satanizo.

Luego de haber desarrollado algunas aplicaciones, tratando de aplicar algunas buenas prácticas en el desarrollo, siento que un patrón que mejora mucho el tiempo de desarrollo y ordena mucho las cosas es el Model View Controller (MVC), bueno actualmente trabajo con una de sus variantes Model View Template (MVT o MTV) como algunos le dicen. Ya que la necesidad me obligaba a usar .Net decidi buscar algun framework en .Net que permita aplicar dicho patron y si existía algún poderoso ORM como el de django pues mucho mejor. Bueno llegué a este site, los videos son muy interesantes. Sin embargo luego de revisarlos un poco pues ya no son tan interesantes. En primer lugar para verlos necesitan silverlight.

Bueno luego tener que pasarme a windows para que funcione silverlight y poder ver los videos me di con la sorpresa de que el desarrollo MVC en .Net no es imposible, el ORM descrito en los videos ayuda bastante. Pero no esta ni cerca del ORM de django o incluso el de Cakephp. Creo que aun le falta mucho. Por otro lado sus controladores se basan en nombres de archivos, es decir si ejecuto Posts/Add entonces se llamara al metodo add del controlador Posts, lo cual me parece un poco inutil ya que seria mucho mas facil poder plantear urls como expresiones regulares y mapearlas a fuciones como lo hace django.

Definitivamente estas herramientas alivian mucho el desarrollo en .Net sin embargo creo que aun estan en un nivel bastante basico y no las considero del todo “usables” … solo espero pronto se haga algun hack de .Net para que soporte python 🙂