Django snippets: Upload progress handler using cache framework

from django.core.files.fileuploadhandler import FileUploadHandler
from django.core.cache import cache

class UploadProgressCachedHandler(FileUploadHandler):
    """
    Tracks progress for file uploads.
    The http post request must contain a header or query parameter, 'X-Progress-ID'
    which should contain a unique string to identify the upload to be tracked.
    """

    def __init__(self, request=None):
        super(UploadProgressCachedHandler, self).__init__(request)
        self.progress_id = None
        self.cache_key = None

    def handle_raw_input(self, input_data, META, content_length, boundary, encoding=None):
        self.content_length = content_length
        if 'X-Progress-ID' in self.request.GET :
            self.progress_id = self.request.GET['X-Progress-ID']
        elif 'X-Progress-ID' in self.request.META:
            self.progress_id = self.request.META['X-Progress-ID']
        if self.progress_id:
            self.cache_key = "%s_%s" % (self.request.META['REMOTE_ADDR'], self.progress_id )
            cache.set(self.cache_key, {
                'length': self.content_length,
                'uploaded' : 0
            })

    def new_file(self, field_name, file_name, content_type, content_length, charset=None):
        pass

    def receive_data_chunk(self, raw_data, start):
        if self.cache_key:
            data = cache.get(self.cache_key)
            data['uploaded'] += self.chunk_size
            cache.set(self.cache_key, data)
        return raw_data
    
    def file_complete(self, file_size):
        pass

    def upload_complete(self):
        if self.cache_key:
            cache.delete(self.cache_key)



# A view to report back on upload progress:

from django.core.cache import cache
from django.http import HttpResponse, HttpResponseServerError 

def upload_progress(request):
    """
    Return JSON object with information about the progress of an upload.
    """
    progress_id = ''
    if 'X-Progress-ID' in request.GET:
        progress_id = request.GET['X-Progress-ID']
    elif 'X-Progress-ID' in request.META:
        progress_id = request.META['X-Progress-ID']
    if progress_id:
        from django.utils import simplejson
        cache_key = "%s_%s" % (request.META['REMOTE_ADDR'], progress_id)
        data = cache.get(cache_key)
        return HttpResponse(simplejson.dumps(data))
    else:
        return HttpResponseServerError('Server Error: You must provide X-Progress-ID header or query param.')

Ticket #2070 allows you to create your own file upload handlers. Here's an example handler that tracks a file's upload so you might display a progress meter after form submission.

The snippet has two parts, the upload handler which tracks progress, and an upload_progress view used to report back to the browser.

The upload handler uses django's cache framework to store the data, which can then be retrieved by the view to send back to the browser.

Note: Your form's http post request must have a query parameter (X-Progress-ID) sent along with it, which should contain a unique key to identify the upload. That key will also be sent with your ajax requests to the upload_progress view to retrieve the progress data.

Setup: place the UploadProgressCachedHandler anywhere you like on the python path, and add to your settings.py:

from django.conf import global_settings 
FILE_UPLOAD_HANDLERS = ('path.to.UploadProgressCachedHandler', ) + \
    global_settings.FILE_UPLOAD_HANDLERS

Set up the upload_progress view in any of your apps along with a corresponding entry in your urlconf.

Here's some javascript example code to make the ajax requests and display the progress meter: http://www.djangosnippets.org/snippets/679/

Author:: ebartels
Posted:: April 2, 2008
Language:: Python
Tags:: ajax progress upload
Score:: 5 (after 5 ratings)

Tools

Comments

kayamb (on May 2, 2008):

Hi,

Thanks for those 2 snippets, they look very promising!

However, I can't quite figure out how to use them. In particular what kind of view should the html form's action attribute point to, and how would that view handle the form? Should that view wait for the completion of the upload before returning anything?

Here follows my first shot at making that view. Is that a good start, and if so, what is missing?

def frontpage(request):
    if request.method == 'POST':
        # [????] I don't know what to put there!...
    else:
        form = UploadForm()
    return render_to_response('frontpage.html', { 'form': form }, context_instance=RequestContext(request))

Thanks a lot!

Julien

kayamb (on May 3, 2008):

Never mind the comment above, I think I hadn't applied the patch properly.

revolunet (on July 10, 2008):

Hello and thanx a lot for these great two samples. Is it compatible with actual trunk ?

Can you show an example of the view who 'starts' the upload please ? i cannot figure it :(

can i use request.upload_handlers.insert(0, UploadProgressCachedHandler()) in my view instead of modify the settings.py ?

revolunet (on July 17, 2008):

hello

ok after some hours i finally made it work with some changes :

im using apache2+mod_wsgi with latest trunk

i use session instead of cache middleware, with a unique sessionkey for each upload ID. i have to use self.request.session.save() in the receive_data_chunk to update the data for the progress view.

instead of tweaking settings.py, i add in my view :

if request.method == 'POST':
    request.upload_handlers.insert(0, UploadProgressSessionHandler(request))

Thanks !

in my upload_file_handler, i had to add

os.remove(f.temporary_file_path())

so the temporary file in /tmp can be removed

tuttle (on July 26, 2008):

Thanks for this snippet!

Anyway, I think it is worth mentioning that this will

1) NOT currently work with Django devel webserver (cannot serve multiple requests at once),

2) NOT work with a cache not sharing data between processes such as locmem://.

Finally I made this to work on prefork Apache with memcached://.

Django snippets

Upload progress handler using cache framework

Tools

Comments

Snippets

About