Login

iter_fetchmany

Author:
jdunck
Posted:
January 8, 2009
Language:
Python
Version:
1.0
Score:
0 (after 0 ratings)

When executing custom sql, the temptation is to use fetchall or fetchone, since the API for fetchmany is a bit awkward. (fetchall makes all records resident in client memory at once; fetchone takes a network round-trip to the DB for each record.)

This snippet, hoisted from django.db.models.sql.query.results_iter, presents a nice, simple iterator over multiple fetchmany calls which hits a sweet spot of minimizing memory and network usage.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
def iter_fetchmany(cursor, size=100): 
    """ 
    Handles the awkwardness of iterating using fetchmany, and exposes all records as a (generated) iterable. 
 
    Assumes django.db is the connection for the given cursor.  FIXME for multi-db support. 
    """ 
    from django.db import connection 
    rowset_iter = iter((lambda: cursor.fetchmany(size)), 
            connection.features.empty_fetchmany_value) 
    for rows in rowset_iter: 
        for row in rows: 
            yield row 

More like this

  1. Template tag - list punctuation for a list of items by shapiromatron 2 months, 2 weeks ago
  2. JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 2 months, 3 weeks ago
  3. Serializer factory with Django Rest Framework by julio 9 months, 2 weeks ago
  4. Image compression before saving the new model / work with JPG, PNG by Schleidens 10 months, 1 week ago
  5. Help text hyperlinks by sa2812 11 months ago

Comments

Please login first before commenting.