The function slices a queryset into smaller querysets containing chunk_size objects and then yield them. It is used to avoid memory error when processing huge queryset, and also database error due to that the database pulls whole table at once. Concurrent database modification wouldn't make some entries repeated or skipped in this process.
This snippet is based on [Bill Freeman's MultiSelect checkbox iterator template filter](http://djangosnippets.org/snippets/2151/). Usage: See docstring
While checking up on some cronjobs at [YouTellMe](http://www.youtellme.nl/) we had some problems with large cronjobs that took way too much memory. Since Django normally loads all objects into it's memory when iterating over a queryset (even with .iterator, although in that case it's not Django holding it in it's memory, but your database client) I needed a solution that chunks the querysets so they're only keeping a small subset in memory. Example on how to use it: `my_queryset = queryset_iterator(MyItem.objects.all()) for item in my_queryset: item.do_something()` [More info on my blog](http://www.mellowmorning.com/2010/03/03/django-query-set-iterator-for-really-large-querysets/)
Call a function for each element in a queryset (actually, any list). Features: * stable memory usage (thanks to Django paginators) * progress indicators * wraps batches in transactions * can take managers or even models (e.g., `Assertion.objects`) * warns about `DEBUG`. * handles failures of single items without dying in general. * stable even if items are added or removed during processing (gets a list of ids at the start) Returns a `Status` object, with the following interesting attributes * `total`: number of items in the queryset * `num_successful`: count of successful items * `failed_ids`: list of ids of items that failed
4 snippets posted so far.