djangosnippets.org: Latest snippets tagged with 'memoryerror'http://djangosnippets.org/tags/memoryerror/2017-01-09T08:59:51.123659-06:00Django chunked queryset iterator
2017-01-09T08:59:51.123659-06:00mingdongthttp://djangosnippets.org/snippets/10599/<p>The function slices a queryset into smaller querysets containing chunk_size objects and then yield them.
It is used to avoid memory error when processing huge queryset, and also database error due to that the database pulls whole table at once.
Concurrent database modification wouldn't make some entries repeated or skipped …</p>
Freely redistributablestreaming dump_data
2009-03-29T21:03:52.979071-05:00kcarnoldhttp://djangosnippets.org/snippets/1402/<p><code>dumpdata</code> without <code>MemoryErrors</code>, with progress notification. Most of the real work is done by snippets <a href="http://www.djangosnippets.org/snippets/1400/">1400</a> and <a href="http://www.djangosnippets.org/snippets/1401/">1401</a>.</p>
<pre><code>./manage.py dumpdata_stream --format=xml > big_dump.xml
</code></pre>
<p>This is basically the stock Django <code>dumpdata</code> with a few modifications. Django devs: it's hard to reuse parts of most Django management commands. A little …</p>
Freely redistributablestreaming serializer
2009-03-29T20:55:02.602772-05:00kcarnoldhttp://djangosnippets.org/snippets/1401/<p>Trying <code>./manage.py dumpdata</code> on a huge database and getting <code>MemoryError</code>s? Here's part of your solution.
</p>
<p><a href="http://www.djangosnippets.org/snippets/1400/">Snippet 1400</a> provides a queryset_foreach utility that we've found very useful. This snippet uses it on a serializer that can output to a stream, such as the XML serializer.
</p>
<p>Management command coming momentarily...
</p>
Freely redistributable