- Author:
- mingdongt
- Posted:
- January 9, 2017
- Language:
- Python
- Version:
- Not specified
- Score:
- 1 (after 1 ratings)
The function slices a queryset into smaller querysets containing chunk_size objects and then yield them. It is used to avoid memory error when processing huge queryset, and also database error due to that the database pulls whole table at once. Concurrent database modification wouldn't make some entries repeated or skipped in this process.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | def chunked_queryset(queryset, chunk_size):
""" Slice a queryset into chunks. """
start_pk = 0
queryset = queryset.order_by('pk')
while True:
# No entry left
if not queryset.filter(pk__gt=start_pk).exists():
break
try:
# Fetch chunk_size entries if possible
end_pk = queryset.filter(pk__gt=start_pk).values_list(
'pk', flat=True)[chunk_size - 1]
# Fetch rest entries if less than chunk_size left
except IndexError:
end_pk = queryset.values_list('pk', flat=True).last()
yield queryset.filter(pk__gt=start_pk).filter(pk__lte=end_pk)
start_pk = end_pk
|
More like this
- Template tag - list punctuation for a list of items by shapiromatron 9 months ago
- JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 9 months, 1 week ago
- Serializer factory with Django Rest Framework by julio 1 year, 4 months ago
- Image compression before saving the new model / work with JPG, PNG by Schleidens 1 year, 4 months ago
- Help text hyperlinks by sa2812 1 year, 5 months ago
Comments
Please login first before commenting.