- January 9, 2017
- Not specified
- 1 (after 1 ratings)
The function slices a queryset into smaller querysets containing chunk_size objects and then yield them. It is used to avoid memory error when processing huge queryset, and also database error due to that the database pulls whole table at once. Concurrent database modification wouldn't make some entries repeated or skipped in this process.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
def chunked_queryset(queryset, chunk_size): """ Slice a queryset into chunks. """ start_pk = 0 queryset = queryset.order_by('pk') while True: # No entry left if not queryset.filter(pk__gt=start_pk).exists(): break try: # Fetch chunk_size entries if possible end_pk = queryset.filter(pk__gt=start_pk).values_list( 'pk', flat=True)[chunk_size - 1] # Fetch rest entries if less than chunk_size left except IndexError: end_pk = queryset.values_list('pk', flat=True).last() yield queryset.filter(pk__gt=start_pk).filter(pk__lte=end_pk) start_pk = end_pk
More like this
- Image compression before saving the new model / work with JPG, PNG by Schleidens 4 days, 13 hours ago
- Help text hyperlinks by sa2812 1 month ago
- Stuff by NixonDash 3 months ago
- Add custom fields to the built-in Group model by jmoppel 5 months, 1 week ago
- Month / Year SelectDateWidget based on django SelectDateWidget by pierreben 8 months, 3 weeks ago
Please login first before commenting.