Incredibly useful for storing just about anything in the database (provided it is Pickle-able, of course) when there isn't a 'proper' field for the job:
A field which can store any pickleable object in the database. It is database-agnostic, and should work with any database backend you can throw at it.
Pass in any object and it will be automagically converted behind the scenes, and you never have to manually pickle or unpickle anything. Also works fine when querying.
Please note that this is supposed to be two files, one fields.py and one tests.py (if you don't care about the unit tests, just use fields.py)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 | # --------------------------------------- fields.py --------------------------------------- #
from django.db import models
try:
import cPickle as pickle
except ImportError:
import pickle
class PickledObject(str):
"""A subclass of string so it can be told whether a string is
a pickled object or not (if the object is an instance of this class
then it must [well, should] be a pickled one)."""
pass
class PickledObjectField(models.Field):
__metaclass__ = models.SubfieldBase
def to_python(self, value):
if isinstance(value, PickledObject):
# If the value is a definite pickle; and an error is raised in de-pickling
# it should be allowed to propogate.
return pickle.loads(str(value))
else:
try:
return pickle.loads(str(value))
except:
# If an error was raised, just return the plain value
return value
def get_db_prep_save(self, value):
if value is not None and not isinstance(value, PickledObject):
value = PickledObject(pickle.dumps(value))
return value
def get_internal_type(self):
return 'TextField'
def get_db_prep_lookup(self, lookup_type, value):
if lookup_type == 'exact':
value = self.get_db_prep_save(value)
return super(PickledObjectField, self).get_db_prep_lookup(lookup_type, value)
elif lookup_type == 'in':
value = [self.get_db_prep_save(v) for v in value]
return super(PickledObjectField, self).get_db_prep_lookup(lookup_type, value)
else:
raise TypeError('Lookup type %s is not supported.' % lookup_type)
# --------------------------------------- tests.py --------------------------------------- #
# -*- coding: utf-8 -*-
"""Unit testing for this module."""
from django.test import TestCase
from django.db import models
from fields import PickledObjectField
class TestingModel(models.Model):
pickle_field = PickledObjectField()
class TestCustomDataType(str):
pass
class PickledObjectFieldTests(TestCase):
def setUp(self):
self.testing_data = (
{1:1, 2:4, 3:6, 4:8, 5:10},
'Hello World',
(1, 2, 3, 4, 5),
[1, 2, 3, 4, 5],
TestCustomDataType('Hello World'),
)
return super(PickledObjectFieldTests, self).setUp()
def testDataIntegriry(self):
"""Tests that data remains the same when saved to and fetched from the database."""
for value in self.testing_data:
model_test = TestingModel(pickle_field=value)
model_test.save()
model_test = TestingModel.objects.get(id__exact=model_test.id)
self.assertEquals(value, model_test.pickle_field)
model_test.delete()
def testLookups(self):
"""Tests that lookups can be performed on data once stored in the database."""
for value in self.testing_data:
model_test = TestingModel(pickle_field=value)
model_test.save()
self.assertEquals(value, TestingModel.objects.get(pickle_field__exact=value).pickle_field)
model_test.delete()
|
More like this
- Template tag - list punctuation for a list of items by shapiromatron 10 months, 4 weeks ago
- JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 11 months ago
- Serializer factory with Django Rest Framework by julio 1 year, 5 months ago
- Image compression before saving the new model / work with JPG, PNG by Schleidens 1 year, 6 months ago
- Help text hyperlinks by sa2812 1 year, 7 months ago
Comments
I'm having problems with this...
It works fine until I save an object and load it back:
... as you can see, once it's gone via the database I get the pickle instead of the object.
I'm using the svn version of django.
#
Not working :o(
#
yaxu: I'll take a look at this problem now and see what I can come up with. I probably ought to write some unit tests while I'm at it, too…
nikolay: I don't use PostgreSQL (I use MySQL), but I'll have to install it and give it a go.
#
Right, I've revised this code, and added some unit tests. (Please notice that these are two separate files now!)
yaxu: Your issue should be fixed.
nikolay: Haven't tested yours yet, though it's on my to-do list!
#
Oh, and the unit tests do pass over here!
#
Very nice snippet, I use it a lot. I have one suggestion which is to use the highest protocol available for any given system when dumping objects to pickles. This can be accomplished by using pickle.HIGHEST_PROTOCOL when dumping. For this case, line #33 should look something like
value = PickledObject(pickle.dumps(value,pickle.HIGHEST_PROTOCOL))
In my testing this boosted performance 28 percent. Thanks again!
#
PostgreSQL Users
To get this to work with PostgreSQL you need to register PickledObject with it's marshaler:
Admin Site Users
Mark your field editable=False to retain data while editing the object with admin.
#
Please login first before commenting.