This fields.py file defines a new model field type, "JsonObjectField," which is designed to allow the storage of arbitrary Python objects in Django TextFields. It is intended primarily to allow the storage of Python dictionaries or list objects. As the name implies, it converts objects to JSON for storage; this conversion happens transparently, so from your model's perspective, the field stores and retrieves the actual objects.
*Incredibly useful for storing just about anything in the database (provided it is Pickle-able, of course) when there isn't a 'proper' field for the job:* A field which can store any pickleable object in the database. It is database-agnostic, and should work with any database backend you can throw at it. Pass in any object and it will be automagically converted behind the scenes, and you never have to manually pickle or unpickle anything. Also works fine when querying. **Please note that this is supposed to be two files, one fields.py and one tests.py (if you don't care about the unit tests, just use fields.py)**
When you want to save integers to the db, you usually have the choice between 16-, 32- and 64-bit Integers (also 8- and 24-bit for MySQL). If that doesn't fit your needs and you want to use your db-memory more efficient, this field might be handy to you. Imagine you have 3 numbers, but need only 10 bit to encode each (i.e. from 0 to 1000). Instead of creating 3 smallint-fields (48 bit), you can create one 'ByteSplitterField' which implements 3 'subfields' and automatically encodes them inside a 32 bit integer. You don't have to take care how each 10-bit chunk is encoded into the 32-bit integer, it's all handled by the field (see also field's description). Additionally, the Field offers opportunity to use decimal_places for each of your subfields. These are 'binary decimal places', meaning the integer-content is automatically divided by 2, 4, 8, etc. when you fetch the value from the field. You can also specify how values are rounded ('round' parameter) and what happens when you try to save a value out of range ('overflow' parameter) Not implemented (maybe in the future if I should need it sometime): * signed values. All values are positive right now! * real (10-based) decimal places (actually you could probably directly use DecimalFields here) * further space optimization, i.e. saving into CharField that's length can be chosen byte-wise
[Based on snippet #513 by obeattie.](http://www.djangosnippets.org/snippets/513/) **Update 10/10/09:** [Further development is now occurring on GitHub, thanks to Shrubbery Software.](http://github.com/shrubberysoft/django-picklefield) Incredibly useful for storing just about anything in the database (provided it is Pickle-able, of course) when there isn't a 'proper' field for the job. `PickledObjectField` is database-agnostic, and should work with any database backend you can throw at it. You can pass in any Python object and it will automagically be converted behind the scenes. You never have to manually pickle or unpickle anything. Also works fine when querying; supports `exact`, `in`, and `isnull` lookups. It should be noted, however, that calling `QuerySet.values()` will only return the encoded data, not the original Python object. *Please note that this is supposed to be two files, one fields.py and one tests.py (if you don't care about the unit tests, just use fields.py).* This PickledObjectField has a few improvements over the one in [snippet #513](http://www.djangosnippets.org/snippets/513/). 1. This one solves the `DjangoUnicodeDecodeError` problem when saving an object containing non-ASCII data by base64 encoding the pickled output stream. This ensures that all stored data is ASCII, eliminating the problem. 2. `PickledObjectField` will now optionally use `zlib` to compress (and uncompress) pickled objects on the fly. This can be set per-field using the keyword argument "compress=True". For most items this is probably **not** worth the small performance penalty, but for Models with larger objects, it can be a real space saver. 3. You can also now specify the pickle protocol per-field, using the protocol keyword argument. The default of `2` should always work, unless you are trying to access the data from outside of the Django ORM. 4. Worked around a rare issue when using the `cPickle` and performing lookups of complex data types. In short, `cPickle` would sometimes output different streams for the same object depending on how it was referenced. This of course could cause lookups for complex objects to fail, even when a matching object exists. See the docstrings and tests for more information. 5. You can now use the `isnull` lookup and have it function as expected. A consequence of this is that by default, `PickledObjectField` has `null=True` set (you can of course pass `null=False` if you want to change that). If `null=False` is set (the default for fields), then you wouldn't be able to store a Python `None` value, since `None` values aren't pickled or encoded (this in turn is what makes the `isnull` lookup possible). 6. You can now pass in an object as the default argument for the field without it being converted to a unicode string first. If you pass in a callable though, the field will still call it. It will *not* try to pickle and encode it. 7. You can manually import `dbsafe_encode` and `dbsafe_decode` from fields.py if you want to encode and decode objects yourself. This is mostly useful for decoding values returned from calling `QuerySet.values()`, which are still encoded strings. The tests have been updated to match the added features, but if you find any bugs, please post them in the comments. My goal is to make this an error-proof implementation. **Note:** If you are trying to store other django models in the `PickledObjectField`, please see the comments for a discussion on the problems associated with doing that. The easy solution is to put django models into a list or tuple before assigning them to the `PickledObjectField`. **Update 9/2/09:** Fixed the `value_to_string` method so that serialization should now work as expected. Also added `deepcopy` back into `dbsafe_encode`, fixing #4 above, since `deepcopy` had somehow managed to remove itself. This means that lookups should once again work as expected in **all** situations. Also made the field `editable=False` by default (which I swear I already did once before!) since it is never a good idea to have a `PickledObjectField` be user editable.
I needed the ability to serialize and deserialize my database, which contains millions of objects. The existing XML serializer encountered spurious parse errors; the JSON serializer failed to handle UTF-8 even when it was asked to; and both the JSON and YAML serializers tried to keep all the representations in memory simultaneously. This custom serializer is the only one that has done the job. It uses YAML's "stream of documents" model so that it can successfully serialize and deserialize large databases.
I once needed to convert a Django project from PostgreSQL to SQLite. At that time I was either unaware of manage.py dumpdata/loaddata or it they didn't yet exist. I asked for advice on the #django IRC channel where ubernostrum came up with this plan: simple process: 1) Select everything. 2) Pickle it. 3) Save to file. 4) Read file. 5) Unpickle. 6) Save to db. :) Or something like that. First I thought it was funny, but then started to think about it and it made perfect sense. And so dbpickle.py was born. I've used this script also for migrating schema changes to production databases. For migration you can write plugins to hook on dbpickle.py's object retrieval and saving. This way you can add/remove/rename fields of objects on the fly when loading a dumped database. It's also possible to populate new fields with default values or even values computed based on the object's other properties. A good way to use this is to create a database migration plugin for each schema change and use it with dbpickle.py to migrate the project. See also [original blog posting](http://akaihola.blogspot.com/2006/11/database-conversion-django-style.html) and [my usenet posting](http://groups.google.com/group/django-users/browse_thread/thread/6a4e9781d08ae815/c5c063a288483e07#c5c063a288483e07) wondering about the feasibility of this functionality with manage.py dumpdata/loaddata. See [trac site](http://trac.ambitone.com/ambidjangolib/browser/trunk/dbpickle/dbpickle.py) for version history.
The snippet enables decoupling model classes, associated with a ForeignKey, for the purpose of separating them into two databases. Looking at the following example: class Reporter(models.Model): ... class Article(models.Model): reporter = models.ForeignKey(Reporter) We want to separate the `Reporter` and `Article` into two separate databases, but this won't work in Django because of the [cross model relations](http://docs.djangoproject.com/en/dev/topics/db/multi-db/#cross-database-relations). The solution is to use an ID field and not a `ForeignKey` one. This makes the access very uncomfortable. The above class will make this a bit less awkward. It doesn't support the [RelatedManager](http://docs.djangoproject.com/en/dev/ref/models/relations/#django.db.models.fields.related.RelatedManager), but it will support accessing the related field as you normally would. The article class will look like this (assuming the reporter model id field is an `IntegerField`): class Article(DecoupledModel): reporter_id = models.IntegerField() _linked_fields = { 'reporter': Reporter, } Once you have an article object, you can access the reporter as you normally would for both read and writing. For example: my_reporter = article1.reporter article2.reporter = my_reporter
*WARNING* This is *extremely* slow. This snippet allows you to easily prevent *most* race conditions (if used properly). Feel free to extend on top of this as you like, I'd appreciate any comments to [email protected]
Small changes to [Snippet 1694](http://djangosnippets.org/snippets/1694/) to that QueryAPI works for django 1.2 and higher. Changes: * Replaced `get_db_prep_value` with `get_prep_value`. * Replaced `get_db_prep_lookup` with modified `get_prep_lookup`.
Script to help manage database migrations. Explanation and background can be found in blog post at [paltman.com](http://paltman.com/2008/07/03/managing-database-changes-in-django/).
I hate when my unittest hits database. Especially when each test case needs different dataset. So I wrote this db mock, that's local to specific test and uses sqlite3 in-memory db. Usage (nosetests): class TestMainNoData(DbMock): 'testing main function with no meaningful data' def test_no_posts(self): 'there are no posts' assert models.Post.objects.count() == 0, 'should be no feeds'
Check database constraints after dirty import
First off: This snippet is more or less obsoleted by [snippet #259](/snippets/259/). The way that snippet #259 uses feels cleaner and more usable. It is basically the same code but the magic works a little bit different. In case you prefer this way, I've left this snippet as-is. Maybe you know this problem: you have some objects in your database and would like to display them in some way. The problem: the Django ORM does not provide any way to sort model instances by a user-defined criterion. There was already a plug-in in SQLAlchemy called [orderinglist](http://www.sqlalchemy.org/docs/plugins.html#plugins_orderinglist), so implementing this in Django was basically not a big deal. Usage is (due to use of meta-classes) quite simple. It is recommended to save this snippet into a separate file called `positional.py`. To use it, you only have to import `PositionalSortMixIn` from the `positional` module and inherit from it in your own, custom model (but *before* you inherit from `models.Model`, the order counts).
There are a lot of reasons why you should not do this. But nevertheless it might be useful in certain situations. If your database does not change and you want to use validators etc you better use inspectdb from django-admin.py
This is an improvement of snippet 253 in that it supports database queries. Implementing autocompletion for foreign keys takes a few steps: 1) Put the snippet above into <app>/widgets/autocomplete.py. 2) Create a view of your foreign key model (here: Donator) in <app>/donator/views.py: from models import Donator from widgets.autocomplete import autocomplete_response def autocomplete(request): return autocomplete_response( request.REQUEST['text'], Donator, ( 'line_1', 'line_2', 'line_3', 'line_4', 'line_5', 'line_6', 'line_7', 'line_8', '^zip_code', 'location' ) ) This view returns the autocompletion result by searching the fields in the tuple. Each word from the form field must appear at least in one database field. 3) Create a URLconf that points to this new view. 4) In the form where you need the autocompletion, define the widget of the foreign key field as an instance of AutoCompleteField: from widget.autocomplete import AutoCompleteField field.widget = AutoCompleteField( url='/donator/autocomplete/'), options={'minChars': 3} ) The url parameter is the URL connected to the view in step 3), the options dict is passed on to the Ajax.Autocompleter JavaScript object. Links: * [Snippet 253](http://www.djangosnippets.org/snippets/253/) * [Django and scriptaculous integration](http://wiki.script.aculo.us/scriptaculous/show/IntegrationWithDjango) * [Ajax.Autocompleter](http://wiki.script.aculo.us/scriptaculous/show/Ajax.Autocompleter)