Log the time taken to execute each DB query
Use this snippet to log the time taken to execute a specific DB query.
- django
- db
- logging
Snippet List
TinyIntegerField
Django lacks support of MySQL's "tinyint" 8-bit-integer datatype. This snippet gives you a TinyIntegerField and a PositiveTinyIntegerField. Falls back Django's SmallIntegerField if a different database-engine is used
- model
- db
- 8-bit-integer
- MySQL
- IntegerField
- tiny-integer
JSONField
This is a custom field that lets you easily store JSON data in one of your model fields. This is updated to work with Django 1.1. **Example: (models.py)** from django.db import models import JSONField class MyModel(models.Model): info = JSONField() ** Example: (shell)** >>> obj = MyModel.objects.all()[0] >>> type(obj.info) <type 'NoneType'> >>> obj.info = {"test": [1, 2, 3]} >>> obj.save() **[Code at GitHub](http://github.com/bradjasper/django-jsonfield/tree/master)**
- models
- fields
- model
- json
- db
- field
- json-field
- jsonfield
JSONField
This is a great way to pack extra data into a model object, where the structure is dynamic, and not relational. For instance, if you wanted to store a list of dictionaries. The data won't be classically searchable, but you can define pretty much any data construct you'd like, as long as it is JSON-serializable. It's especially useful in a JSON heavy application or one that deals with a lot of javascript. **Example** (models.py): from django.db import models from jsonfield import JSONField class Sequence(models.Model): name = models.CharField(maxlength=25) list = JSONField() **Example** (shell): fib = Sequence(name='Fibonacci') fib.list = [0, 1, 1, 2, 3, 5, 8] fib.save() fib = Sequence.objects.get(name='Fibonacci') fib.list.append(13) print fib.list [0, 1, 1, 2, 3, 5, 8, 13] fib.get_list_json() "[0, 1, 1, 2, 3, 5, 8, 13]" *Note:* You can only save JSON-serializable data. Also, dates will be converted to string-timestamps, because I don't really know what better to do with them. Finally, I'm not sure how to interact with forms yet, so that realm is a bit murky.
- models
- model
- json
- db
- field
- json-field
Ordered items in the database - alternative
Every now and then you need to have items in your database which have a specific order. As SQL does not save rows in any order, you need to take care about this for yourself. No - actually, you don't need to anymore. You can just use this file - it is designed as kind-of plug-in for the Django ORM. Usage is (due to use of meta-classes) quite simple. It is recommended to save this snippet into a separate file called `positional.py`. To use it, you only have to import `PositionalSortMixIn` from the `positional` module and inherit from it in your own, custom model (but *before* you inherit from `models.Model`, the order counts). Usage example: Add this to your `models.py` from positional import PositionalSortMixIn class MyModel(PositionalSortMixIn, models.Model): name = models.CharField(maxlength=200, unique=True) Now you need to create the database tables: `PositionalSortMixIn` will automatically add a `postition` field to your model. In your views you can use it simply with `MyModel.objects.all().order_by('position')` and you get the objects sorted by their position. Of course you can move the objects down and up, by using `move_up()`, `move_down()` etc. In case you feel you have seen this code somewhere - right, this snippet is a modified version of [snippet #245](/snippets/245/) which I made earlier. It is basically the same code but uses another approach to display the data in an ordered way. Instead of overriding the `Manager` it adds the `position` field to `Meta.ordering`. Of course, all of this is done automatically, you only need to use `YourItem.objects.all()` to get the items in an ordered way. Update: Now you can call your custom managers `object` as long as the default manager (the one that is defined first) still returns all objects. This Mix-in absolutely needs to be able to access all elements saved. In case you find any errors just write a comment, updated versions are published here from time to time as new bugs are found and fixed.
- db
- orm
- database
- plugin
- mixin
MySQL "Text" Type Model Field
Custom field for using MySQL's `text` type. `text` is more compact than the `longtext` field that Django assigns for `models.TextField` (2^16 vs. 2^32, respectively)
- text
- models
- mysql
- db
- database
- field
- custom-field
Allow filtering and ordering by counts of related query results
I know you're thinking, *what the heck could that title mean?* I often find myself wanting to filter and order by the result of a COUNT(*) of a query using a method similar to the [entry_count example](http://www.djangoproject.com/documentation/db-api/#extra-select-none-where-none-params-none-tables-none). Writing this many times is tedious and hardcoding the table and column names made me cringe, I also wanted the counts to result from more complex queries. This is a method you can add to your custom Manager to do this easily. It's not an ideal syntax, but it's good for the amount of code required. Example: suppose we have some articles we want to filter and order by comments and visit logs to show the most popular... class ArticleManager(models.Manager): count_related = _count_related class Article(models.Model): pub_date = models.DateTimeField(auto_now_add=True) objects = ArticleManager() class Comment(models.Model): article = models.ForeignKey(Article) is_spam = models.BooleanField(default=False) class Visit(models.Model): article = models.ForeignKey(Article) referrer = models.URLField(verify_exists=False) search_query = models.CharField(maxlength=200) Notice how the ArticleManager is given the `count_related` method. Now you can find the most popular like so... Order by non-spam comments: Article.objects.count_related(Comment.objects.filter( is_spam=False)).order_by('-comment__count') Order by incoming non-search-engine links: Article.objects.count_related(Visit.objects.filter( referrer__isnull=False, search_query__isnull=True), 'links').order_by('-links') Order by total visits: Article.objects.count_related(Visit).order_by('-visit__count') Note: Doesn't work if `query` contains joins or for many-to-many relationships, but those could be made to work identically if there's demand.
- sql
- model
- db
- count
- manager
Template tag to dump database query info
This snippet introduces two tags: `{%dbinfo%}` and `{%dbquerylist%}`. The `{%dbinfo%}` tag returns a string with the # of database queries and aggregate DB time. The `{%dbquerylist%}` tag expands to a set of <LI> elements containing the actual SQL queries executed. If `settings.TEMPLATE_DEBUG` is False, both tags return empty strings.
- tag
- db
- debugging
Pickled Object Field
*Incredibly useful for storing just about anything in the database (provided it is Pickle-able, of course) when there isn't a 'proper' field for the job:* A field which can store any pickleable object in the database. It is database-agnostic, and should work with any database backend you can throw at it. Pass in any object and it will be automagically converted behind the scenes, and you never have to manually pickle or unpickle anything. Also works fine when querying. **Please note that this is supposed to be two files, one fields.py and one tests.py (if you don't care about the unit tests, just use fields.py)**
- model
- db
- orm
- database
- pickle
- object
- field
- pickled
ByteSplitterField
When you want to save integers to the db, you usually have the choice between 16-, 32- and 64-bit Integers (also 8- and 24-bit for MySQL). If that doesn't fit your needs and you want to use your db-memory more efficient, this field might be handy to you. Imagine you have 3 numbers, but need only 10 bit to encode each (i.e. from 0 to 1000). Instead of creating 3 smallint-fields (48 bit), you can create one 'ByteSplitterField' which implements 3 'subfields' and automatically encodes them inside a 32 bit integer. You don't have to take care how each 10-bit chunk is encoded into the 32-bit integer, it's all handled by the field (see also field's description). Additionally, the Field offers opportunity to use decimal_places for each of your subfields. These are 'binary decimal places', meaning the integer-content is automatically divided by 2, 4, 8, etc. when you fetch the value from the field. You can also specify how values are rounded ('round' parameter) and what happens when you try to save a value out of range ('overflow' parameter) Not implemented (maybe in the future if I should need it sometime): * signed values. All values are positive right now! * real (10-based) decimal places (actually you could probably directly use DecimalFields here) * further space optimization, i.e. saving into CharField that's length can be chosen byte-wise
- model
- db
- database
- field
- custom
- custom-model-field
- IntegerField
- multibit-field
- model-field
Improved Pickled Object Field
[Based on snippet #513 by obeattie.](http://www.djangosnippets.org/snippets/513/) **Update 10/10/09:** [Further development is now occurring on GitHub, thanks to Shrubbery Software.](http://github.com/shrubberysoft/django-picklefield) Incredibly useful for storing just about anything in the database (provided it is Pickle-able, of course) when there isn't a 'proper' field for the job. `PickledObjectField` is database-agnostic, and should work with any database backend you can throw at it. You can pass in any Python object and it will automagically be converted behind the scenes. You never have to manually pickle or unpickle anything. Also works fine when querying; supports `exact`, `in`, and `isnull` lookups. It should be noted, however, that calling `QuerySet.values()` will only return the encoded data, not the original Python object. *Please note that this is supposed to be two files, one fields.py and one tests.py (if you don't care about the unit tests, just use fields.py).* This PickledObjectField has a few improvements over the one in [snippet #513](http://www.djangosnippets.org/snippets/513/). 1. This one solves the `DjangoUnicodeDecodeError` problem when saving an object containing non-ASCII data by base64 encoding the pickled output stream. This ensures that all stored data is ASCII, eliminating the problem. 2. `PickledObjectField` will now optionally use `zlib` to compress (and uncompress) pickled objects on the fly. This can be set per-field using the keyword argument "compress=True". For most items this is probably **not** worth the small performance penalty, but for Models with larger objects, it can be a real space saver. 3. You can also now specify the pickle protocol per-field, using the protocol keyword argument. The default of `2` should always work, unless you are trying to access the data from outside of the Django ORM. 4. Worked around a rare issue when using the `cPickle` and performing lookups of complex data types. In short, `cPickle` would sometimes output different streams for the same object depending on how it was referenced. This of course could cause lookups for complex objects to fail, even when a matching object exists. See the docstrings and tests for more information. 5. You can now use the `isnull` lookup and have it function as expected. A consequence of this is that by default, `PickledObjectField` has `null=True` set (you can of course pass `null=False` if you want to change that). If `null=False` is set (the default for fields), then you wouldn't be able to store a Python `None` value, since `None` values aren't pickled or encoded (this in turn is what makes the `isnull` lookup possible). 6. You can now pass in an object as the default argument for the field without it being converted to a unicode string first. If you pass in a callable though, the field will still call it. It will *not* try to pickle and encode it. 7. You can manually import `dbsafe_encode` and `dbsafe_decode` from fields.py if you want to encode and decode objects yourself. This is mostly useful for decoding values returned from calling `QuerySet.values()`, which are still encoded strings. The tests have been updated to match the added features, but if you find any bugs, please post them in the comments. My goal is to make this an error-proof implementation. **Note:** If you are trying to store other django models in the `PickledObjectField`, please see the comments for a discussion on the problems associated with doing that. The easy solution is to put django models into a list or tuple before assigning them to the `PickledObjectField`. **Update 9/2/09:** Fixed the `value_to_string` method so that serialization should now work as expected. Also added `deepcopy` back into `dbsafe_encode`, fixing #4 above, since `deepcopy` had somehow managed to remove itself. This means that lookups should once again work as expected in **all** situations. Also made the field `editable=False` by default (which I swear I already did once before!) since it is never a good idea to have a `PickledObjectField` be user editable.
- model
- db
- orm
- database
- pickle
- object
- field
- type
- pickled
- store
Extended db cache backend with 'filter() / LIKE' support (and now scheduled cache clean!)
Because the db caching doesn't support atomic operations, it was unsafe to store a list of 'keys' in a single key. So, rather than store the list, I just append each key with a specific tag, and then filter for it later. This means I don't need to worry too much about atomic usage with lists (i.e. queued requests). However - I still can think of many instances where I would need atomic usage, so I will probably implement this later on down the line. Hopefully, the atomic modifications will be accepted by the core devs. This also contains threaded cache cleaning, which means you no longer need to rely on requests to clean the cache (which would have potentially slowed the user query down), and will remove any cache entries past their expiry date every 3 minutes. Enjoy! Cal
- filter
- cache
- db
- backend
- like
Finding related objects for instances in a queryset
When deleting objects in Django's admin interface, it lists other objects which would be deleted and asks for confirmation. This snippet does the same programmatically. The snippet works in Django 1.3 (more specifically, revision 14507 or later). It uses Django internals which are not a part of the public API, so this might not work with future versions. Usage: `polls/models.py`: from django.db import models class Poll(models.Model): question = models.CharField(max_length=200) def __unicode__(self): return self.question class Choice(models.Model): poll = models.ForeignKey(Poll) choice = models.CharField(max_length=200) def __unicode__(self): return '%s %s' % (self.poll, self.choice) `$ ./manage.py shell` >>> from polls.models import Poll, Choice >>> from datetime import datetime >>> from pprint import pprint >>> poll1 = Poll.objects.create(question='Me?') >>> Choice.objects.create(poll=poll1, choice='Yes') >>> Choice.objects.create(poll=poll1, choice='No') >>> poll2 = Poll.objects.create(question='Really?') >>> Choice.objects.create(poll=poll2, choice='Yes') >>> Choice.objects.create(poll=poll2, choice='No') >>> pprint(get_related(Poll.objects.all())) {<class 'polls.models.Poll'>: [<Poll: Me?>, <Poll: Really?>], <class 'polls.models.Choice'>: [<Choice: Me? Yes>, <Choice: Me? No>, <Choice: Really? Yes>, <Choice: Really? No>]}
- db
- orm
- related
Improved Pickled Object Field (Fixed for Django 1.2)
Small changes to [Snippet 1694](http://djangosnippets.org/snippets/1694/) to that QueryAPI works for django 1.2 and higher. Changes: * Replaced `get_db_prep_value` with `get_prep_value`. * Replaced `get_db_prep_lookup` with modified `get_prep_lookup`.
- model
- db
- orm
- database
- pickle
- object
- field
- type
- pickled
- store
Log all interaction with user to the DB
Due to compliance requirements in the financials industry we needed to log every request a user made to our system, the action taken (view function) and response from the server. I found a lot of other logging solution bit most revolved around debugging and DB query logging. I needed to be able to tell what a user did while being logged in as much detail as I could with out tracking the mouse pointer position on screen. So I created this (, *my first* ,) middleware. Its very simple really. keeping track of a request, view_func and response object in a single model called Record (models.py file included in the code). The fields I used are optimized to what I intend to show in the UI I am planning for this model. Depending on how you use the doc string of your views they can be tapped to explain to the user what each request/func/responce group in a session is meant to do. There were a few gotcha's: 1. I only care about authenticated requests. So I added the 'quest.user.is_authenticated()' test. 2. I did not care about the favicon request so I skipped them. 2. The actual login request is not authenticated while the response is. This caused the process_response/view to look for a record that is not there. So I added the 'except ObjectDoesNotExist' to skip this case. I added one bell: Logging a full HTML reply is wasteful and mostly useless. I added two values in the setting files. LOGALL_LOG_HTML_RESPONSE to toggle if we want to log them or not. And LOGALL_HTML_START to describe what a full HTML starts with. Personally I use the first few characters of my base.html template that all the rest of my templates expend. I simplified the code to the left for readability.
- middleware
- log
- db
- compliance
- financial