Login

Tag "database"

47 snippets

Snippet List

Db Mock

I hate when my unittest hits database. Especially when each test case needs different dataset. So I wrote this db mock, that's local to specific test and uses sqlite3 in-memory db. Usage (nosetests): class TestMainNoData(DbMock): 'testing main function with no meaningful data' def test_no_posts(self): 'there are no posts' assert models.Post.objects.count() == 0, 'should be no feeds'

  • testing
  • unittest
  • database
  • test
Read More

Pull ID from arbitrary sequence

Django does not currently allow one to pull ID values from arbitrarily named sequences. For example, if you did not create your ID column using the serial data type in PostgreSQL, you likely will not be able to use your sequences. This is quite a problem for those integrating with legacy databases. While ultimately the best place to fix this is django proper, this decorator will help people get by for now. Note that in this case, all of my sequences are named "pk_TABLENAME". You'll likely have a different convention and should update the decorator appropriately. While I could have made the pattern a parameter, it didn't seem like that would gain much here.

  • database
  • sequence
  • postgresql
Read More

Friendly ID(Python 3.X)

This is just modified version of [friendly id](https://djangosnippets.org/snippets/1249/) for make this script compatible with python 3.x Invoice numbers like "0000004" are a little unprofessional in that they expose how many sales a system has made, and can be used to monitor the rate of sales over a given time. They are also harder for customers to read back to you, especially if they are 10 digits long. This is simply a perfect hash function to convert an integer (from eg an ID AutoField) to a unique number. The ID is then made shorter and more user-friendly by converting to a string of letters and numbers that wont be confused for one another (in speech or text). To use it: import friendly_id class MyModel(models.Model): invoice_id = models.CharField(max_length=6, null=True, blank=True, unique=True) def save(self, *args, **kwargs): super(MyModel, self).save(*args, **kwargs) # Populate the invoice_id if it is missing if self.id and not self.invoice_id: self.invoice_id = friendly_id.encode(self.id) self.save() if self.id and not self.invoice_id When an object from this model is saved, an invoice ID will be generated that does not resemble those surrounding it. For example, where you are expecting millions of invoices the IDs generated from the AutoField primary key will be: obj.id obj.invoice_id 1 TTH9R 2 45FLU 3 6ACXD 4 8G98W 5 AQ6HF 6 DV3TY ... 9999999 J8UE5 The functions are deterministic, so running it again sometime will give the same result, and generated strings are unique for the given range (the default max is 10,000,000). Specifying a higher range allows you to have more IDs, but all the strings will then be longer. You have to decide which you need: short strings or many strings :-) This problem could have also been solved using a random invoice_id generator, but that might cause collisions which cost time to rectify, especially when a decent proportion of the available values are taken (eg 10%). Anyhow, someone else has now already written this little module for you, so now you don't have to write your own :-)

  • database
  • field-id
  • invoice-id
  • invoice
Read More

Django chunked queryset iterator

The function slices a queryset into smaller querysets containing chunk_size objects and then yield them. It is used to avoid memory error when processing huge queryset, and also database error due to that the database pulls whole table at once. Concurrent database modification wouldn't make some entries repeated or skipped in this process.

  • django
  • python
  • database
  • queryset
  • iterator
  • memoryerror
Read More

Decoupling models with cross-database relations

The snippet enables decoupling model classes, associated with a ForeignKey, for the purpose of separating them into two databases. Looking at the following example: class Reporter(models.Model): ... class Article(models.Model): reporter = models.ForeignKey(Reporter) We want to separate the `Reporter` and `Article` into two separate databases, but this won't work in Django because of the [cross model relations](http://docs.djangoproject.com/en/dev/topics/db/multi-db/#cross-database-relations). The solution is to use an ID field and not a `ForeignKey` one. This makes the access very uncomfortable. The above class will make this a bit less awkward. It doesn't support the [RelatedManager](http://docs.djangoproject.com/en/dev/ref/models/relations/#django.db.models.fields.related.RelatedManager), but it will support accessing the related field as you normally would. The article class will look like this (assuming the reporter model id field is an `IntegerField`): class Article(DecoupledModel): reporter_id = models.IntegerField() _linked_fields = { 'reporter': Reporter, } Once you have an article object, you can access the reporter as you normally would for both read and writing. For example: my_reporter = article1.reporter article2.reporter = my_reporter

  • django
  • model
  • database
Read More

PostgreSQL ON DELETE CASCADE

Have you always been annoyed by how you set up this elaborate big database schema and weren't able to have **ON DELETE CASCADE ON UPDATE CASCADE** in dbshell? This solves the problem; create the two files and and empty *__init__.py* and put them somewhere in your path. Then say DATABASE_ENGINE='postgresql_psycopg2_cascade' in settings. Really I'd like this to be in the ForeignKey object, were it upstream Django or an own version of it, but it doesn't seem possible. Ideas on how to make this configurable are more than welcome! Props go out to Ari Flinkman for the inspiration to do this!

  • database
  • postgres
  • foreign-key
  • postgresql
  • databases
Read More
Author: mjt
  • 1
  • 1

Testing for pending migrations in Django

DB migration support has been added in Django 1.7+, superseding South. More specifically, it's possible to automatically generate migrations steps when one or more changes in the application models are detected. Definitely a nice feature! I've written a small generic unit-test that one should be able to drop into the tests directory of any Django project and that checks there's no pending migrations, ie. if the models are correctly in sync with the migrations declared in the application. Handy to check nobody has forgotten to git add the migration file or that an innocent looking change in models.py doesn't need a migration step generated. Enjoy!

  • testing
  • unittest
  • database
  • migration
Read More

Multi-DB Reconnecting Persistent Postgres Connection

This is a modification of http://djangosnippets.org/snippets/1707/ that handles the database going down or PG Bouncer killing the connection. This also works in things like Twisted to make sure the connection is alive before doing a real query. Thanks @mike_tk for the original post! EDIT: Updated the wrapper to handle multi-db. Before it was using the first connection it made, now it creates an attribute name for the connection based on the name of the database.

  • database
  • multi-db
  • twisted
  • connection
  • persistent
  • multiple-databases
  • socket
  • web-socket
Read More

Improved Pickled Object Field (Fixed for Django 1.2)

Small changes to [Snippet 1694](http://djangosnippets.org/snippets/1694/) to that QueryAPI works for django 1.2 and higher. Changes: * Replaced `get_db_prep_value` with `get_prep_value`. * Replaced `get_db_prep_lookup` with modified `get_prep_lookup`.

  • model
  • db
  • orm
  • database
  • pickle
  • object
  • field
  • type
  • pickled
  • store
Read More

Custom model field for mysql time type.

Django does not have a suitable model field can process time type in mysql, the DateTimeField built-in django can not process more than 24 hours. That's why the code born. **Simply usage** `from myapp.models import TimeAsTimeDeltaField, SECONDS_PER_MIN, SECONDS_PER_HOUR, SECONDS_PER_DAY class EstimatedTime: days = None hours = None minutes = None seconds = None class Case(models.Model): estimated_time = TimeAsTimeDeltaField(null=True, blank=True) def get_estimated_time(self): estimated_time = EstimatedTime() if self.estimated_time: total_seconds = self.estimated_time.seconds + (self.estimated_time.days * SECONDS_PER_DAY) days = total_seconds / SECONDS_PER_DAY hours = total_seconds / SECONDS_PER_HOUR - days * 24 minutes = total_seconds / SECONDS_PER_MIN - hours * 60 - days * 24 * 60 seconds = total_seconds % SECONDS_PER_MIN estimated_time.days = days estimated_time.hours = hours estimated_time.minutes = minutes estimated_time.seconds = seconds return estimated_time else: return estimated_time

  • models
  • orm
  • database
  • field
  • timedelta
Read More

Export Models

Warning: This python script is designed for Django 0.96. It exports data from models quite like the `dumpdata` command, and throws the data to the standard output. It fixes glitches with unicode/ascii characters. It looked like the 0.96 handles very badly unicode characters, unless you specify an argument that is not available via the command line. The simple usage is: $ python export_models.py -a <application1> [application2, application3...] As a plus, it allows you to export only one or several models inside your application, and not all of them: $ python export_models.py application1.MyModelStuff application1.MyOtherModel Of course, you can specify the output format (serializer) with the -f (--format) option. $ python export_models.py --format=xml application1.MyModel

  • tool
  • dump
  • serialization
  • export
  • db
  • database
Read More