Login

Extended db cache backend with 'filter() / LIKE' support (and now scheduled cache clean!)

Author:
sleepycal
Posted:
January 7, 2010
Language:
Python
Version:
1.1
Score:
0 (after 0 ratings)

Because the db caching doesn't support atomic operations, it was unsafe to store a list of 'keys' in a single key. So, rather than store the list, I just append each key with a specific tag, and then filter for it later. This means I don't need to worry too much about atomic usage with lists (i.e. queued requests).

However - I still can think of many instances where I would need atomic usage, so I will probably implement this later on down the line. Hopefully, the atomic modifications will be accepted by the core devs.

This also contains threaded cache cleaning, which means you no longer need to rely on requests to clean the cache (which would have potentially slowed the user query down), and will remove any cache entries past their expiry date every 3 minutes.

Enjoy!

Cal

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
#example
    from django.core.cache import cache
    cache.set('hello_19374', 'some value here') # will set the key hello_19374 to 'some value here'
    cache.set('hello_xfhh', 'some value here') # etc
    cache.set('19374hello', 'some value here') # etc
    cache.set('alal_hello_19374', 'some value here') # etc
    cache.set('lalahellohdgf', 'some value here') # etc
    print cache.filter('hello%') # will show all keys beginning with 'hello'
    print cache.filter('%%') # will show all keys
    print cache.filter('%hello') # will show all keys ending with 'hello'
    print cache.filter('%hello%') # will show all keys containing 'hello'

    cache.clean() # will delete any cache entries which have past their expiry

#app/settings.py
    CACHE_BACKEND = 'app.cache://cache_table_name_goes_here'

#app/cache.py
"Database cache backend."
    from django.core.cache.backends.db import CacheClass as _CacheClass

    from django.core.cache.backends.base import BaseCache
    from django.db import connection, transaction, DatabaseError
    import base64, time
    from datetime import datetime
    try:
        import cPickle as pickle
    except ImportError:
        import pickle

    # Import the threading and time
    import threading

    def RoutineCacheClean():
        # Import the cache
        from django.core.cache import cache

        # Loop forever!
        while True:
            # Clean it!
            try:
                cache.clean()

            except:
                pass

            # Clean every 3 minutes
            time.sleep(60*3)

    def StartCacheClean():
        # Create the new thread
        t = threading.Thread(
            target = RoutineCacheClean
        )

        # Set the thread daemon flag to true
        t.setDaemon(True)

        # Start the thread
        t.start()

    class CacheClass ( _CacheClass ):
        def filter(self, key, default=None):
            lx={}
            cursor = connection.cursor()
            cursor.execute("SELECT cache_key, value, expires FROM %s WHERE cache_key LIKE %%s" % self._table, [key])

            for row in cursor.fetchall():
                if row is not None:
                    now = datetime.now()
                    if row[2] < now:
                        cursor.execute("DELETE FROM %s WHERE cache_key = %%s" % self._table, [row[0]])
                        transaction.commit_unless_managed()
                    else:
                        value = connection.ops.process_clob(row[1])
                        lx[row[0]]=pickle.loads(base64.decodestring(value))

            return lx

        def clean(self):
            cursor = connection.cursor()
            now = datetime.now().replace(microsecond=0)
            cursor.execute("DELETE FROM %s WHERE expires < %%s" % self._table, [str(now)])
            transaction.commit_unless_managed()

    StartCacheClean()

More like this

  1. Template tag - list punctuation for a list of items by shapiromatron 3 months, 1 week ago
  2. JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 3 months, 2 weeks ago
  3. Serializer factory with Django Rest Framework by julio 10 months, 2 weeks ago
  4. Image compression before saving the new model / work with JPG, PNG by Schleidens 11 months ago
  5. Help text hyperlinks by sa2812 12 months ago

Comments

Please login first before commenting.