Optimizing variables cache in Drupal 6

In Drupal 6, a number of caching strategies are incorporated to handle large traffic. One of them is the serialization of the whole variable table. It is being cached in the database and gets extracted into global $conf variable in each invoke.

In one of our production sites, we faced hard time to keep up with the memory requirement of PHP for the unserialization of this variable from the cache. The variables table was so large that we had to assign around 1GB memory to each PHP thread so that the value can be unserialized without memory exhaustion. This made it much harder to scale the application.

So, we decided to do something about it and successfully handled it by doing the following:

1. First of all, we installed the memcache module to move the cache storage from DB to memory

2. We then edited the memcache module’s cache_get and cache_set functions to store/retrieve individual items from the variables array and split/join them when requested.

3. This requires a memcache call for each of the items in the variable array, but consumes a small amount of memory as there is no huge unserialize operation going on.

4. We performed a few tests to see if the site is working as before, and found its working perfectly!

Here is the code in case you are facing similar issue:

/sites/all/modules/contrib/memcache/memcache.inc

 1<?php
 2 
 3// ...beginning part of the file
 4 
 5function cache_set($cid, $data, $table = 'cache', $expire = CACHE_PERMANENT, $headers = NULL) {
 6 
 7  // Handle database fallback first.
 8  $bins = variable_get('memcache_bins', array());
 9  if (!is_null($table) && isset($bins[$table]) && $bins[$table] == 'database') {
10    return _cache_set($cid, $data, $table, $expire, $headers);
11  }
12 
13  // In case of special cache items, we keep the individual items as
14  // separate cache items. Later in the retrieval time, we join them together.
15  if (memcache_is_special_cache_item($cid)) {
16 
17    $keys = array_keys($data);
18    foreach ($keys as $key) {
19      cache_set($cid . '_' . $key, $data[$key]);
20    }
21 
22    cache_set($cid . '_keys', $keys);
23    return true;
24 
25  }
26 
27  // ...remaining part of the function
28}
29 
30function cache_get($cid, $table = 'cache') {
31 
32  // Handle excluded bins first.
33  $bins = variable_get('memcache_bins', array());
34  if (!is_null($table) && isset($bins[$table]) && $bins[$table] == 'database') {
35    return _cache_get($cid, $table);
36  }
37 
38  // The special cache item was previously saved as individual items,
39  // so now we have to retrieve them separately and join them together
40  // and send as one item.
41  if (memcache_is_special_cache_item($cid)) {
42 
43    $keys = cache_get($cid . '_keys');
44    if (is_null($keys->data)) {
45      return false;
46    }
47 
48    $data = array();
49    foreach ($keys->data as $key) {
50      $data[$key] = cache_get($cid . '_' . $key);
51    }
52 
53    $cache = new stdClass();
54    $cache->data = $data;
55 
56    return $cache;
57  }
58 
59  // ...remaining part of the function
60}
61 
62function memcache_is_special_cache_item($cid) {
63  $specials = array('variables', 'strongarm');
64  return in_array($cid, $specials);
65}
66 
67// ...remaining part of the file