In Drupal 6, a number of caching strategies are incorporated to handle large traffic. One of them is the serialization of the whole variable table. It is being cached in the database and gets extracted into global $conf variable in each invoke.
In one of our production sites, we faced hard time to keep up with the memory requirement of PHP for the unserialization of this variable from the cache. The variables table was so large that we had to assign around 1GB memory to each PHP thread so that the value can be unserialized without memory exhaustion. This made it much harder to scale the application.
So, we decided to do something about it and successfully handled it by doing the following:
1. First of all, we installed the memcache module to move the cache storage from DB to memory
2. We then edited the memcache module’s cache_get and cache_set functions to store/retrieve individual items from the variables array and split/join them when requested.
3. This requires a memcache call for each of the items in the variable array, but consumes a small amount of memory as there is no huge unserialize operation going on.
4. We performed a few tests to see if the site is working as before, and found its working perfectly!
Here is the code in case you are facing similar issue: (more…)