Lesson learned: hierarchical custom post types don’t scale well

The other day I ran into an issue where an in-house developed WordPress plugin with custom post types was causing an out of memory error when attempting to view the custom posts in the admin dashboard.

It turns out that the issue is caused by the fact that listing hierarchical post types in the admin dashboard does the following (taken from Nacin’s comment here http://core.trac.wordpress.org/ticket/15459):

  • Query all posts
  • Group them based on ID and post_parent
  • Begin rendering posts until we’ve reached the total to display per page

That first bit of query all posts just so happens to load all of the posts into memory.  If your custom post type doesn’t have a ton of posts, this isn’t a problem, but in our case we had 40k custom posts that were causing an out of memory error (eg Allowed memory size of 268435456 bytes exhausted).  The 256MB limit is set by default in WordPress via the WP_MAX_MEMORY_LIMIT constant.

As I examined the plugin in question, I realized that it had no need to be registering its custom post type as hierarchical, so in my case it was an easy fix.  However, if you have custom post types that are hierarchical for good reason, note that you may run into this issue.  There is an enhancement ticket for this issue here:


WordPress uploads and load balancing

I am currently working on a project building out a load balanced WordPress install with multiple web servers and I came across an issue that I think I have found an innovative way to address.  Recently at WordCamp San Francisco, Mark Jaquith in his session Scaling, Servers, and Deploys — Oh My! stated that his preferred way to handle uploads in a load balanced WordPress environment is to use a single server for WordPress admin and media uploads and then use some type of task to push the media out to the other servers.  I think this is great for most load balanced WordPress environments since the load from admins is minimal compared to end users.

However, what happens when you give your end users the capability to upload media?  You certainly don’t want to move your end users to a single server — that defeats the purpose of having a load balancer.

In my case, the project I am working on is being built on RackSpace cloud servers and is using RackSpace cloud files for CDN distribution of static images.  The thought occurred to me that there had to be a way for my cloud servers to somehow share uploaded files.  After googling around I came across CloudFuse.  CloudFuse essentially allows me to mount RackSpace cloud files as a directory on my web servers.  Each web server has a directory that mounts the cloud files and WordPress’ upload path is set to that directory.  Multiple web servers seem to play nicely with uploading files to the directory, though I have not tested it under an type of heavy load.

There is also another benefit to this solution.  Since cloud files include CDN distribution, the uploaded files can then be served using the CDN which circumvents two things:

  1. Having to push the uploaded images to multiple servers and/or a CDN.
  2. The CloudFuse mount only has to be used for write operations.

Admittedly this solution only works if you are using RackSpace, though I believe you could pull off something similar with other CDNs.

My Video from WordCampSF

Recently I had the privilege of speaking at WordCamp San Francisco on “WordPress for the Greater Good”.  It was a shared presentation with Zach Berke of Exygy who spoke on the excellent work his team did for UNICEF in Uganda.

In this presentation I talked about how CURE International (my employer) moved to WordPress to become an award-winning website as well as providing their constituents with the world’s first social sponsorship platform (cure.org/curekids). I also covered the release of Personal Fundraiser, an open source (GPL) WordPress plugin that provides organizations the ability to allow their fans and constituents to create their own custom online fundraisers using Paypal donations and other payment methods.

You can watch the video here:

More details about the presentation are available at the WordCamp San Francisco site.

nginx config for TouchPad WordPress

Here is the nginx config I used for my TouchPad WordPress server:

# Upstream to abstract backend connection(s) for php
upstream php {
        server unix:/tmp/php-cgi.socket;

server {
        ## Your website name goes here.
        server_name localhost;
        ## Your only path reference.
        root /usr/local/share/wordpress;
        ## This should be in your http block and if it is, it's not needed here.
        index index.php;

        location = /favicon.ico {
                log_not_found off;
                access_log off;

        location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;

        location / {
                # This is cool because no php is touched for static content
                try_files $uri $uri/ /index.php;

        location ~ \.php$ {
            fastcgi_index index.php;
            include fastcgi_params;

        location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
                expires max;
                log_not_found off;

HP TouchPad WordPress Server

Thanks to HP’s recent fire sale of their TouchPad tablet, I found myself with a new device to hack.  After digging around webOSroundup, I found that I could install Ubuntu on the TouchPad.  This got me thinking:

I use Ubuntu for WordPress development.  Could I run a WordPress server on a TouchPad?

It turns out that by using nginx, MySQL and php5-fpm the answer is yes:

TouchPad WordPress serverHere is how:

  1. Install Ubuntu on the TouchPad using Preware, Meta-Doctor, and UbuntuChroot.  I followed the directions from the webOSroundup forums.   Once you have Ubuntu up and running do the following:
  2. Install nginx: apt-get install nginx
  3. Before installing mysql, you will need to use dpkg-divert to avoid the error Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused:
    dpkg-divert --local --rename --add /sbin/initctl
    ln -s /bin/true /sbin/initctl

    For more details see here and here.

  4. Install mysql and php-fpm:
    apt-get install mysql-server php5-fpm php5-mysql
  5. Download WordPress latest from wordpress.org and extract locally:
    cd /usr/local/share
    wget http://wordpress.org/latest.tar.gz
    tar -xzf latest.tar.gz
  6. Edit /etc/php5/fpm/pool.d/www.conf and change the listen port to 9001 (9000 is already in use).
  7. Edit /etc/nginx/sites-available/default for use with wordpress. I’ll post my config as a separate post. I used the basic setup detailed here: http://wiki.nginx.org/WordPress, but I did have to tweak it slightly.
  8. (Re)start the necessary services:
    mysql: mysqld_safe &
    php5-fpm: /etc/init.d/php5-fpm restart
    nginx /etc/init.d/nginx restart
  9. Perform the WordPress Famous 5 Minute Install.
  10. Enjoy!