Dynamic backend servers with Varnish 3.0

Note: ┬áThis only works in Varnish 3 – Varnish 4 removed the DNS director and we’ll have to wait on a VMOD or make one ourselves to get this working in version 4!

Let’s say you’re wanting to host multiple websites on multiple backend servers and you want a single caching reverse proxy in front of all of them to make them super speedy.

For example each of my sites lives in its own OpenVZ container for security purposes as well as super easy backups and restores if needed. Of course I could set up varnish in each container along with httpd and MySQL, however for a bunch of smaller sites this is less efficient than just having one beefy caching proxy in front of all of them as you can then have a relatively full cache server rather than multiple mostly-empty caches.

Please do bear in mind the following guide assumes Varnish is on its own server. If it’s not, be sure to either change the listen port or have your web server listen on an alternate port.

Before getting started you’ll need

  • Backend web servers/sites
  • A working DNS zone to contain internal IPs (e.g. .ws.int, .internal.example.com, etc)
  • Varnish cache 3.0

Firstly if you’ve not done so already, install Varnish 3.0.

CentOS / RHEL

CentOS/RHEL 5

rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-release-3.0-1.el5.centos.noarch.rpm

CentOS/RHEL 6

rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/varnish-release-3.0-1.el6.noarch.rpm

Then do a

yum install varnish

Debian / Ubuntu

curl http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add -
echo "deb http://repo.varnish-cache.org/debian/ wheezy varnish-3.0" >> /etc/apt/sources.list
apt-get update
apt-get install varnish

We’ll then set up our varnish server how we like it. In CentOS you’ll want /etc/sysconfig/varnish, Debian keeps it in /etc/default/varnish I believe. It’s probably wise to grab a spare copy of the file before we modify it – Just in case.

Replace the contents with the following, changing the settings to fit your environment. This’ll mainly be the options VARNISH_LISTEN_ADDRESS and VARNISH_STORAGE_SIZE

VARNISH_RUN_USER=varnish
VARNISH_RUN_GROUP=varnish

# Maximum number of open files (for ulimit -n)
NFILES=131072
# Locked shared memory (for ulimit -l)
# Default log size is 82MB + header
MEMLOCK=82000
# Maximum number of threads (for ulimit -u)
NPROCS="unlimited"
# Maximum size of corefile (for ulimit -c). Default in Fedora is 0
# DAEMON_COREFILE_LIMIT="unlimited"

RELOAD_VCL=1

# # Should probably change this
VARNISH_VCL_CONF=/etc/varnish/default.vcl

# # Not setting VARNISH_LISTEN_ADDRESS makes Varnish listen on all IPs on this box
# # (Both IPv4 and IPv6 if available). Set manually to override this.
# VARNISH_LISTEN_ADDRESS=
VARNISH_LISTEN_PORT=80

# # Telnet admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082

# # Shared secret file for admin interface
VARNISH_SECRET_FILE=/etc/varnish/secret

# # The minimum number of worker threads to start
VARNISH_MIN_THREADS=50

# # The Maximum number of worker threads to start
VARNISH_MAX_THREADS=5000

# # Idle timeout for worker threads
VARNISH_THREAD_TIMEOUT=120

# Best option is malloc if you can. malloc will make use of swap space smartly if
# you have it and need it.
VARNISH_STORAGE_TYPE=malloc

# # Cache file size: in bytes, optionally using k / M / G / T suffix,
# # or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=2G

VARNISH_STORAGE="${VARNISH_STORAGE_TYPE},${VARNISH_STORAGE_SIZE}"

# # Default TTL used when the backend does not specify one
VARNISH_TTL=60

# # DAEMON_OPTS is used by the init script.  If you add or remove options, make
# # sure you update this section, too.
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT}
             -f ${VARNISH_VCL_CONF}
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT}
             -t ${VARNISH_TTL}
             -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT}
             -u ${VARNISH_RUN_USER} -g ${VARNISH_RUN_GROUP}
             -S ${VARNISH_SECRET_FILE}
             -s ${VARNISH_STORAGE}"

Now that we’ve got our config sorted we need to edit the config file that tells Varnish what to do with various requests and whatnot. In the example above it’ll be /etc/varnish/default.vcl – but will be whatever you set VARNISH_VCL_CONF to be.

Again it is worth grabbing a copy of your current config (or the sample) in case you need it later.

In here we want the following config. Be sure to modify as required.

In the director’s .list you’ll need to change the range that Varnish allows as backends. Please don’t set this to a /8 or anything too big (/24s at most work nicely I’ve found) else it’ll take forever for Varnish to start up. A /8 means Varnish will attempt to make 16 million or so backends.. Yeah.

It does need to allow for all possible backend IPs however. If the IP you set later in DNS doesn’t exist here Varnish wont try to connect to it and will output an error. You can specify each IP as a /32 if you like, but in my case I know all of my web servers will have a 10.5.0. IP so “10.5.0.0”/24 works nicely.

Also change the .suffix option from .ws.int to the internal DNS zone you’re using, e.g. .internal.example.com

/* Does a DNS lookup on .ws.int
 * if result is one of the listed IPs, use that IP as backend
 */

director default dns {
    .list = {
        .port = "80";
        .connect_timeout = 5s;
        .first_byte_timeout = 600s;
        .between_bytes_timeout = 600s;
        .max_connections = 10000;
        "10.5.0.0"/24;
    }
    .ttl = 1m;
    .suffix = ".ws.int";
}

acl purge {
    "127.0.0.0"/8;
    "10.0.0.0"/8;
}

sub vcl_recv {
    set req.grace = 10s;

    if (req.request == "PURGE") {
        if (!client.ip ~ purge) {
            error 405 "Not allowed.";
        }
        return (lookup);
    }

    if (req.request != "GET" &&
        req.request != "HEAD" &&
        req.request != "PUT" &&
        req.request != "POST" &&
        req.request != "TRACE" &&
        req.request != "OPTIONS" &&
        req.request != "DELETE") {
           /* Non-RFC2616 or CONNECT which is weird. */
           error 405 "Not allowed.";
        }

    if (req.request != "GET" && req.request != "HEAD") {
         /* We only want to cache GET and HEAD */
         return (pass);
    }
    if (req.http.Authorization || req.http.Cookie) {
        /*
         * Not cacheable by default. Usually means its an authenticated request,
         * which we don't want to accidentally server to another user
         */
        return (pass);
    }

    /* Otherwise we're good. Send it to the cache logic */
     return (lookup);
}

sub vcl_pipe {
     return (pipe);
}

sub vcl_pass {
     return (pass);
}

sub vcl_hash {
     hash_data(req.url);
     if (req.http.host) {
          /* Add the requested domain/virtual host to the hash */
          hash_data(req.http.host);
     } else {
          /* Server IP if it's not specified */
          hash_data(server.ip);
     }
     return (hash);
}

sub vcl_hit {
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
return (deliver);
}

sub vcl_miss {
    if (req.request == "PURGE") {
         purge;
         error 200 "Purged.";
    }
    return (fetch);
}

sub vcl_fetch {
    /* How old should we allow the cache to be if the backend server doesn't respond? */
    set beresp.grace = 10m;
    set beresp.http.Vary = "Accept-Encoding";

    if (beresp.ttl <= 0s ||
        beresp.http.Set-Cookie ||
        beresp.http.Vary == "*") {
            /*
             * Mark as "Hit-For-Pass" for the next 2 minutes
             */
             set beresp.ttl = 120 s;
             return (hit_for_pass);
         }
    return (deliver);
}

sub vcl_deliver {
    /* It's a bit paranoid, but lets not show all of our cards to the other players */
    remove resp.http.X-Varnish;
    remove resp.http.Via;
    remove resp.http.X-Powered-By;
    remove resp.http.X-Secure;
    set resp.http.Server = "Apache";
    return (deliver);
}

sub vcl_error {
    set obj.http.Content-Type = "text/html; charset=utf-8";
    set obj.http.Retry-After = "60";
    synthetic {"<!DOCTYPE html>
<html lang='en'>
<head>
<meta charset='utf-8'>
<title>503 Service Unavailable</title>
<style>
::-moz-selection{background:#b3d4fc;text-shadow:none}::selection{background:#b3d4fc;text-shadow:none}html{padding:30px 10px;font-size:20px;line-height:1.4;color:#000;background:#f0f0f0;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}html,input{font-family:'Helvetica Neue',Helvetica,Arial,sans-serif}body{max-width:1024px;_width:1024px;padding:30px 20px 50px;border:1px solid #b3b3b3;border-radius:4px;margin:0 auto;box-shadow:0 1px 10px #a7a7a7,inset 0 1px 0 #fff;background:#fcfcfc}h1{margin:0 10px;font-size:50px;text-align:center}h1 span{color:#bbb}h3{margin:1.5em 0 .5em}p{margin:1em 0}ul{padding:0 0 0 40px;margin:1em 0}.container{max-width:960px;_width:960px;margin:0 auto}
</style>
</style>
</head>
<body>
<div class='container'>
<h1>503 Service Unavailable</h1>
<p>Apologies, it appears the server handling this request is unavailable, overloaded or just completely broken..</p>
<p>Please try again later</p>
</div>
</body>
</html>"};
    return (deliver);
}

sub vcl_init {
    return (ok);
}
sub vcl_fini {
    return (ok);
}

Now you’re free to start Varnish up!

service varnish start

To finish it off and make it all work add a web server by adding an A-record to the internal zone. For example if icnerd.com was served by a machine on 10.5.0.6 you’d configure an A record icnerd.com.ws.int. with the IP of 10.5.0.6 and as low a TTL as you can possibly set.
If your nameservers support it you can also set a wildcard record *.icnerd.com.ws.int. too, or just www.icnerd.com.ws.int. and any other domains you’d use to access this site.

When you request icnerd.com through the Varnish server we set up above, it’ll now connect to 10.5.0.6 on port 80 as its backend, caching anything it can. Awesome.

To have Varnish do a new lookup on a backend server (e.g. the backend’s IP has changed) issue a service varnish reload to have it re-lookup the domain without losing its cache.

From here the world is your oyster, because Drew, that’s all the world really is. Maybe you could have a script automatically provision web servers and add them to DNS, issuing a reload to Varnish when done. Good stuff.

Hope this helps!

Leave a Reply