Using varnish to accelerate file serving out of GridFS

After migrating some image files from file system to gridfs and serving them using nginx-gridfs, I realized that serving files out of gridfs is significantly slower than before (This is also stated in this article.). Even worse, since there is no non-blocking implementation of mongodb c driver, fetching single file from gridfs blocks nginx worker from accepting other requests. So it’s necessary to minimize gridfs queries by using caches.

Caching in front of web server

Varnish is a caching HTTP reverse proxy, can be installed in front of any server that speaks HTTP and configured to cache the contents.

Below is a brief result of running a reply-rate benchmark on four different configurations:

Requests/second	without varnish	with varnish
filesystem	392.34	641.98
GridFS	161.95	646.54

From the “without varnish” column we can discover that, serving files from GridFS degrades the performance by 58.7% compared to serving directly from file system. But after setting up the cache, they performs almost the same. Further more, they both take the great benefit of the cache acceleration, which makes it quite fast for serving static files.

Be careful of Cookies

When taking cache proxy into production, things are not always ideal. By default, varnish will not process a request as cacheable if there is a cookie header. Use a cookie-free domain for static files if possible.

In some circumstance you may want to ignore particular cookies, this can be done by adding some code to the vcl_recv section of the configuration file, like:

if (req.http.Cookie) { set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *__utm.=[^;]+;? *", "\1"); if (req.http.Cookie == "") { remove req.http.Cookie; } }

Appendix: Raw Results of Apache Benchmark

The benchmark runs on a Virtualbox virtual machine, with single CPU core, 512MB RAM and 8GB disk.

Read filesystem, without varnish cache:

$ ab -n10000 -c10 http://192.168.1.183:8080/fs/image/2012/05/11/small_e3491035-6bb5-469f-a20c-eaa9f2ec7e3f.jpg   Server Software: nginx/1.2.0 Server Hostname: 192.168.1.183 Server Port: 8080   Document Path: /fs/image/2012/05/11/small_e3491035-6bb5-469f-a20c-eaa9f2ec7e3f.jpg Document Length: 32095 bytes   Concurrency Level: 10 Time taken for tests: 25.488 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 323790000 bytes HTML transferred: 320950000 bytes Requests per second: 392.34 [#/sec] (mean) Time per request: 25.488 [ms] (mean) Time per request: 2.549 [ms] (mean, across all concurrent requests) Transfer rate: 12405.81 [Kbytes/sec] received   Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.1 0 3 Processing: 4 25 9.6 22 111 Waiting: 2 23 8.5 20 97 Total: 4 25 9.5 22 112   Percentage of the requests served within a certain time (ms) 50% 22 66% 23 75% 25 80% 27 90% 36 95% 48 98% 57 99% 62 100% 112 (longest request)

Read GridFS, without varnish cache:

$ ab -n10000 -c10 http://192.168.1.183:8080/gridfs/image/2012/05/11/small_e3491035-6bb5-469f-a20c-eaa9f2ec7e3f.jpg   Server Software: nginx/1.2.0 Server Hostname: 192.168.1.183 Server Port: 8080   Document Path: /gridfs/image/2012/05/11/small_e3491035-6bb5-469f-a20c-eaa9f2ec7e3f.jpg Document Length: 32095 bytes   Concurrency Level: 10 Time taken for tests: 61.748 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 323990000 bytes HTML transferred: 320950000 bytes Requests per second: 161.95 [#/sec] (mean) Time per request: 61.748 [ms] (mean) Time per request: 6.175 [ms] (mean, across all concurrent requests) Transfer rate: 5123.97 [Kbytes/sec] received   Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.1 0 6 Processing: 11 61 25.3 58 696 Waiting: 10 54 20.3 54 637 Total: 11 62 25.3 58 696   Percentage of the requests served within a certain time (ms) 50% 58 66% 63 75% 68 80% 71 90% 84 95% 96 98% 106 99% 115 100% 696 (longest request)

Read GridFS, with varnish cache:

$ ab -n10000 -c10 http://192.168.1.183/gridfs/image/2012/05/11/small_e3491035-6bb5-469f-a20c-eaa9f2ec7e3f.jpg   Server Software: nginx/1.2.0 Server Hostname: 192.168.1.183 Server Port: 80   Document Path: /gridfs/image/2012/05/11/small_e3491035-6bb5-469f-a20c-eaa9f2ec7e3f.jpg Document Length: 32095 bytes   Concurrency Level: 10 Time taken for tests: 15.467 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 324840000 bytes HTML transferred: 320950000 bytes Requests per second: 646.54 [#/sec] (mean) Time per request: 15.467 [ms] (mean) Time per request: 1.547 [ms] (mean, across all concurrent requests) Transfer rate: 20509.98 [Kbytes/sec] received   Connection Times (ms) min mean[+/-sd] median max Connect: 0 1 30.0 0 1003 Processing: 1 14 7.9 13 78 Waiting: 0 11 6.2 10 75 Total: 1 15 30.9 13 1023   Percentage of the requests served within a certain time (ms) 50% 13 66% 16 75% 19 80% 20 90% 25 95% 28 98% 34 99% 39 100% 1023 (longest request)

Read filesystem, with varnish cache:

$ ab -n10000 -c10 http://192.168.1.183/fs/image/2012/05/11/small_e3491035-6bb5-469f-a20c-eaa9f2ec7e3f.jpg   Server Software: nginx/1.2.0 Server Hostname: 192.168.1.183 Server Port: 80   Document Path: /fs/image/2012/05/11/small_e3491035-6bb5-469f-a20c-eaa9f2ec7e3f.jpg Document Length: 32095 bytes   Concurrency Level: 10 Time taken for tests: 15.577 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 324420000 bytes HTML transferred: 320950000 bytes Requests per second: 641.98 [#/sec] (mean) Time per request: 15.577 [ms] (mean) Time per request: 1.558 [ms] (mean, across all concurrent requests) Transfer rate: 20339.06 [Kbytes/sec] received   Connection Times (ms) min mean[+/-sd] median max Connect: 0 2 43.6 0 1003 Processing: 1 13 8.4 12 109 Waiting: 0 10 6.8 10 90 Total: 1 15 44.4 12 1033   Percentage of the requests served within a certain time (ms) 50% 12 66% 16 75% 18 80% 20 90% 24 95% 28 98% 33 99% 38 100% 1033 (longest request)