Introduction
I have been looking into node.js for a while and have been coding a few toy projects with it. I'm infamously known between my friends as that idiot who loves to premature optimize everything and is really hung up on performance even when it makes no sense.
It's a seriously crippling flaw to have because it's hard for me to choose a technology knowing it could be inferior to something else in terms of performance.
I recently looked into clojure because I watched Rich Hickey's "simple made easy" talk and I really enjoyed it. I have no prior experience with Java. I just know a lot of people love to bash it for being so verbose.
My first instinct was to setup benchmarks that tried to semi-emulate a real world site and here I am, posting this article to show the results.
Test setup
Node: Version 0.8.14
Express 3.0.0 with Jade for tests that use templates. I also only loaded up the body parser, static server and session/cookie middleware packages because I figured these would be used 80% of the time.
I ran node with NODE_ENV=production, and as a note I will say if you're using node on a public facing site you best be using this before starting your server because it makes a tremendous difference for the better.
Clojure: Version 1.4
Ring 1.1.6 with Compojure 1.1.3 and Hiccup for tests that use templates. I did my best to match up the same set of middleware functionality. Today was day #1 with clojure so I probably did a few things wrong. I loaded it up with the file-info, params, keyword-params, nested-params, cookies and session middleware.
I also am using Jetty as the web server inside ring which is the default. Is there something better for a general use case?
My computer: C2D 2.13ghz with 2GB of ram and a 7200 RPM SATA drive
I am running everything inside of a light weight lubuntu VM with virtual box. The VM has 256mb of ram allocated to it and it's hard capped to only using 1 core. Overall it's pretty slow so don't be afraid when you see low requests/seconds. Just remember that it's relative to my computer speed, so the test ratios between technologies should be accurate.
I used apache bench with -n 5000 and -c x where x = 1, 100, 500 and 1000. I ran each test 7 times. I ignored the first and last and took an unofficial average of the middle 5. Unofficial as in, I didn't sit there and get a perfect average. I eyeballed the values and did quick mental math.
There was absolutely no tuning done to the JVM because I'm new to it. I don't know if there's anything I can do. Does jetty have a production mode like node? I'd really like any advice on how to tune the JVM or make clojure more production ready other than running the defaults.
These tests also involve no I/O at all. There's no calls to third party sites or databases. I'm just performing addition on a session variable for each request. I don't know how to setup an async test properly because if I wrap the response in a setTimeout() then it takes REQUESTS * delay to run apache bench because it blocks.
I think this is unfair to node because it's all about async and there's a very good chance a typical website will be making calls to a database. Maybe someone can modify the tests to include db calls with both languages?
HTTP headers
It's important to make note of what is being sent and processed by each server.
Express returns:
User-Agent: Opera/9.80 (Windows NT 6.1; U; en) Presto/2.10.289 Version/12.02
Host: 192.168.1.7:3000
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate
Cookie: connect.sid=s%3AU3MnXXYj1%2FR8ijTm0tAPgtjW.cXdM3F%2Bg71%2B8iFi6Cy0XC3tDnoxhtuBCsDH59vtAkR4
Cache-Control: no-cache
Connection: Keep-Alive
-RESPONSE-:
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 4639
ETag: "-1106054829"
Date: Sat, 27 Oct 2012 19:21:31 GMT
Connection: keep-alive
Clojure returns:
User-Agent: Opera/9.80 (Windows NT 6.1; U; en) Presto/2.10.289 Version/12.02
Host: 192.168.1.7:3000
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate
Cookie: ring-session=fa24bfe4-955e-4d72-ab65-ec2b1e5b3af7
Cache-Control: no-cache
Connection: Keep-Alive
-RESPONSE-:
Date: Sat, 27 Oct 2012 19:47:40 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 4645
Server: Jetty(7.6.1.v20120215)
As you can see express is calculating an etag and ring is not. I couldn't figure out how to get etags working in ring. There appeared to be middleware for it but it was 2 years old and had no documentation. The ring documentation seemed to be a little messed up too. I am sad to say we will have to deal with this inconsistency for now.
There's also slight byte variance in most tests due to how express and ring differ in handling template outputs. I did my best to manually change each test to get them close but I didn't make them exact.
Whenever you see bytes transferred it's just HTML transferred, not total bytes.
Tests
Test #1: Templates and ~4.6kb of html
Node: (4,639 bytes)
-c 1: 745 reqs/second | 1.34ms per req | 1.34ms each conc req
-c 100: 888 reqs/second | 112ms per req | 1.13ms each conc req
-c 500: 888 reqs/second | 112ms per req | 1.13ms each conc req
-c 1000: 760 reqs/second | 1310ms per req | 1.41ms each conc req
Clojure: (4,645 bytes)
-c 1: 938 reqs/second | 1.06ms per req | 1.06ms each conc req
-c 100: 992 reqs/second | 107ms per req | 1.07ms each conc req
-c 500: 829 reqs/second | 603ms per req | 1.21ms each conc req
-c 1000: 487 reqs/second | 2050ms per req | 2.05ms each conc req
Test #2: Templates and ~42kb of html
Node: (40,783 bytes)
-c 1: 334 reqs/second | 2.99ms per req | 2.99ms each conc req
-c 500: 222 reqs/second | 2246ms per req | 3.50ms each conc req
-c 1000: 190 reqs/second | 5256ms per req | 5.26ms each conc req
Clojure: (45,208 bytes)
-c 1: 462 reqs/second | 2.16ms per req | 2.16ms each conc req
-c 500: 318 reqs/second | 1569ms per req | 3.14ms each conc req
-c 1000: 225 reqs/second | 4432ms per req | 4.43ms each conc req
Test #3: Templates and ~640 bytes of html
Node: (637 bytes)
-c 1: 892 reqs/second | 1.12ms per req | 1.12ms each conc req
-c 500: 982 reqs/second | 508ms per req | 1.08ms each conc req
-c 1000: 929 reqs/second | 1075ms per req | 1.07ms each conc req
Clojure: (650 bytes)
-c 1: 1118 reqs/second | 0.98ms per req | 0.98ms each conc req
-c 500: 613 reqs/second | 814ms per req | 1.63ms each conc req
-c 1000: 429 reqs/second | 2328ms per req | 2.33ms each conc req
Test #4: No templates and ~700 bytes of html
Node: (704 bytes)
-c 1: 1000 reqs/second | 0.999ms per req | 0.999ms each conc req
-c 500: 966 reqs/second | 517ms per req | 1.03ms each conc req
-c 1000: 903 reqs/second | 1107ms per req | 1.11ms each conc req
Clojure: (695 bytes)
-c 1: 1384 reqs/second | 0.722ms per req | 0.722ms each conc req
-c 500: 1109 reqs/second | 450ms per req | 0.901ms each conc req
-c 1000: 756 reqs/second | 1321ms per req | 1.322ms each conc req
Conclusion
From what I can see clojure is actually really reasonable. For low concurrency it pulls a head of node but once concurrency starts to occur it gets really bogged down but then you have to ask yourself, will you ever have an app serving 500 or more concurrenct connections with the amount of bytes being transferred? I'm not really sure. What do you guys think?
I also would be interested to see how both perform on real servers. Like a quad core instance on some VPS or amazon because my test conditions are on a really old computer. Especially so if someone fixed my tests to include etag calculations in clojure and simulated I/O for both versions.
A websockets test would be really interesting too. Who wants to set that up? Gogogo.
Thanks for reading.
The code
You can download both sets of code here:
http://www3.zippyshare.com/v/51790013/file.html
It's split into clojure and node folders. I didn't include each test variation though. I included the files for both the template and non-template app files. Just modify the string being sent in the response (or in the template) to vary the size.
