Thursday, May 1, 2008

Does Django really scale better than Rails?

I do quite a bit of Rails work. A very good bit actually...perhaps 60% of my professional time is now spent on Rails.

I like it, but have heard horror stories regarding Rails scaling. I personally haven't run into any problems, but most of our applications run internally at businesses serving hundreds of users...not many thousands or millions.

I recently read an article on highscalability.com about a Rails-based Facebook app that was serving up 300 million page views monthly. The article describes (and touts) the way they've scaled the application across 12 servers:
- 4 DB servers, 6 application servers, 1 staging server, and 1 front end server.
- 6, 4 core 8 GB application servers.
- Each application server runs 16 mongrels for a total of 96 mongrels. -
- 4 GB memcache instance on each application server
- 2 32GB 4 core servers with 4x 15K SCSI RAID 10 disks in a master-slave setup
Holy crap Batman...that's a lot of iron. I did a double take when I saw this....I mean, I know 9.6 million requests a day is a lot, but this is a very simple application...a Facebook toy...not what I would consider representative of a significantly complex web application (please someone correct me if I'm wrong). I mean, if 12 servers were used to serve up something like Facebook itself, I'd understand, but a widget for Facebook? Something smells.

Again, I'm no scaling expert, and I'm perhaps a little bitter that I've never had written an app that generated 9.6 mil page views daily, but that's beside the point. To me, what this says is that Rails may scale, but you'd better have lots of cash on hand to do so. Please, if this is expected of an app with this many hits, beit PHP or Java or Perl or Python or Ruby, please let me know.

While living in Ruby camp for the past several years, one common drum I hear thumping from the Python camp is that fact that Python, and particularly, Pylons and Django, are superior from a scaling perspective. However, while researching this today, I've had a hard time coming up with cold hard facts. I'm working on a small app I'd like to dream will catch on one day, and I'm considering ridding myself of the scaling problems early by switching to Django; that is, if Django really avoids the problems Rails faces.

I can't find any articles stating that it does (supported by facts and figures), nor can I find examples of heavily hit applications in the wild and what they're using. I do realize that the Washington Post uses Django, but I've also read that it's not used pervasively and mainly for smaller, lower-traffic sections of the site.

So, help a Python/Django newb understand...is there something inherently better about Python versus Ruby, or Django versus Rails, that makes the first scale and perform better than the latter?

Thanks guys. I appreciate your continued patience and commentary.

46 comments:

Joe said...

It's not hard to find some hard numbers that folks have posted re: scaling Django. Google for "Django Scaling" had this great post in the top 3 hits:

http://www.djangoproject.com/weblog/2007/may/25/curse/

There are others out there too, but really the work is in what you're doing. Django has some excellent scaling features: like caching being a core part of the game from the start and mod_python to drive heavy load with Apache. Ultimately, it will all depend on how you write your app.

skeptictank said...

Well, if you're already reasonably happy with Ruby you might give Merb a try:
http://merbivore.net/index.html

If you're really worried about performance why not go with a compiled language framework:
http://ocsigen.org/

Ocsigen is written in OCaml which tends to do quite well on the Alioth language shootout benchmarks. And it's a pretty nice language in it's own right once you get used to some of the syntactical quirks.

Fiona said...

I'd highly recommend you try out Python/Django anyway, even if just for this one project.

Having another language under your belt is a great thing - and then once you've built it you can do the research on scalability yourself, while being able to put another skill on your CV.

Who knows, you might like Python? I personally love it.

Anonymous said...

personally i think it all boils down to the application design as a whole. i've written apps that get the millions of hits a month using coldfusion and sql server and have never had a problem scaling.

Anonymous said...

" I mean, if 12 servers were used to serve up something like Facebook itself, I'd understand, but a widget for Facebook?"

Facebook runs on 10,000 servers (literally). Also, six of those servers (DB, staging, load balancing) will be there no matter what language you use. Changes languages won't solve DB performance issues. You'd also probably have the same number of memcached servers, so what you're really talking about is four servers for the app. Can Python language bring that down to three? Perhaps. Two? Hard to tell. One? Doubtful. So you're talking about saving one or two servers.

pjz said...

From a practical point of view, django scales *easier* since it can run on GAE... is there a similar service for RoR ?

Todd Hoff said...

pjz, Python isn't really key to with GAE scales. It will scale well for new languages too. It's shared nothing so it can start a lot of instance to handle load. And it makes you compose your service in terms of already scalable services so there's no choke point to prevent scaling More at http://highscalability.com/google-appengine-first-look.

Ryan Heneise said...

FWIW, Facebook itself lives on over 10,000 servers.
source: http://www.insidefacebook.com/2008/04/23/facebook-infrastructure-up-to-10000-web-servers/ and http://www.datacenterknowledge.com/archives/2008/Apr/23/facebook_now_running_10000_web_servers.html

Vita Rara said...

So, by my math, that's just under 10pps. That seems really really low given the size of the infrastructure.

Is Rails really that slow?

woeye said...

I really read comments like " Changes languages won't solve DB performance issues." rather often these days. Maybe it's time to rethink the behaviour on databases and stop treating them as a black box? ORM might be nice. But in order to get maximum performance it might help to get the most out of the database as well. Especially the Rails guys drive me mad with their ignorance about databases. Saying joins are always bad is really silly if you use the right database and now how to leverage its power properly.

pk11 said...

without telling what the requirement is it's impossible to talk about scalability. is django more scalable? depends.


rails/ruby and python/django are really similar in nature and probably it all comes down to whether you prefer implicit vs explicit.


anyway, for python/django scalability see:
http://highscalability.com/tags/python

http://highscalability.com/yandex-architecture

http://antoniocangiano.com/2008/03/20/djangos-tipping-point/

http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks/

http://mydjangoblog.com/2008/04/24/deploying-and-tuning-django/

Jake said...

@pk11: fair enough...use Twitter as an example. What about Django would allow it to avoid the problems the Twitter guys have had?

@skeptictank: I've wanted to try merb, but found the docs very weak when I looked.

@anonymous: doesn't coldfusion run ultimately in a java servlet container? I wouldn't be surprised that it scaled well.

@other anonymous: ok, when mentioning facebook I was really meaning any social network app...the point is that the facebook widget they provide is silly simple, and I shouldn't take 10 servers to run it.

@pjz: come on, that's a weak point. GAE forces you to make some hard decisions about your app design and locks you in to always run on their servers. I'd trade scalability for portability any day.

@vita rara: exactly my thoughts.

pk11 said...

jake,
i would not use neither django nor rails for a twitter kind of application, since both of these frameworks really shine if you have to build a traditional db-backed web application. (twitter is more like a messaging service)

for twitter, i would probably use activemq or something like that for storage instead of the db and python or ruby with a templating and routing library.

Luke Hoersten said...

I think the anon user put it best: "six of those servers (DB, staging, load balancing) will be there no matter what language you use. Changes languages won't solve DB performance issues."

So when we say scalability, we're talking load balancing, caching, and DB scalability. Load bal. and DB scale are largely language unspecific and solved with reverse proxies (etc.) and scalable DBs. Caching is made easier by certain langs but apps like Memcached can be used almost agnostically. But scalability also includes parallelism and distribution from the application aspect, not just the application architecture.

The language aspect boils down to how well the language runtimes scales. In other words, how much overhead does it take to run your applications. Ruby has a much younger runtime and probably runs slower but I'm willing to bet it's a constant factor (though I'm not sure how large). Real language scalability comes from factoring out the data and task parallelizable code and distributing it. As far as I know with both Ruby and Python, there is no good SMP support or distribution primitives. Without SMP on the application side, your left with web server coarse grain parallelism.

Spawning off or threading slow procedures is critical. For most of the tasks, forcing the main thread to wait isn't even necessary. I recently wrote an article about how to use Python/Django + Erlang for these type of tasks. You could easily use Ruby/Rails as well but the key is using a language with distribution and concurrency primitives for the fine grain application parallelism.

http://humani.st/scalable-web-apps-erlang-python/

Facebook used this technique with their new Facebook chat product.

the daniel said...

Use whatever framework you want. It's just there to fill your memcache anyway - I'd worry more about smart cache expiry than anything else.

zephyr said...

I am not gone through the scalability issues with django. however the use of the django as first supported framework by Google Appengine, this means lot. At least googler has seen future of django.
so its better if you try django.

Kador said...

At the end, the source of the problem is always those disk drives that lie below ...

cyx said...

Hey guys,

Google didn't use django because it was better than rails / merb or anything, but because it's probably the best framework on python.

Google only uses 4 techs (javascript, python, C++, java) and you'd probably guess that python is the most feasible language among the 4 for google app engine.

As for not using JOINs, its not specific to Rails or ruby programmers, and don't even start by saying that rails programmers are ignorant on DB. If you've worked for any of these social networks, you'd know that database sharding is the only way to get by these super large datasets, and if you'd even consider joining on sharded tables I"d say your insane.

Google Bigtable doesn't even have any notions of JOINS as far as I know.

Roger Pack said...

it is pretty true--from my own experience django is 2x-6x faster than rails--so probably 1/4 the number of servers needed :)
That being said, merb is faster, and for small projects (and/or prototypes), they're both ok it seems.
-=R

sohbet15 said...

Thank you very much for this useful article and the comments.
sohbet

bedava chat said...

iste size çetsitesi bedava -bedava chat ortamı vardır-sohbet odasıherşey için tesekkurler-chat sitesi yorumlar sizden-
chat sitesi- destekleme bizden chat sitesi- alem buysa chat odası- kral benim :) sex chat- cinsellik serbest sex sohbet

Anonymous said...

Inconclusive results as in many other such discussions.

sigh ...

ethicMinds
george.naing.blog@gmail.com
george kyaw naing

rane said...

as far as application servers are concerned, Rails is on the slower side, but imho thats rarely the bottle neck. If you are using a RDBMS, then joins etc. are going to be equally costly. If you are doing CPU heavy activity then both of them are equally good/bad except that python has numpy/scipy and ruby has no equivalent. But If you are comfortable with ruby, you will find django much more flexible... if you don't want to shift try optimizing your DB or use a noSQL alternatives.

bedava chat said...

chat konya chat bedava chat ankara chat ankara sohbet chat ankara çet arkadaşlık siteleri chat çet chat odaları

bedava chat said...

sohbet güzel sözler chat

chat said...

değişik

konyachat said...

konyachat, konya chat

konya chat said...

konyachat, konya chat

chat siteleri said...

chat, chat siteleri

ercan said...

sohbet Availability, and also chat more than the muhabbet rooms, male female Çet friends.

youlacka said...

That is really very good article.Thanks! kizlarla chatcetseviyeli sohbetchat odalarisohbet odalarisohbet odalariask sozlerisevgi sozleri

ercan said...

sohbet
chat
sex sohbet
cinsel sohbet
sex chat
seks sohbet

DeadLy said...

sohbet

Anonymous said...

lez sohbet odaları
lez sohbet
cinsel sohbet odaları
türkiye sohbet
çet odaları

chat
yetiskin sohbet
denizli sohbet
chat sohbet
cinsel sohbet

altan said...

Sohbet Odaları

Anadolu Design said...

Totally love how this came out--looks like something from an editorial. Love the feather earrings.
cam balkon
alüminyum
düğün salonu
çankırı

sohbet odaları said...

bedava sohbet
yazgülü sohbet
sevda chat
adana sohbet
kerizim
sohbet kanalları
yazgülü
sohbet yap
yonja
cinsel sohbet

Avrasya Dizayn said...

Perhaps. Two? Hard to tell. One? Doubtful. So you're talking about saving one or two servers.

cialis

küpeşte

sineklik

dugun salonu said...

Totally love how this came out--looks like something from an editorial. Love the feather earrings.

maltepe dugun salonlari said...


There are others out there too, but really the work is in what you're doing..

cinsellik sohbet said...

Very informative and trustworthy blog. Please keep updating with great posts like this one.
cinsel sohbet
I have booked marked your site and am about to email it to a few friends of mine that I know would enjoy reading
çet sohbet
Thanks for your explanation was very good effort, while health information in your hand
sohbetci

They look very nice, wonderful =)

sohbet net -
yazgülü sohbet -
seviyeli sohbet -
adana sohbet -
kerizim -
çet sohbet -
yazgülü -
sohbet et -
yonja -
cinsel chat

sohbet odaları said...

Thanks admin please :D please

kaan kara said...

Harika popüler 2013 güzel sözler yollayabilirsiniz ayrıca gülmek isterseniz komik sözler bulabilirsiniz, duygusallaşmak istiyorsanızda hemen duygusal sözler'e bakabilirsiniz.

erotorrent said...

Awesome site! I am loving it!! Will come back again. I am bookmarking your feeds also. Very informative post. Looking more to something like this

mynet Sohbet said...

PayLaşım iÇin TeşekkürLer.
bedava sohbet - mynet sohbet - sohbet odalari - güzel sözler - sohbet net - cinsel sohbet - sohbetci - seviyorsun - güzel sözler fullarabeskrap - Çet Sohbet Siteleri

hdfilm izle said...

full izle
film hd izle
vizyon filmler seyret
dis cephe mantolama
hd film izle vk
hd vizyon film
en iyi film sitesi
hd full vizyon
boyaci
hd film full izle
boya dekorasyon
turkce dublaj film izle
filmi full izle
hd full izle