Thursday, May 1, 2008

Does Django really scale better than Rails?

I do quite a bit of Rails work. A very good bit actually...perhaps 60% of my professional time is now spent on Rails.

I like it, but have heard horror stories regarding Rails scaling. I personally haven't run into any problems, but most of our applications run internally at businesses serving hundreds of users...not many thousands or millions.

I recently read an article on highscalability.com about a Rails-based Facebook app that was serving up 300 million page views monthly. The article describes (and touts) the way they've scaled the application across 12 servers:
- 4 DB servers, 6 application servers, 1 staging server, and 1 front end server.
- 6, 4 core 8 GB application servers.
- Each application server runs 16 mongrels for a total of 96 mongrels. -
- 4 GB memcache instance on each application server
- 2 32GB 4 core servers with 4x 15K SCSI RAID 10 disks in a master-slave setup
Holy crap Batman...that's a lot of iron. I did a double take when I saw this....I mean, I know 9.6 million requests a day is a lot, but this is a very simple application...a Facebook toy...not what I would consider representative of a significantly complex web application (please someone correct me if I'm wrong). I mean, if 12 servers were used to serve up something like Facebook itself, I'd understand, but a widget for Facebook? Something smells.

Again, I'm no scaling expert, and I'm perhaps a little bitter that I've never had written an app that generated 9.6 mil page views daily, but that's beside the point. To me, what this says is that Rails may scale, but you'd better have lots of cash on hand to do so. Please, if this is expected of an app with this many hits, beit PHP or Java or Perl or Python or Ruby, please let me know.

While living in Ruby camp for the past several years, one common drum I hear thumping from the Python camp is that fact that Python, and particularly, Pylons and Django, are superior from a scaling perspective. However, while researching this today, I've had a hard time coming up with cold hard facts. I'm working on a small app I'd like to dream will catch on one day, and I'm considering ridding myself of the scaling problems early by switching to Django; that is, if Django really avoids the problems Rails faces.

I can't find any articles stating that it does (supported by facts and figures), nor can I find examples of heavily hit applications in the wild and what they're using. I do realize that the Washington Post uses Django, but I've also read that it's not used pervasively and mainly for smaller, lower-traffic sections of the site.

So, help a Python/Django newb understand...is there something inherently better about Python versus Ruby, or Django versus Rails, that makes the first scale and perform better than the latter?

Thanks guys. I appreciate your continued patience and commentary.

44 comments:

Joe said...

It's not hard to find some hard numbers that folks have posted re: scaling Django. Google for "Django Scaling" had this great post in the top 3 hits:

http://www.djangoproject.com/weblog/2007/may/25/curse/

There are others out there too, but really the work is in what you're doing. Django has some excellent scaling features: like caching being a core part of the game from the start and mod_python to drive heavy load with Apache. Ultimately, it will all depend on how you write your app.

Anonymous said...

Well, if you're already reasonably happy with Ruby you might give Merb a try:
http://merbivore.net/index.html

If you're really worried about performance why not go with a compiled language framework:
http://ocsigen.org/

Ocsigen is written in OCaml which tends to do quite well on the Alioth language shootout benchmarks. And it's a pretty nice language in it's own right once you get used to some of the syntactical quirks.

Anonymous said...

I'd highly recommend you try out Python/Django anyway, even if just for this one project.

Having another language under your belt is a great thing - and then once you've built it you can do the research on scalability yourself, while being able to put another skill on your CV.

Who knows, you might like Python? I personally love it.

Anonymous said...

personally i think it all boils down to the application design as a whole. i've written apps that get the millions of hits a month using coldfusion and sql server and have never had a problem scaling.

Anonymous said...

" I mean, if 12 servers were used to serve up something like Facebook itself, I'd understand, but a widget for Facebook?"

Facebook runs on 10,000 servers (literally). Also, six of those servers (DB, staging, load balancing) will be there no matter what language you use. Changes languages won't solve DB performance issues. You'd also probably have the same number of memcached servers, so what you're really talking about is four servers for the app. Can Python language bring that down to three? Perhaps. Two? Hard to tell. One? Doubtful. So you're talking about saving one or two servers.

pjz said...

From a practical point of view, django scales *easier* since it can run on GAE... is there a similar service for RoR ?

Todd Hoff said...

pjz, Python isn't really key to with GAE scales. It will scale well for new languages too. It's shared nothing so it can start a lot of instance to handle load. And it makes you compose your service in terms of already scalable services so there's no choke point to prevent scaling More at http://highscalability.com/google-appengine-first-look.

Anonymous said...

FWIW, Facebook itself lives on over 10,000 servers.
source: http://www.insidefacebook.com/2008/04/23/facebook-infrastructure-up-to-10000-web-servers/ and http://www.datacenterknowledge.com/archives/2008/Apr/23/facebook_now_running_10000_web_servers.html

Vita Rara said...

So, by my math, that's just under 10pps. That seems really really low given the size of the infrastructure.

Is Rails really that slow?

Anonymous said...

I really read comments like " Changes languages won't solve DB performance issues." rather often these days. Maybe it's time to rethink the behaviour on databases and stop treating them as a black box? ORM might be nice. But in order to get maximum performance it might help to get the most out of the database as well. Especially the Rails guys drive me mad with their ignorance about databases. Saying joins are always bad is really silly if you use the right database and now how to leverage its power properly.

Unknown said...

without telling what the requirement is it's impossible to talk about scalability. is django more scalable? depends.


rails/ruby and python/django are really similar in nature and probably it all comes down to whether you prefer implicit vs explicit.


anyway, for python/django scalability see:
http://highscalability.com/tags/python

http://highscalability.com/yandex-architecture

http://antoniocangiano.com/2008/03/20/djangos-tipping-point/

http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks/

http://mydjangoblog.com/2008/04/24/deploying-and-tuning-django/

Jake said...

@pk11: fair enough...use Twitter as an example. What about Django would allow it to avoid the problems the Twitter guys have had?

@skeptictank: I've wanted to try merb, but found the docs very weak when I looked.

@anonymous: doesn't coldfusion run ultimately in a java servlet container? I wouldn't be surprised that it scaled well.

@other anonymous: ok, when mentioning facebook I was really meaning any social network app...the point is that the facebook widget they provide is silly simple, and I shouldn't take 10 servers to run it.

@pjz: come on, that's a weak point. GAE forces you to make some hard decisions about your app design and locks you in to always run on their servers. I'd trade scalability for portability any day.

@vita rara: exactly my thoughts.

Unknown said...

jake,
i would not use neither django nor rails for a twitter kind of application, since both of these frameworks really shine if you have to build a traditional db-backed web application. (twitter is more like a messaging service)

for twitter, i would probably use activemq or something like that for storage instead of the db and python or ruby with a templating and routing library.

Anonymous said...

I think the anon user put it best: "six of those servers (DB, staging, load balancing) will be there no matter what language you use. Changes languages won't solve DB performance issues."

So when we say scalability, we're talking load balancing, caching, and DB scalability. Load bal. and DB scale are largely language unspecific and solved with reverse proxies (etc.) and scalable DBs. Caching is made easier by certain langs but apps like Memcached can be used almost agnostically. But scalability also includes parallelism and distribution from the application aspect, not just the application architecture.

The language aspect boils down to how well the language runtimes scales. In other words, how much overhead does it take to run your applications. Ruby has a much younger runtime and probably runs slower but I'm willing to bet it's a constant factor (though I'm not sure how large). Real language scalability comes from factoring out the data and task parallelizable code and distributing it. As far as I know with both Ruby and Python, there is no good SMP support or distribution primitives. Without SMP on the application side, your left with web server coarse grain parallelism.

Spawning off or threading slow procedures is critical. For most of the tasks, forcing the main thread to wait isn't even necessary. I recently wrote an article about how to use Python/Django + Erlang for these type of tasks. You could easily use Ruby/Rails as well but the key is using a language with distribution and concurrency primitives for the fine grain application parallelism.

http://humani.st/scalable-web-apps-erlang-python/

Facebook used this technique with their new Facebook chat product.

Anonymous said...

Use whatever framework you want. It's just there to fill your memcache anyway - I'd worry more about smart cache expiry than anything else.

zephyr said...

I am not gone through the scalability issues with django. however the use of the django as first supported framework by Google Appengine, this means lot. At least googler has seen future of django.
so its better if you try django.

Anonymous said...

At the end, the source of the problem is always those disk drives that lie below ...

Unknown said...

Hey guys,

Google didn't use django because it was better than rails / merb or anything, but because it's probably the best framework on python.

Google only uses 4 techs (javascript, python, C++, java) and you'd probably guess that python is the most feasible language among the 4 for google app engine.

As for not using JOINs, its not specific to Rails or ruby programmers, and don't even start by saying that rails programmers are ignorant on DB. If you've worked for any of these social networks, you'd know that database sharding is the only way to get by these super large datasets, and if you'd even consider joining on sharded tables I"d say your insane.

Google Bigtable doesn't even have any notions of JOINS as far as I know.

Roger Pack said...

it is pretty true--from my own experience django is 2x-6x faster than rails--so probably 1/4 the number of servers needed :)
That being said, merb is faster, and for small projects (and/or prototypes), they're both ok it seems.
-=R

bedava chat said...

iste size çetsitesi bedava -bedava chat ortamı vardır-sohbet odasıherşey için tesekkurler-chat sitesi yorumlar sizden-
chat sitesi- destekleme bizden chat sitesi- alem buysa chat odası- kral benim :) sex chat- cinsellik serbest sex sohbet

Anonymous said...

Inconclusive results as in many other such discussions.

sigh ...

ethicMinds
george.naing.blog@gmail.com
george kyaw naing

rane said...

as far as application servers are concerned, Rails is on the slower side, but imho thats rarely the bottle neck. If you are using a RDBMS, then joins etc. are going to be equally costly. If you are doing CPU heavy activity then both of them are equally good/bad except that python has numpy/scipy and ruby has no equivalent. But If you are comfortable with ruby, you will find django much more flexible... if you don't want to shift try optimizing your DB or use a noSQL alternatives.

bedava chat said...

sohbet güzel sözler chat

chat said...

değişik

konyachat said...

konyachat, konya chat

konya chat said...

konyachat, konya chat

chat siteleri said...

chat, chat siteleri

ercan said...

sohbet Availability, and also chat more than the muhabbet rooms, male female Çet friends.

Anonymous said...

sohbet

sohbet odaları said...

bedava sohbet
yazgülü sohbet
sevda chat
adana sohbet
kerizim
sohbet kanalları
yazgülü
sohbet yap
yonja
cinsel sohbet

Avrasya Dizayn said...

Perhaps. Two? Hard to tell. One? Doubtful. So you're talking about saving one or two servers.

cialis

küpeşte

sineklik

dugun salonu said...

Totally love how this came out--looks like something from an editorial. Love the feather earrings.

maltepe dugun salonlari said...


There are others out there too, but really the work is in what you're doing..

sohbet odaları said...

Thanks admin please :D please

Anonymous said...

Harika popüler 2013 güzel sözler yollayabilirsiniz ayrıca gülmek isterseniz komik sözler bulabilirsiniz, duygusallaşmak istiyorsanızda hemen duygusal sözler'e bakabilirsiniz.

erotorrent said...

Awesome site! I am loving it!! Will come back again. I am bookmarking your feeds also. Very informative post. Looking more to something like this

Anonymous said...

I enjoyed your weblog. It’s facile to survey, the please is worthy, also you’re an conversant writer nay cherish most of the blogs I pinpoint whereas tally on this susceptible.streaming movies

DeathwisH said...

Gerçekten çok güzel bir paylaşım teşekkürler admin.
erotik hikayeler,
izmir escort,
seks hikayeleri,
bursa masaj salonu
izmir masaj salonu,
bursa escort
bodrum escort
islami sohbet
islami forum
mobil sohbet
mobil chat
iptv
iptv server
emeğinize sağlık. :*

sohbet said...

sohbet
mobil sohbet

Unknown said...




almanya sohbet

hollanda sohbet

sohbet siteleri

turk chat

gabile sohbet

mobil sohbet

mynet sohbet

Ferhat said...
This comment has been removed by the author.
Unknown said...

mobil sohbet

cinsel sohbet

Sohbet said...

Teen Porn Sohbet ve chat yapmanızı kolay ve güvenli hale getiren sohbet odaları, sorunsuz ve kesintisiz mobil sohbet siteleri ile arkadaşlık ve yeni kişilerle tanışma imkanı.

Porn android uyumlu dokunmatik ekran akıllı cep telefonları, tablet, ipad, iphone gibi Mobil cihazlarla tek bir tıkla Mobil Sohbet’e katılabilırsıniz.

kerizim said...

2001 yılından günümüze 2023 yılına da en komik eğlenceli ve eğlendirici içerikleri ile keriz ve arkadaşlık sitemiz tanış kaynaş sizlerle olmaya devam ediyor.