Design patterns (and large-scale web sites)…

Ah… design patterns are wonderful aren’t they?

Singletons, Factories, Commands, Proxies, Visitors, Adapters, etc., etc. Yes, especially in designing and developing software, patterns like the trusty GoF are great. They give you a well-recognized, robust, reliable, reusable, flexible, etc. way to approach problems. Well-recognized because if I name something ObjectFactory, it instantly conveys a lot of structural and behavioral information to the seasoned (even the lightly seasoned) Software Engineer. Robust, reliable, reusable, flexible, etc. because well – these patterns have been put through their paces and are the result of refinement through usage in many systems.

But design patterns are not just useful in software systems. It’s easy to miss the beauty of design patterns in other areas. Consider for example, the network architecture for a large scale web site. At first, it seems like a big secret no one is willing to share. Kind of like an Alliance of Magicians. Not to worry though – there’s a few Gobs out there. So who’s coming to the rescue? Well, we start with … Google! Yes, it takes a while to filter through all the crap to get to some solid secrets-revealing but reading through the good stuff, you will again notice that the design for these sites takes the shape of a few common patterns.

So, here is some of the most interesting stuff available: The LiveJournal Backend Architecture (PDF), the Flickr Architecture, and a great series called the Database War Stories over on Tim O’Reilly’s blog (make sure to click on the links at the bottom to read through all the war stories).

Well, so what are the design patterns for the architecture of a large scale site? First are the front-facing web servers headed by load-balancer. On the other side of these web servers are one or more clusters of databases. Within each of these clusters, simple master-slave replication may be setup, or there may be horizontal or vertical data partitioning setup (which may also be across clusters, depending on the exact setup). There may also be a cluster for serving static content (such as files, HTML, etc.). Cache server (ex. memcached) clusters are also possible. The point is: there are clear patterns, and even solutions to easily implement these patterns. I leave it to you to get the details – they are all good reads (if you’re into that kind of thing).