Amazon: Distributed Architecture, Distributed Teams

I love this piece on Amazon from Leading Lean Software Development, by Mary and Tom Poppendieck...

“Amazon.com is all about growth. Back in the late 1990’s, when the company was getting started, it didn’t worry much about scale. So Amazon’s systems grew into a massive front end and a massive back end – an architecture that was considered best practice at the time. But every holiday season, things nearly fell off a cliff. Around the year 2000, the company realized that it had hit a wall. Something different would have to be done.

So Amazon changed its architecture. There is no such thing, really, as a database anymore; there are only services that encapsulate both data and business logic. There is no direct database access from outside a service, and there’s no data sharing among the services. There are hundreds of services, and many application servers that aggregate data from services.

In order for this to work (=scale) services had to be decomposed into small, autonomous building blocks. Much like the Internet, there is no central control to fail and services run locally. They make decisions based solely on local information so they can keep on running no matter what is going on in the overall system. The reason for this distributed architecture is simple – CTO Werner Vogels says that “If you need to do something under high load with failures occurring and you need to reach agreement - you're lost.”

And guess what. Because the architecture was distributed, the organization could also be distributed. Amazon found that each service could have its own autonomous team that does it all – customer interaction, decisions on what to develop, choice of tools, programming, testing, deployment, operations, support. Everything. There are no handovers. Services interact with other services through well documented interfaces, such as an SLA with agreed-upon demand levels.

The size of a service team is no more people than can be fed with 2 pizzas – maybe eight people. Amazon.com has many, many 2-pizzas teams, each completely owning a service, cradle to grave. If an architectural feature is too big for a 2-pizza team, Amazon’s bias is to break the feature into smaller pieces, because effectiveness of the team is as important as architectural consistency. Teams stay together over time – a minimum of two years is expected – and own long term responsibility for everything about its service.

It took several years for Amazon.com to evolve to the new architecture and learn how to make it work. Dependencies still exist, of course, and the company has developed some well honed dependency management tools. Configuration management is a challenge, as is testing. But these challenges are creating opportunities for novel solutions. By-and-large, the low dependency architecture – both technical and organizational – works amazingly well.”

Iterating a Design

Some great quotes from Inside Steve's Brain:

"When you start looking at a problem and think it's really simple, you don't understand how complex the problem really is,....
Once you get into the problem... you see that it's complicated and you come up with all these convoluted solutions. That's where most people stop, and the solutions work for a while. But the really great person will keep going, find the underlying problem, and come up with an elegant solution that works on every level. That's what we wanted to do with the Mac."
Steve Jobs

"It sounds simple and obvious, but often, getting to that level of simplicity requires enormous iteration in design. You have to spend considerable energy understanding the problems that exist and issues people have - even when they find it difficult to articulate those issues and problems themselves."
Jonathan Ive

Project Tax

Beyond the skill of the team, the amount of time to deliver functionality is significantly influenced by a combination of:

• team size,
  
• the process they use,
  
• and technology they base their application on.

For example, it will take a large distributed team using complex technologies (e.g. Portlets, ESB, SOA) far longer than a small team using Rails to deliver the same functionality.

A nice metaphor to express this difference is “Project Tax”. I first heard this concept in a blog post comment by Steve McConnel…

“Some of our clients have used the "tax" metaphor. I've mostly heard it used to refer to overhead required to develop a component within a larger system. I.e., to create even a "hello world" program the tax burden to work with the other programs is 5 staff weeks--after that you can actually start adding functionality.”
Steve McConnel

Project Tax is not necessarily a bad thing. The tax helps pay for governance, standardization, along with potentially predictability and maintainability. But the degree of Project Tax may affect the nature and structure of the team.

Good developers may find it harder to shine on Project with a high Project Tax. Instead of delivering features finding themselves focused jumping through hoops – an intellectual dead end. My instinct tells me that great developers will avoid high tax projects.

“The distinguishing feature of nasty little problems is that you don't learn anything from them. Writing a compiler is interesting because it teaches you what a compiler is. But writing an interface to a buggy piece of software doesn't teach you anything, because the bugs are random. So it's not just fastidiousness that makes good hackers avoid nasty little problems. It's more a question of self-preservation. Working on nasty little problems makes you stupid. Good hackers avoid it for the same reason models avoid cheeseburgers.”
Paul Graham

With a higher tax the overall performance of the team may drop, which in turn reduces the delivery capability, which in turn increases the size of the team, which in turn increases the project tax, and so on. This would certainly explain the awful state of many large projects.

Reek for Ruby

Kevin Rutherford's Reek is a great code analysis tool for Ruby.

It's available as gem:
$ gem install reek

$ reek file_or_directory

More info at http://wiki.github.com/kevinrutherford/reek/

HTML Emails: Design like it's 1996

The guys over at mailchimp have some great tips for coding HTML email newsletters.

http://www.mailchimp.com/articles/how_to_code_html_emails/

Email newsletters can be a design nightmare based on the wide number of programs that may render it (Outlook, Outlook 2007, Outlook 2003, Outlook Express, Thunderbird, Apple Mail, Eudora, Lotus, Gmail, Yahoo!Mail, AOL, AOL Web, Hotmail, MSN, Comcast, Earthlink, and on and on and on).

Basically they summed up "as design your pages like it's 1996".