Gmail (Google Mail) was down from about 12:30pm PDT (8:30pm BST) Tuesday 2nd Sep 2009 to about 2:30pm PDT (10:30pm BST), affecting millions of Gmail customers who depend on the service for everything from fantasy football roster updates to business-critical information. The problem was caused by a classic cascade in which servers became overwhelmed with traffic in rapid succession.
According to Google, the problem began when it took several Gmail servers offline for maintenance, a routine procedure that normally is transparent to users. However, the twist this time around was that Google had made some changes to the routers that direct Gmail traffic to servers in hopes of improving reliability, and those changes backfired.
"As we now know, we had slightly underestimated the load which some recent changes (ironically, some designed to improve service availability) placed on the request routers — servers which direct web queries to the appropriate Gmail server for response," Google explained in a post to its Gmail blog late Tuesday.
"At about 12:30 p.m. Pacific a few of the request routers became overloaded and in effect told the rest of the system 'stop sending us traffic, we're too slow!' This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded," wrote Ben Treynor, vice president of engineering and site reliability czar.
Google fixed the problem by allocating traffic across the rest of its prodigious network, a luxury that it enjoys given the resources it has put in place to operate the world's leading search engine. But what's next?
Google said it would focus on making sure that the request routers have sufficient headroom to handle future spikes in demand, as well as working out a way to make sure that problems in one sector can be isolated without bringing down the entire service. "We'll be hard at work over the next few weeks implementing these and other Gmail reliability improvements — remains more than 99.9 percent available to all users, and we're committed to keeping events like today's notable for their rarity," Treynor wrote.
Several Google Apps customers who use Gmail for internal email at their businesses and organisations did not return calls on Tuesday seeking information on the degree to which they were affected, making it difficult to know the magnitude of the failure. However, Google has put an awful lot of time and money this year behind promoting Gmail as a back-end email software alternative to products from Microsoft and IBM, and embarrassments like this will not help it sell the service to other organisations.
"We know how many people rely on Gmail for personal and professional communications, and we take it very seriously when there's a problem with the service," Treynor wrote. "Thus, right up front, I'd like to apologize to all of you — today's outage was a Big Deal, and we're treating it as such."
This article was originally posted on CNET News.
Posts
