Future States blog: Swimming analogy to understand risks of overconfidence

Heard Malcolm Gladwell's outstanding presentation today (at The Art of Management conference), re: the overconfidence of experts that led to the 2008 financial crisis, as compared to the mistakes of General Joseph Hooker's overconfidence in the Civil War Battle of Chancellorsville . I was reminded of an outstanding "Swim Season" Future States blog post by Demian Entrekin in his Future States blog. Both Gladwell and Entrekin suggest the very real risks of overconfidence, even by those considered experts in their fields. Here is part of Entrekin's IMHO brilliant swimming analogy (full post at http://it.toolbox.com/blogs/ppmtoday/swim-season-37499):

"Imagine two swimmers.

Both swimmers are trying to get from one island to another.

On the other island there is a clear destination...

Both swimmers jump into the water at the same time.

The first swimmer puts his head down and begins to swim...

The plan is to waste no time.

The plan is to keep swimming until the destination has been achieved...

The second swimmer has a different plan.

The second swimmer will take 100 strokes and then stop.

When the second swimmer stops, he will wait until he sees the destination...

The second swimmer notices that the target has moved.

The first swimmer keeps going hard and strong.

The second swimmer stops again and notices a current has taken him off course.

He makes a small adjustment and continues swimming.

The first swimmer begins to tire.

The second swimmer continues to stop and make adjustments...

The first swimmer finally decides he must stop and take his bearing.

The second swimmer sees the shore.

The first swimmer sees the two islands in the distance.

The second swimmer feels tired but also sees the end.

The first swimmer begins to swim back against the current, a sense of panic setting in.

The second swimmer reaches the shore.

The first swimmer now wonders which island is the destination."

Well-done quality improvement is not punitive; it’s educational: Dr. Brent James, NYTimes 8Nov2009

Seems doctors may have some meaningful insights to share with IT executives and project managers, regarding effective management of a quality improvement initiative. A fascinating NYTimes article by David Leonhardt explains how Dr. Brent James is standardizing certain healthcare procedures, thereby significantly improving certain types of patient outcomes. Even straightforward checklists seem to help a lot. Leonhardt also provides the successful example of

"...the Pronovost checklist. As many as 28,000 people in this country die each year from infections that come from intravenous lines. Several years ago, Peter Pronovost, a Johns Hopkins physician, developed a simple list of five steps that intensive-care doctors should take before inserting an IV line, in order to prevent the introduction of bacteria. The checklist reduced the infection rate to essentially zero at 108 hospitals in Michigan where it was adopted. Pronovost published the results in The New England Journal of Medicine in 2006. But most intensive-care doctors are still not using the checklist. To insert an IV line, they continue to rely on their own judgment."

Also helpful and relevant for IT in my experience, is the statement by Dr. James that

"quality improvement is a process, not an event. In part it works by finding variation and drawing attention to it... And well-done quality improvement is not punitive; it’s educational."

Full article:
If Health Care Is Going to Change, Dr. Brent James's Ideas Will Change It - NYTimes.com

Gmail outage resolved thanks to flexible technical architecture [Official Gmail Blog]

Google's Site Reliability Czar: A clear explanation of today's gmail outage, as quoted below.

IMHO, as painful and widespread as the outage was (a side-effect of "routine upgrades"), urgent and ultimately successful action was taken towards resolution. And of course, the true "secret sauce" that enabled this resolution, is the underlying foundation of the "flexible capacity" which "is one of the advantages of Google's architecture".

Here is the full Official Gmail Blog post:

Official Gmail Blog: More on today's Gmail issue: Tuesday, September 01, 2009 6:59 PM
Posted by Ben Treynor, VP Engineering and Site Reliability Czar

"Gmail's web interface had a widespread outage earlier today, lasting about 100 minutes. We know how many people rely on Gmail for personal and professional communications, and we take it very seriously when there's a problem with the service. Thus, right up front, I'd like to apologize to all of you — today's outage was a Big Deal, and we're treating it as such. We've already thoroughly investigated what happened, and we're currently compiling a list of things we intend to fix or improve as a result of the investigation.

Here's what happened: This morning (Pacific Time) we took a small fraction of Gmail's servers offline to perform routine upgrades. This isn't in itself a problem — we do this all the time, and Gmail's web interface runs in many locations and just sends traffic to other locations when one is offline.

However, as we now know, we had slightly underestimated the load which some recent changes (ironically, some designed to improve service availability) placed on the request routers — servers which direct web queries to the appropriate Gmail server for response. At about 12:30 pm Pacific a few of the request routers became overloaded and in effect told the rest of the system 'stop sending us traffic, we're too slow!'. This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded. As a result, people couldn't access Gmail via the web interface because their requests couldn't be routed to a Gmail server. IMAP/POP access and mail processing continued to work normally because these requests don't use the same routers.

The Gmail engineering team was alerted to the failures within seconds (we take monitoring very seriously). After establishing that the core problem was insufficient available capacity, the team brought a LOT of additional request routers online (flexible capacity is one of the advantages of Google's architecture), distributed the traffic across the request routers, and the Gmail web interface came back online.

What's next: We've turned our full attention to helping ensure this kind of event doesn't happen again. Some of the actions are straightforward and are already done — for example, increasing request router capacity well beyond peak demand to provide headroom. Some of the actions are more subtle — for example, we have concluded that request routers don't have sufficient failure isolation (i.e. if there's a problem in one datacenter, it shouldn't affect servers in another datacenter) and do not degrade gracefully (e.g. if many request routers are overloaded simultaneously, they all should just get slower instead of refusing to accept traffic and shifting their load). We'll be hard at work over the next few weeks implementing these and other Gmail reliability improvements — Gmail remains more than 99.9% available to all users, and we're committed to keeping events like today's notable for their rarity."

MS Exchange2007/WinServer2008 compatibility issues

Seems that waiting for Exchange Server 2010 - rather than upgrading now to Exchange Server 2007 - may make sense for many organizations currently running Exchange Server 2003 [or earlier]. Although this seems somewhat inconvenient, I expect many organizations can deal with this successfully in several ways, given sufficient planning. This sort of constraint reminds me of the situation approximately 4 years ago, when some Microsoft shops wanted to wait until MS SQL Server 2005 became available for purchase, rather than immediately purchasing a SQL Server 2000 license. In fact, the Software Assurance up-front purchase (to purchase SQL Server 2000 with SA that would include upgrade to SQL Server 2005) might have seemed logical at the time for many organizations.... But of course, evaluating the full TCO of a subsequent upgrade to SQL Server 2005 would have involved more than the initial hard cost of purchasing Software Assurance. So perhaps the best decision (then and now) as to whether to invest in Software Assurance depends on the particular upgrade and change management capabilities for the specific organization.

The recent Exchange2007/WinServer2008 issue was explained in the recent Microsoft Exchange Team blog posting explaining that Exchange Server 2007 won't run on upcoming Windows Server 2008 R2. Further insights available from SearchITChannel.com: Dilemma: Exchange Server 2007 won't run on next Windows Server:

"...Customers upgrading to Windows Server 2008 R2, due later this year, will have to upgrade to Exchange Server 2010, which is also due this year. This news, disclosed in the fourth bullet point in a July 17 Microsoft Exchange Team Blog post, means Microsoft partners have to sell Exchange Server 2007 without the promise of support on future OSes and without Windows Server 2008 R2 interoperability with Exchange Server 2007... said Planet Technologies lead infrastructure engineer Mike Crowley: "If people stick with Exchange 2007 on Windows 2008 because 'it works,' I wouldn't expect them to want to upgrade the underlying OS just because a new one is available."

Safire: Who is a 'Straw Man'? vs. Brooks: 'Pilot System'

William Safire has penned an excellent piece regarding the phrase "Straw Man", in his New York Times "On Language" column, On Language - Who is a 'Straw Man'? - NYTimes.com:

"Accepting the Democratic nomination in a huge football stadium way back in the presidential campaign of ’08, Senator Barack Obama displayed his oratorical talent by using one of his favorite tried-and-true devices in argument: “Don’t tell me that Democrats won’t defend this country!”

Who was telling him that? To be sure, his opponents were claiming that a Republican administration would be stronger on defense, but nobody was telling him or the voters that Democrats preferred abject surrender. At the time, reviewing that speech, I noted the rhetorical technique: “By escalating criticism, he knocked down a straw man, the oldest speechifying trick in the book.”..."

Safire explains these fascinating origins of "Straw Man" in the context of political speeches. Those of us involved in technology and other business projects, know that a "Straw Man" can also be used in a business context, typically to refer to the first attempt at documenting a project's initial charter and/or requirements, or even a preliminary prototype of a proposed new or enhanced system's user interface.

As explained nicely on mindtools.com:

"When you begin a project or start looking into a problem, you often have incomplete information to work with. So you can spend time gathering facts and data until you are ready to build a really strong argument or plan, or, you can get going straight away and jump in with a not-so-complete solution, with the intention of finding a much better one, as you learn more and more.

That's the premise behind building a straw man - creating a first draft for criticism and testing, and then using the feedback you receive to develop a final outcome that is rock solid..."

A similar, and very concise definition of "Straw Man" can be found in J. LeRoy Ward's excellent Dictionary of Project Management Terms :

"Working draft copy circulated for comments or suggested changes."

Finally, I would like to suggest a connection between the idea behind a "Straw Man" in business and technology, to the very similar concept in Software Engineering, as originally explained brilliantly in the classic Fred Brooks work: The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (2nd Edition):

"The management question, therefore, is not whether to build a pilot system and throw it away. You will do that [anyway]."

Thus, Brooks suggests that one may wish to "plan to throw one away", since it is quite possible that you will, anyhow. And by building this "pilot system" into the plan as an explicit milestone, communication is simpler, and expectations are more realistic from the start.

So perhaps, although not identical, Safire's "straw man" and Brooks' "pilot system" are both trying to capture the same notion: That we sometimes need to start with something we know is imperfect, in order to be able to proceed from there to a truly excellent result.

Implementing Business Process Management: Five Not-So-Easy Pieces

A nice common-sense summary of 5 factors, one or more of which "almost always" causes "failure to achieve results after attempting to implement a business process management (BPM) initiative":

Implementing Business Process Management: Five Not-So-Easy Pieces

My favorite (and the factor which I have noticed most often is the true culprit when these types of process improvement initiatives fail), is the last of the 5 items:

"5. Either shortchanging, or entirely skipping, the all-important task of documenting and analyzing the AS-IS process before designing the TO-BE process. I touched upon this in Reason No. 1, but it deserves its own bullet, because most enterprises have never fully documented their current processes. Typically, many things are done on a daily basis simply because that’s the way they’ve always been done. But without capturing and understanding the “why” for each of these, you are flying blind in determining what your future processes should look like. And take due caution regarding your new path: Often, as we fix one problem, we create another. Even if you’re clear on where you want to go, you need to know where you are to determine the best route."

Who owns your "IT projects"? [Hint: That is a Misnomer] - The Content Economy

Nice grain of salt to keep in mind when planning the next perceived "IT-only" project, e.g. anything related to the unavoidable Upgrade Treadmill that appears not to be a Business Project:

The Content Economy: Who owns your "IT projects"?: "...there are no IT projects, only business projects involving more or less IT. Every IT initiative should be owned and driven by the business. Some might object to this way of seeing things. I would expect IT people that want to keep their influence and power and business people not interested in IT to belong to this category. Upgrading the mail server software to the latest version since the current version will no longer be supported by the vendor might be used as examples of a 'pure' IT project that should be owned by the IT department. But it should not. It would only be a pure IT project if the mail server was not used in the business at all, which then would question its existence and cause it to be terminated. The main point is that someone in the business should always have the ownership of initiatives involving IT, even those needs and initiatives that have been identified and requested by the IT department (such as upgrading a mail server software due to that the current version will no longer be supported by the vendor)...."

Change Management/ITIL/Root Causes - Information Security Magazine

Nice perspective on ITIL, and the ideal goal of proactively identifying root-causes for meaningful followups:

Seven Winners, One Mission - Information Security Magazine:
"'...we shouldn't be doing security for the sake of doing security. We should be doing security because we're running a business,' Riggs says. Service management is a Reuters-wide mandate, one spawned three years ago as a regimen of strict best practices based on the popular U.K. ITIL standard. Riggs is also working to integrate IT security as a global discipline into ITIL best practices. 'As a company we've been banging the drum about customer service, and we're pushing hard to ensure things are done in a systematic, disciplined way to make the customer experience even better,' Riggs says. Reuters' customers measure performance in hundredths or thousandths of seconds; latency is not tolerated. Thus it is dogged work tracking a complex environment of real-time data feeds, historical databases and an infrastructure of 30,000 switches, routers and more than 1,300 firewalls. A standardized service management approach is the only logical means of keeping such complexity reined in, Riggs says. In addition, the company has unified operations and security around incident, problem, configuration, change and release management processes. For example, Reuters' security analysts examine every security incident--whether it caused a disruption or not--to understand a root cause of the management behavior that failed and why a service was not resilient. Finding the root cause allows Riggs' team to apply that information elsewhere and mitigate future events. Modeling exercises, meanwhile, allow them to anticipate problems in the event of future incidents or scheduled network changes, which can number hundreds per week.

"You always expect your infrastructure to come under attack. But if it fails, you have to understand the real underlying root cause. Was it a network design problem, a third-party quality failure, capacity overrun or did it fail because of a configuration problem?" Riggs says. "We want to pinpoint this as well as any aggravating factors and triggers...and then see where we may be exposed elsewhere and fix it before it causes customer pain."

Riggs has tried to instill that uniformity up and down Reuters' supply chain as well.

"I treat them as a virtual extension of my team, and expect them to behave in a certain way." Riggs says. "That's what I expect of the products they deliver...."

----------------
Now playing: Jimmy Buffett - Island
via FoxyTunes