SOX Compliance and the Waterfall Method

So here I am, working away at my job, coding in a vacuum as always, a development team one. Some things never change. But other things, inevitably, do.

Our company was recently purchased. The new company has grand plans to eventually go public. With its eyes set on that prize, they have hired a consulting firm to help them achieve SOX compliance. This firm (who shall remain nameless) is busily churning out reams of process drafts to help us in that endeavor and submitting them to us for approval. It was only a matter of time before the SDLC for software development arrived on my desk for review.

Now, for those not familiar with how things run at our company, I'll simply refer you to this post, which rather succinctly sums it up. While some things have changed, most things, by and large, remain status quo. I have managed to convince them of the value of hiring temp testers prior to releasing builds, so we've shown moderate improvement there. But I'm still wearing tons of hats, and completely driving the entire development process single-handedly. And the company adamantly refuses to hire any other developers to help out. It's also worth noting that since the acquisition, the number of new software projects piling up on my to-do list is rapidly approaching the double-digits. So any process that these guys throw at me is going to affect me and it's going to affect me pretty damned profoundly.

You can imagine my utter shock and amazement when the plan that was presented to me for review and acceptance clearly stated that we were to implement, in excruciating detail, the Waterfall Method.

This presented several problems to me right off the bat:

1. Whoever presented this plan is clearly unaware of the fact that the waterfall method clearly doesn't work. The very man who initially described it (Winston Royce), pointed out quite clearly that it doesn't work, and suggested an iterative model as a clearly superior alternative. See the actual article for proof.
2. There is no way that we'd be able to implement that process with a development staff of one person. The process outlined requires that the roles are separately defined and filled by distinct individuals. We don't have individuals to fill those separate roles, and the company refuses to hire them.
3. Even if we did implement the process, the timeline to implement software solutions for our customers would become so bloated that the customers would drop us like a rock. Our biggest customer demands a release every three months. If we adopted the Waterfall Model as it's spelled out in the SOX compliant process they submitted, it would take three months just to spec out the iteration. Not that the model would permit the iteration.

Consider this quote, dated in 2004, for crying out loud:

Asked for the chief reasons project success rates have improved, Standish Chairman Jim Johnson says, “The primary reason is the projects have gotten a lot smaller. Doing projects with iterative processing as opposed to the waterfall method, which called for all project requirements to be defined up front, is a major step forward.”

In his blog entry, Waterfall Method: A Colossal Blunder, Jeff Sutherland points out the following interesting tidbits in his comments:

The Waterfall process is a "colossal" blunder because it has cost 100s of billions of dollars of failed projects in the U.S. alone. Capers Jones noted 63% failure rates in projects over 1M lines of code in 1993. By the late 1990's, military analysts were documenting a 75% failure rate on billions of dollars worth of projects. In the U.K. the failure rate was 87%.

...

Let me reiterate, for projects over $3M-$5M, the Waterfall has an 85% failure rate. For those projects that are successful, an average of 65% of the software is never used. The Waterfall is a collosal blunder. The most successful Waterfall company I have worked with had a 100% Waterfall project success rate with on time, on features, and on budget. This led to a 100% failure rate in customer acceptance because the customer's business had changed or because the customer did not understand the requirements.

In his article, Improve Your Odds of Project Success hosted to SAP NetWeaver Magazine, David Bromlow provides the following chart that shows how the Waterfall Method makes it difficult to start effectively managing project risk until much later in the project compared to more agile methodologies:

In their article, From Waterfall to Evolutionary Development (EVO), Trand Johnson and Tom Gilb had this to say:

After a few years with the Waterfall model, we experienced aspects of the model that we didn’t like:

• Risk mitigation was postponed until late stages;
• Document-based verification was postponed until late stages;
• Attempts to stipulate unstable requirements too early: change of requirements is perceived as a bad thing in waterfall;
• Operational problems discovered too late in the process (acceptance testing);
• Lengthy modification cycles, and much rework;
• Most importantly, the requirements were nearly entirely focused on functionality, not on quality attributes.

Others have reported similar experiences:

• In a study of failure factors in 1027 IT projects in the UK, scope management related to Waterfall practices was cited to be the largest problems in 82% of the projects. Only approximately 13% of the projects surveyed didn’t fail (Taylor 2000);
• A large project study, Chaos 2000 by The Standish Group showed that 45% of requirements in early specifications were never used (Johnson 2002).

Finally, I'll offer this, from the article Proof Positive by Scott Ambler in Dr. Dobb's Journal:

Agility’s been around long enough now that a significant amount of proof is emerging. Craig Larman, in his new book Agile and Iterative Development: A Manager’s Guide (Addison-Wesley, 2003), summarizes a vast array of writings pertaining to both iterative and incremental (I&I) development, two of agility’s most crucial tenets, noting the positive I&I experiences of software thought leaders (including Harlan Mills, Barry Boehm, Tom Gilb, Tom DeMarco, Ed Yourdon, Fred Brooks and James Martin). More importantly, he discusses extensive studies that examine the success factors of software development. For example, he quotes a 2003 study conducted by Allen MacCormack and colleagues, to be published in IEEE Software, which looked at a collection of project teams of a median size of nine developers and 14 months’ duration. Seventy-five percent of the project teams took an iterative and incremental approach, and 25 percent used the waterfall method. The study found that releasing an iteration’s result earlier in the lifecycle seems to contribute to a lower defect rate and higher productivity, and also revealed a weak relationship between the completeness of a detailed design specification and a lower defect rate. Larman also cites a 2003 Australian study of agile methods, in which 88 percent of organizations found improved productivity, 84 percent experienced improved quality, 46 percent had no change to the cost of development, and 49 percent lowered costs. He also cites evidence that serial approaches to development, larger projects and longer release cycles lead to a greater incidence of project failure. A 2001 British study of 1,027 projects, for example, revealed that scope management related to waterfall practices, including detailed design up-front, was the single largest factor contributing to failure, cited by 82 percent of project teams.

So, with all this overwhelming information at our disposal (which is just the little bit I could scrape up with Google in about an hour), and years of historical evidence that proves empirically that Waterfall doesn't work, why on earth would you wield impose it as the one and only process to be used for all projects, regardless of size or complexity across your entire organization?

It's like voluntarily picking up a cursed +4 Vorpal Sword of Mighty Cleaving: it chops your own head off the moment you touch it.

It's sheer, absolute lunacy. Particularly in our case, where we lack the time, the resources, or the desire to acquire the resources to properly implement it as it's written. We'll be bogged down in a bureaucratic quagmire of Dagoban proportions.

You'll have to forgive me if this seems like a rant. But that's exactly what it is.

It might be time to brush off that resume. Sometimes, enough lunacy just piles up that you start to realize that there's no one behind the wheel who has any firing synapses in the brain.

Technorati Tags: SOX, Waterfall, Waterfall Method, Best Practices, Process, Software Development