Knowing How is Not Enough

There’s an old adage that I heard once, and it’s stuck with me through the years:

He who knows how to do a thing is a good employee. He who knows why is his boss.

I’m also fond of this one:

If you can’t explain it, you don’t understand it.

So I’ve been ramping up on some technology that I’ve really not had an opportunity to really use before, and I’m very excited about it. To make sure I understand it, I’ve decided to go back to the MSDN examples, reproduce them one line at a time, and then document the source code as I understand it. It’s a great way to learn, and sheds a great deal of light on what you think is happening, versus what’s actually happening.

To be perfectly honest, the technology is AJAX. Over the last few years, I’ve predominantly worked for companies that haven’t had any use for Web services, so there’s been no compelling need for it. I’m starting a new job soon and it will rely heavily on Web services, and I really want to make sure I understand them well before I set my foot in the door. It has never been enough for me to know that you just drag a control onto a form or page, set a few properties and press F5. To me, that degree of abstraction is a double-edged sword.

When abstraction reaches the level that it has with Microsoft AJAX, you start to run into some fairly significant issues when it comes time to test and debug the application. The MS AJAX framework is no small accomplishment, and it hides a lot of complexity from you. It makes it so easy to write AJAX applications that you really don’t need to understand the underlying fundamentals of Asynchronous Javascript and XML that make the whole thing work. Consequently, when things go wrong, you could very well be left scratching your head, without a clue, and no idea where to begin looking.

Where, in all of this enormously layered abstraction did something go wrong? Was it my code? Was it the compiler? Was it IIS? Was it permissions? Was it an update? Was it a configuration setting? Was it a misunderstanding of the protocol? Did the Web service go down or move? Was the proxy even generated? If it was, was it generated correctly? Do I even know what a proxy is and why I need it?!

When I started learning about AJAX, we coded simple calls against pages that could return anything to you in an HTTP request using the XMLHTTPRequest object. Sure, it was supposed to be XML, but that was by convention only. The stuff I wrote back then (and I only wrote this stuff on extremely rare occasions, thank the gods), returned the smallest piece of data possible: a single field of data in flat text. It was enough to satisfy the business need, and didn’t require XML DOM parsing.

But even with DOM parsing, the code to make a request and get its data back via XMLHTTPRequest was a lot smaller than all the scaffolding you have to erect now. You might argue that you don’t have create a lot of code now, but that is just an illusion. You’re not writing it, but Microsoft is. Just because you don’t see it doesn’t mean it’s not there. Do you know what that code is doing?

In theory, the Why of Microsoft AJAX, or any AJAX library is to make our lives easier when it comes time to write dynamic Web applications that behave more like desktop applications. To a certain degree, they have. When they work. But when they don’t, I wonder if the enormous degree of abstraction they’ve introduced hasn’t dumbed us down to the point where we’ve ignored essential knowledge that we should have.

If you’re going to write Web services, or consume them, you should, at a minimum, understand what they are, and how they work. You should understand their history, how they evolved, and the problem that AJAX tries to solve. It’s not enough to know how to write a Web service, you have to know why you’re doing it, and why you’re doing it the way you are. That sort of knowledge can be crucial in making the right choices about algorithms, protocols, frameworks, caching, security, and so on.

But this could be true of any technology or practice we learn. AJAX, LINQ, design patterns, TDD, continuous integration, pair programming, and so on. Know why.

Try this simple litmus test. Explain something you think you know to one of your peers. If you can’t explain it clearly without having to pull out a reference or go online, you don’t understand it the way you think you did. Consider relearning it. It’ll only improve your value to yourself, your peers, and your employer.

Technorati Tags: Productivity,Software Development,Technology

The Sad State of SQL Refactoring

I love refactoring tools.

The ability to select a variable in code, right-click on it, choose the Rename command from a context menu, and then safely rename that variable in every location in which it appears is a boon to my programming productivity. Similarly, the ability to take a large block of complex logic and extract it to its own method is really handy. Or, the ability to reorder parameters, and know that every piece of code that calls that method will be updated appropriately saves me tons of time and improves my confidence in the quality of the code.

When you refactor code of any kind, you enhance its readability without changing the way it works. But refactoring by hand is frequently difficult, tedious, and error-prone. That's why refactoring tools exist and why the essential service they provide is important. They allow us to improve the maintainability of code, hopefully reducing maintenance costs, quickly and efficiently.

Enter SQL languages.

In theory, there is a standard for SQL languages: the ANSI SQL-92 standard. One would like to think that it would be a simple matter to create refactoring tools for any SQL based on the ANSI standard for SQL. One would be wrong. You can't use the standard as the sole basis of a refactoring tool.

Any given database vender wants to strive to make their product unique, to stand out from the crowd. And so, they don't entirely conform to the standard. They have additional features that separate them from each other. For example: Oracle organizes functions and procedures into packages. Microsoft does not. Microsoft allows a bit data type on columns. Oracle does not. And, although we all loathe to think of Access as a real database, Access provides a boolean data type that you can use in columns, while Oracle does not.

Now, let's also talk about legacy support. Oracle has been around for a very long time. Version 2 came out in 1979, from what I gather. Their legacy join syntax looks nothing like the ANSI standard (and that's not a criticism, just a simple statement of fact):

SELECT Customers.CustomerId, Company, OrderID
FROM Customers C, Orders O
WHERE C.CustomerID (+) = O.CustomerID
UNION
SELECT Customers.CustomerId, Company, OrderID
FROM Customers C, Orders O
WHERE C.CustomerID = O.CustomerID (+)

In ANSI SQL, this is :

SELECT Customers.CustomerId, Company, OrderId
FROM Customers C FULL OUTER JOIN Orders O ON
(C.CustomerId = O.CustomerId)

Now, we could add in all the different SQL variations that we know that are out there:

• ANSI standard
• Interbase/Firebird
• IBM  SQL
• Microsoft Transact SQL
• MySQL
• Oracle PL/SQL
• PostgreSQL
• Access
• FoxPro

...and on and on and on, but you get the point. There are a lot of SQL variants out there. And their goal is to accomplish the same thing:

• Create a data store.
• Occasionally, modify the structure of the data store.
• Get data out of a data store.
• Put data into a data store.
• If supported, execute code within the confines of the data store and (optionally) return a result.

Behold, SQL in a nutshell.

Over the lifetime of these various products, they have added features that they have to maintain for legacy support. Somewhere out there there's a business that absolutely must have that feature in place or their whole process will come crashing down. Don't you dare remove it. It doesn't matter that there are better features (likely based on a standard); there are applications out there for which they don't have the source code anymore, or which no one understands, and they're too afraid to touch.

Now, all these reasons have led us to a scenario where we have vastly different implementations of SQL. Sure, they share a lot in common, but they also have radically different feature sets and syntax. And that situation, in and of itself, has led us to one frightening and sad conclusion:

We will likely never have a tool that is able to connect to any database and be able to correctly refactor its SQL.

And that's just a damned shame. Because there's a lot of SQL out there. Stored procedures, functions, views, triggers, even inline SQL in applications and all that other jazz. But the amount of work it would take to get us to a point where a refactoring tool could recognize any variant of SQL and correctly parse it, refactor it, and not hose the code is enormous.

It's a pity, really. I could see tons of use for a tool like this. In my own office, we work with SQL Server, Oracle, and Atomix databases. What I wouldn't give for a tool that could refactor SQL to enhance its readability without changing the way it worked.

And who knows? Down the road, we may be working with something else.

But this is the world we live in. If we want to refactor SQL, it looks like we'll have to settle for separate refactoring tools for each language. And then each will come with its own quirks. That might be good or bad, but it's likely the best we can hope for for now.

 

Technorati Tags: SQL, Best Practices, Productivity, Code Quality, Refactoring, Software Development

The Absurdity of "Don't Reinvent the Wheel"

As developers, we've had this adage drilled into us from the beginning: Don't reinvent the wheel. In short, don't rewrite what's already been written. The idea is sound, in theory. You can save yourself time and money if you'll simply reuse existing code and/or components rather than writing them yourself from scratch. This time and money is saved up front when you write it (or would have written it), and down the road, when you have to maintain your system.

However, I'd like to point out another, equally applicable adage: There's nothing new under the sun. Anyone who's ever tried to write a novel, a short story, a play, a movie, a song, or a piece of software, will know this one simple truth: somewhere, at some point in time, it's already been written.

Every algorithm, every piece of code that we will ever attempt to write has already been written somewhere, at some point in time, by someone. Only the names have been changed. You're not inventing anything that is completely new, that's never been seen before. You should wisely disabuse yourself of that notion as quickly as possible.

In the grand scheme of things, at the application level, you may very well have an idea for a system that is unlike anything that has been done to date. But the algorithms that drive it have already been written. Bubble sorts, hashes, exception handlers, encryption, data access, socket management, shopping carts, entire application frameworks, date management, document management, serialization, port I/O, and all that other stuff has already been done. Further, it's already been done several times over in many different languages to varying degrees of success.

Tragically, if you're using a large application framework, like Java's EE or Microsoft's .NET, the chances are good that the functionality you're looking for is built right into the framework itself. The problem is that the framework is so vast that you'll spend more time looking for it, and determining whether or not it works the way you need it to work than you would just rewriting it yourself.

Application frameworks are stunningly afflicted with feature creep. They must do everything under the sun, must meet every possible need. The problem, then, is that their scope becomes so broad, so vast, that no one in their right mind could possibly grasp the totality of all that they can do. It is inevitable that anyone using them will reinvent some of their functionality. The scale of that functionality might be small (reformatting dates) or it might be substantial (pooled database connections).

In the end, it's absurd to think that we can possibly avoid reinventing the wheel. Of course we're going to reinvent it. Every application we write is a reinvention of someone else's wheel. It just so happens that our wheel is a custom wheel. All this paranoia about reinventing the wheel is blown out of proportion. A proper buy vs. build decision should never be neglected; but don't ever think for one minute that what you're creating hasn't been created before.

Consider the scenario where you're under the gun to get a product out the door. And I mean it's a really tight schedule. And don't act like it's a perfect world, and you have leverage over the schedule. This is reality here. In the real world, the customer controls the schedule, because it's tied to when the product is released, and that's tied to this big, huge monstrosity in another state or another country. The product's delivery schedule is a train barreling down the track at 120mph and no one short of God can stop it. Now, you have a very finite amount of time to work in. You need an algorithm. You know you could write it. Or you could look to see if someone else has written it.

If you do the whole Web search thing, you have to ask yourself a few questions: Is it from a source you trust? Is it in the language you're using, or do you have to convert it? Does it work? Does it need to be tweaked? If any of these fail, you're back to the drawing board. Time's wasting here, and that train's getting closer to its destination. If all the answers pass, you have to make sure you don't run into any copyright or licensing issues with that code. (You are paying attention to that, aren't you?)

If you decide to peruse your application's framework, you'd better hope it's well documented, and very easy to search. Good luck using the search features in .NET. It's not like the ASK.COM interface, where you can ask, "How do I convert a date in DOD format to Gregorian format?" Yeah. Good luck with that. On the other hand, you could ask your coworkers. They might know. Then again, they might not. If they don't, you're off to Google to get the information. Here's hoping you get a timely and accurate response.

Sure, this is an extreme example. But it makes my point: At some point, the work has to get done. You can't afford to spend days or weeks scratching your head about whether or not that wheel's already been invented. Believe me, it has. The problem is, there are a countless number of wheels, and none of them are labeled, and you don't know where to find the wheel you're looking for.

Stop wasting time, and invent your own damned wheel.

After all, whatever code you might reuse, is just someone else's reinvention of the same wheel.

SOX Compliance and the Waterfall Method

So here I am, working away at my job, coding in a vacuum as always, a development team one. Some things never change. But other things, inevitably, do.

Our company was recently purchased. The new company has grand plans to eventually go public. With its eyes set on that prize, they have hired a consulting firm to help them achieve SOX compliance. This firm (who shall remain nameless) is busily churning out reams of process drafts to help us in that endeavor and submitting them to us for approval. It was only a matter of time before the SDLC for software development arrived on my desk for review.

Now, for those not familiar with how things run at our company, I'll simply refer you to this post, which rather succinctly sums it up. While some things have changed, most things, by and large, remain status quo. I have managed to convince them of the value of hiring temp testers prior to releasing builds, so we've shown moderate improvement there. But I'm still wearing tons of hats, and completely driving the entire development process single-handedly. And the company adamantly refuses to hire any other developers to help out. It's also worth noting that since the acquisition, the number of new software projects piling up on my to-do list is rapidly approaching the double-digits. So any process that these guys throw at me is going to affect me and it's going to affect me pretty damned profoundly.

You can imagine my utter shock and amazement when the plan that was presented to me for review and acceptance clearly stated that we were to implement, in excruciating detail, the Waterfall Method.

This presented several problems to me right off the bat:

1. Whoever presented this plan is clearly unaware of the fact that the waterfall method clearly doesn't work. The very man who initially described it (Winston Royce), pointed out quite clearly that it doesn't work, and suggested an iterative model as a clearly superior alternative. See the actual article for proof.
2. There is no way that we'd be able to implement that process with a development staff of one person. The process outlined requires that the roles are separately defined and filled by distinct individuals. We don't have individuals to fill those separate roles, and the company refuses to hire them.
3. Even if we did implement the process, the timeline to implement software solutions for our customers would become so bloated that the customers would drop us like a rock. Our biggest customer demands a release every three months. If we adopted the Waterfall Model as it's spelled out in the SOX compliant process they submitted, it would take three months just to spec out the iteration. Not that the model would permit the iteration.

Consider this quote, dated in 2004, for crying out loud:

Asked for the chief reasons project success rates have improved, Standish Chairman Jim Johnson says, “The primary reason is the projects have gotten a lot smaller. Doing projects with iterative processing as opposed to the waterfall method, which called for all project requirements to be defined up front, is a major step forward.”

In his blog entry, Waterfall Method: A Colossal Blunder, Jeff Sutherland points out the following interesting tidbits in his comments:

The Waterfall process is a "colossal" blunder because it has cost 100s of billions of dollars of failed projects in the U.S. alone. Capers Jones noted 63% failure rates in projects over 1M lines of code in 1993. By the late 1990's, military analysts were documenting a 75% failure rate on billions of dollars worth of projects. In the U.K. the failure rate was 87%.

...

Let me reiterate, for projects over $3M-$5M, the Waterfall has an 85% failure rate. For those projects that are successful, an average of 65% of the software is never used. The Waterfall is a collosal blunder. The most successful Waterfall company I have worked with had a 100% Waterfall project success rate with on time, on features, and on budget. This led to a 100% failure rate in customer acceptance because the customer's business had changed or because the customer did not understand the requirements.

In his article, Improve Your Odds of Project Success hosted to SAP NetWeaver Magazine, David Bromlow provides the following chart that shows how the Waterfall Method makes it difficult to start effectively managing project risk until much later in the project compared to more agile methodologies:

In their article, From Waterfall to Evolutionary Development (EVO), Trand Johnson and Tom Gilb had this to say:

After a few years with the Waterfall model, we experienced aspects of the model that we didn’t like:

• Risk mitigation was postponed until late stages;
• Document-based verification was postponed until late stages;
• Attempts to stipulate unstable requirements too early: change of requirements is perceived as a bad thing in waterfall;
• Operational problems discovered too late in the process (acceptance testing);
• Lengthy modification cycles, and much rework;
• Most importantly, the requirements were nearly entirely focused on functionality, not on quality attributes.

Others have reported similar experiences:

• In a study of failure factors in 1027 IT projects in the UK, scope management related to Waterfall practices was cited to be the largest problems in 82% of the projects. Only approximately 13% of the projects surveyed didn’t fail (Taylor 2000);
• A large project study, Chaos 2000 by The Standish Group showed that 45% of requirements in early specifications were never used (Johnson 2002).

Finally, I'll offer this, from the article Proof Positive by Scott Ambler in Dr. Dobb's Journal:

Agility’s been around long enough now that a significant amount of proof is emerging. Craig Larman, in his new book Agile and Iterative Development: A Manager’s Guide (Addison-Wesley, 2003), summarizes a vast array of writings pertaining to both iterative and incremental (I&I) development, two of agility’s most crucial tenets, noting the positive I&I experiences of software thought leaders (including Harlan Mills, Barry Boehm, Tom Gilb, Tom DeMarco, Ed Yourdon, Fred Brooks and James Martin). More importantly, he discusses extensive studies that examine the success factors of software development. For example, he quotes a 2003 study conducted by Allen MacCormack and colleagues, to be published in IEEE Software, which looked at a collection of project teams of a median size of nine developers and 14 months’ duration. Seventy-five percent of the project teams took an iterative and incremental approach, and 25 percent used the waterfall method. The study found that releasing an iteration’s result earlier in the lifecycle seems to contribute to a lower defect rate and higher productivity, and also revealed a weak relationship between the completeness of a detailed design specification and a lower defect rate. Larman also cites a 2003 Australian study of agile methods, in which 88 percent of organizations found improved productivity, 84 percent experienced improved quality, 46 percent had no change to the cost of development, and 49 percent lowered costs. He also cites evidence that serial approaches to development, larger projects and longer release cycles lead to a greater incidence of project failure. A 2001 British study of 1,027 projects, for example, revealed that scope management related to waterfall practices, including detailed design up-front, was the single largest factor contributing to failure, cited by 82 percent of project teams.

So, with all this overwhelming information at our disposal (which is just the little bit I could scrape up with Google in about an hour), and years of historical evidence that proves empirically that Waterfall doesn't work, why on earth would you wield impose it as the one and only process to be used for all projects, regardless of size or complexity across your entire organization?

It's like voluntarily picking up a cursed +4 Vorpal Sword of Mighty Cleaving: it chops your own head off the moment you touch it.

It's sheer, absolute lunacy. Particularly in our case, where we lack the time, the resources, or the desire to acquire the resources to properly implement it as it's written. We'll be bogged down in a bureaucratic quagmire of Dagoban proportions.

You'll have to forgive me if this seems like a rant. But that's exactly what it is.

It might be time to brush off that resume. Sometimes, enough lunacy just piles up that you start to realize that there's no one behind the wheel who has any firing synapses in the brain.

Technorati Tags: SOX, Waterfall, Waterfall Method, Best Practices, Process, Software Development

Software Release Engineering

In his Coding Horror post titled How Should We Teach Computer Science?, Jeff Atwood blogs about the lack of coverage of release engineering in computer science courses. At best, he points out, it's given cursory coverage in these courses.

Now, I'm a self-taught developer. I started programming computers in 1985 or so, and I've taught myself everything I know. So I can't really comment about what the courses in a college or university are like. But I can say this: experience has taught me that a few things he says are absolutely, undeniably true. So in this article, I'm going to enumerate those things I think are really important, and how I built a software release process at the company I worked for.

The Ugly Truth About Release Engineering

1. Release Engineering is not simply deploying your product. There's a reason it's called engineering. It involves getting the latest version of the build from source code control, building the software, executing unit tests, building installers, and labeling the build if it's correctly built. It may involve pinning or branching the build. It requires a daily "good code" check-in time policy. It requires daily builds to ensure that you have software the compiles every day, and a means of notifying folks when the build is broken, and fixing the build-breaking code right away. It's NOT simple.
2. Consistent, disciplined use of source-code control is the bedrock of release engineering. At any given time, you might need to fix bug in the prior release. That's hard to do if you've already started changing the code for the new release. Branched builds allow you to do that. Also, versioned files in the repository allow you to view the history of changes to a file to recover from unintentional changes. You can also develop for multiple platforms while using many of the same files, sharing them across projects without having to worry if they're out of synch. Labels on files and projects tell you exactly which version of a file was used to create any given build so that you can recreate a project from the repository if you need to.
3. Building for your environment is not enough. You need a test environment that mimics your client's environment as closely as you can make it, down to the OS, the browser, the applications and the add-ons. Just because it runs on your machine when you press F5 from Visual Studio does not mean it's going to run on the client's machine. If you're developing for multiple browsers, install those browsers and test for them.
   (Ugly true story: our company accidentally allowed IE7 through the group policies. We had IE7 deployed everywhere. Our clients don't plan to upgrade to IE7 for another year, at least. Our product must run on IE6. I had to create a separate machine that was safe from IE7 downloads and strictly ran IE6 to be certain the product ran correctly.)
4. F5 is not enough. Every build should be a clean build. Every build. Don't ship files to the customer that aren't required to run the software. Create a build script that does the job. Excluding a file from a Visual Studio project doesn't delete the file from the folder, but does leave it in source code control (a good thing for versioning). To ensure  clean release, have your release script remove files you aren't using prior to shipment.
5. You need a checkin-time policy. All good source code must be checked in by a certain time every day. Code that isn't checked in does not make it into the daily build. This check-in time should be early enough that the release manager can start the daily build (if it's a manual process), or make the rounds and make sure that all code is checked in prior to it. I favor end-of day check-ins (around 4 PM) for nightly builds, but each organization is different.
6. The software must successfully build every day. A successful build is a good sign of project health. An automated build tool can be set up to execute the build in the off hours after everyone has gone home, and after all files are checked in. Once the build is complete, the pass/fail report is sent to your release manager. However, just because it compiles doesn't mean that it's entirely healthy or bug-free. Therefore...
7. Automated unit tests should be executed on every build. If you aren't using automated unit tests, you should be. They're not hard to learn, the tools to create them are freely available and they can improve the stability and quality of your code immeasurably. Incorporate the unit tests into your build script so that they're executed every time you build the software. Correctly written unit tests alert you to build-breaking defects quickly and immediately.
8. Build-breaking defects must be resolved before anything else. This includes any defect that causes the software to fail to compile or any defect that causes a unit test to fail. The team must adopt a "drop everything and fix the build" mentality. In my own personal experience, this view is not easily accepted in the early stages of a project, but during the later stages, when there's typically a "crunch" mode, and the build isn't riddled with build-breaking defects, developers are thankful that those defects simply aren't there.
9. You need a release manager. While you might have many people who contribute to the build, checking in changes and adding new content, you need one person whose primary responsibility is to ensure that the software builds properly every day. That individual is also responsible for your installer, and for identifying the code that breaks the build and ensuring that it gets resolved. The release manager doesn't resolve the defects himself unless he checked in the build-breaking defect (since he doesn't know anything about the defect); rather, he must play the role of the hard-nosed drill sergeant ensuring that the coder who checked it in drops everything to fix the build right now. If you can't build the product, you can't ship the product, and anything else that developer might be working on is a moot point. It's an ugly, painful job, but it's crucial.
10. You need a dedicated build server. This machine is clean, and does nothing but build your software. This guarantees that it injects no artifacts into your final product. It runs the daily build, executes the unit tests, and sends out the notifications when the build passes or fails. It might also house archived copies of each build's source code and binaries. It must be on the network, and should be backed up regularly. The Release Manager should have access to it, but no one else on the development team.

My Own Personal Release Process

It bears noting here that I've done the release process for two different companies. At one, it was for a full team of developers (about twenty of them), and the release process there was a nightmare. At that time, we couldn't get a release out in a month if we tried. So I volunteered to take on the job, and redesigned the process. It took about a week to get the process reengineered and everyone on board, but after two weeks we had daily builds working and everything was going much more smoothly.

I took many of those same principles and applied them to my new job. Clearly, some of them don't apply in a single-developer shop. But the basic principles are the same.

Source Code Control

1. All developers must use source code control.
2. All working, compilable code that does not break the build must be checked in by 3 PM every day. Code that is not checked in at this time does not make it into the daily build.
3. User names and passwords are required for accessing source code control.
4. The admin password is written on a piece of paper, sealed in an envelope, and stored in the CIO's desk. No one else has it.
5. Minimal rights to access the repository based on need are granted.
6. The main tree has the following subprojects: Build, Dev. Each tree's subprojects are mirrors of each other. Build is where the branched and pinned copies of the successful builds are. Mainline development takes place under the Dev tree.
7. Every file that is required to create or ship the project is included in source code control: source files, SQL scripts, Web pages, images, build scripts, unit tests, test plans, requirements documentation, etc.
8. Because we use SourceSafe, every weekend, during off-peak hours, regularly scheduled maintenance is performed on the repository to keep it in tip-top shape.
9. The repository is stored on on the network. This folder is backed up incrementally nightly, and fully weekly.

Build Process

1. Every afternoon, at 3 PM, all developers must have code they want included in the build checked into the repository.
2. The release manager does a final verification at 3:15 to ensure that all code is checked in.
3. An automated script fires off the build at 3:30 PM. It does the following:
   1. Clean the build folders on the build server. This involves deleting all files and folders from the project's build folder, ensuring a clean build.
   2. Get the latest version of the software from the DEV tree in the repository.
   3. Compiles the software and all of its dependencies. If the compilation fails, an email with high importance is sent to the release manager, notifying him of the failure, and the script aborts.
   4. Executes the unit tests. The unit test results are output to a text log file which are then sent to the release manager in an email.
   5. Executes a cleanup batch file that ensures that any files that should not be shipped with the product are removed.
   6. Creates the installer or archives the build into a ZIP file.
   7. Labels the build in the repository.
   8. If the build was successful, sends a "Build success" message to the release manager.
4. Note that step 3 may execute multiple times depending on whether you are targeting multiple platforms or releases (such as Debug and Release, or various browsers, or various OSes).
5. Upon receipt of a build failure email, the Release Manager reviews its contents, and identifies the offending source code. He then determines who checked that code in, and contacts that developer and asks them to resolve the defect as soon as possible.

   Important: Except in the direst of circumstances, the release manager should not attempt to fix someone else's defects. He should ask the developer to fix his own defects. If the release manager takes this task on himself, he'll quickly become inundated trying to fix all the build-breaking defects, and won't have time to do his own work.

6. The developer resolves the build-breaking defect and checks in the change for inclusion in the next daily build.
7. If enough build-breaking defects were present, the Release Manager may choose to manually rebuild the software once defect corrections are checked in.
8. If the build is shipped to the customer, it is labeled, pinned and branched into the BUILD tree in the source code repository.

In closing

We're a Microsoft shop. Although I've worked in Java houses, my limited experiences have largely focused on the Microsoft stack, and the process that I've outlined above is primarily geared for Microsoft Visual Studio and SourceSafe. But the basic principles should be pretty universal. You should be able to take them and apply them to just about any combination of source code repository tools, unit testing tools, and IDE (or text editor).

The primary thing to remember is this: if you can't build it reliably, predictably, and on a moment's notice, you're in trouble. When a development team knows they can't build the software, and when the testing team is sitting around for days or weeks at a time wondering when they're going to get a new release to test, morale suffers, tempers flare, and things rapidly go downhill. I've been there. I've seen it. It ain't pretty.

Every project needs a good, solid release process. I'm tempted to say that any release process is better than no release process, but that wouldn't be entirely true. A release process needs to be trim, make sense, bolster confidence in the project, and help propel the team forward towards success. That's what this process is designed to do.

I'm sure that others have some ideas on how to improve the process above. I'd love to hear those ideas. I'm sure that others have different ways of doing things. I'd love to hear that too. There is no silver bullet, and I'm not anywhere stupid enough to think that this plan is perfect. But I hope it's enough to help someone, somewhere get a little bit closer to a project that gets out the door a bit faster, healthier, and with its developers' sanity in tact.

 

Technorati Tags: Software, Release Engineering, Deployment, SCC, Unit Testing, Computer Science, Release Manager, Software Builds, Compile, Install

Refactor Yourself

Take a moment to stock of where you are now. What skills do you have? How can you improve them? Every day of your career, you should be learning something, improving something, refining something. Your skill set should be undergoing constant refactoring. This can only make you more efficient. If the stuff you're learning isn't making you more efficient, discard it.

At some point, you have to have the guts to go against the grain. Just because a "best practice" works for someone else at some other company doesn't necessarily make it a "best practice" for you and your company. A "proven methodology" isn't necessarily going to be a "proven methodology" for you. Have the guts to challenge the status quo. If it's not making you more efficient, it's likely hindering you. Refactor it out.

If your team doesn't have the funds to learn some new technique, seek that knowledge personally. There is no reason that your company's inability to fund team education should hold you back. Buy books and read. Search the Internet. Read blogs and programming newsgroups. Experiment with code. Ask your peers. Never stop seeking knowledge. Never stop learning.

Take some of your old code, copy it, and then refactor the hell out of it. You'll be surprised what you can learn by simply refactoring code: more efficient ways to implement things that you did before (and will likely do again), better algorithms that work faster, use less resources, and are easier to maintain. Refactoring improves your skill set. Refactoring your own code, on your own time, is a personal competition against yourself to improve your own skill set.

You don't need to compete against anyone else. Coding cowboys, platform fanboys, methodology purists, conspiracy theorists...you shouldn't be worrying about them. You should worry about yourself. Make yourself as good as you can possibly be. Every day, ask yourself this essential question: "How can I improve myself today?" Find that way, and then do it. Set aside a little time every day to refactor your skill set.

Each day is an opportunity to make yourself a little bit better, a little more efficient than you were the day before. With each passing day, you have the opportunity to become smarter, faster, wiser, more valuable. But that means taking care to constantly revise your skill set. Have the wherewithal to discard habits and ideas that simply don't work. If you suspect you're doing something one way simply because you've always done it that way, or because that's the way everyone else does it, question it. If you can't see a tangible benefit to it, refactor it out.

Look, I'm not Gandhi or anything. But I can tell you this: I firmly believe that the key to success in this field is a personal commitment to growth. Don't trust anyone to just hand you knowledge, or to stumble across the skills you'll need. You have to actively reach out and take the skills and knowledge you need to be successful. It's an active task. It's not going to be something you just acquire through osmosis.

We all have to get to a point where we realize that we're not as efficient, not as smart, not as skilled, and nowhere near as good as we could be. There's always someone out there who's better than we are.

Our goal isn't to compete with them. Our goal is to constantly aspire to be better than we are right now, at this very moment.

A Parameter Validation Framework

I need some help here. I'm hoping you'll review the following "framework" and give me your feedback. I need to know (1) if it's a good idea, (2) if the architecture is sound, and (3) how it can be improved. If it's a generally useful idea, I'll open it up on an open source project forum where it can be expanded for general use. Otherwise, I'll go back to the drawing board. (I'm not sure at this point if it's a worthwhile idea.)

The Background

In reviewing my projects, I noticed that I tend to be lazy about validating parameters to methods. I reviewed a number of projects that I was working on and noticed that it tended to be true no matter what the project was, or what the parameter types were. When I sat back and thought about it, and paid attention to myself doing it, it came down to two primary issues that led to me avoiding it:

• I was usually in a rush to get a product out the door. This tends to always be the case, since the product's schedule is out of my control, and there isn't much that I can do about it. I can only negotiate for a minor change in the project's time. Then, I'm left to work with the amount of time I'm given. So I try to do the best work I can in the time I'm allotted, and hope to high heaven that it's good work. That often means that I code like hell (classic software mistake), and hope that nothing breaks.
• Parameter validation code tended to be verbose. When I really wanted to say, "Parameter y should be a positive number," the resulting code looked like this:
  If myParameter <= 0 Then
     Throw New ArgumentOutOfRangeException("myParameter", "Must be a positive number.")
  End If

These two issues alone might not seem like a big deal in and of themselves. But when it comes to encouraging code quality, I take them very seriously. I abhor sloppy code, especially when it's mine. I want to be sure that parameters are valid, and that when a method receives parameters, they are checked rigorously to ensure that I'm getting valid values throughout any system I'm developing.

The Goal

I set out to come up with a way to make it easier to validate parameters. I wanted a way to make it so easy to validate them that I would find it enjoyable to do so. I wanted it to make my code more legible, with a minimal amount of impact on its performance. Knowing that I basically lack discipline, I knew I needed a software solution that would exploit Visual Studio's capabilities to remind me to check for tests that I might not have normally thought of. (We have a tendency to inadequately document the requirements—again, due to time constraints—and I need to be able to think about logical tests that make sense for the parameter as I'm coding them.)

So the goals for the framework are:

• Ease of use. This was the number one design goal. If the thing wasn't easy to use, it was pointless to build it in the first place, since it was being designed to encourage validation of parameters. If the framework made it harder to do that, it was defeating the purpose.
• Understandability. The code should enhance the readability of the code.
• Performance. The methods were going to be called with a high degree of frequency. It was critical that the methods have a very low overhead on the call stack. If at all possible, the number of objects being created should be kept to a minimum. However, because we're adding code, some overhead is unavoidable.
• Reliability. It has to work. And it has to work reliably and predictably every time.
• Extensibility. It has to support other data types.
• Maintainability. The framework's code has to be easy to read, understand, and maintain. We do not want to have to bloat the development time of the projects that rely upon it with weeks of maintenance time just to fix and expand the parameter validation library.

The Solution

I set out to come up with a way to make it easier to validate parameters. I wanted a way to make it so easy to validate them that I would find it enjoyable to do so. I wanted it to make my code more legible, with a minimal amount of impact on its performance. I wrote a set of classes that, I believe, accomplished that. The resulting "framework" (if you can call it that) is based on the Assert.That model popularized by NUnit and JUnit. When you want to prove that a parameter is positive, you write the following:

Sub Foo(ByVal integerParameter As Integer)
   Validate.That(integerParameter, "integerParameter").IsPositive()
   ' Do something interesting with integerParameter
End Sub

It's short, sweet, and to the point. If integerParameter contains any value that is less than 1, an ArgumentOutOfRangeException is thrown, with an appropriately formatted message bearing the parameter's name.

The solution involves the following classes:

Class	Description
Validate	A class factory that instantiates strongly-typed parameter validator objects. These are derived from ParameterValidatorBase. Provides the heavily overloaded That method.
BooleanValidator	Inherits from ParameterValidatorBase. Provides methods for validating Boolean parameters, such as IsTrue and IsFalse.
ConnectionValidator	Inherits from ParameterValidatorBase. Provides methods for validating IDbConnection parmaeters, such as IsNotNull, IsOpen, and IsClosed.
DateValidator	Inherits from ParameterValidatorBase. Provides methods for validating date parameters, such as Equals, IsBetween, IsGreaterThan, IsGreaterThanOrEqualTo, IsLessThan and IsLessThanOrEqualTo.
EnumValidator	Inherits from ParameterValidatorBase. Provides methods for validating enum parameters. This is the only parameter validator that relies on reflection. It's sole method is IsValid.
IntegerArrayValidator	Inherits from ParameterValidatorBase. Provides methods for validating integer array parameters, such as IsNotNull and IsNotEmpty.
IntegerValidator	Inherits from ParameterValidatorBase. Provides methods for validating integer parameters, such as Equals, IsGreaterThan, IsGreaterThanOrEqualTo, IsInRange, IsLessThan, IsLessThanOrEqualTo, IsNegative, IsNotNegative, IsOneOf, IsPositive, and NotEqualTo.
ListValidator	Inherits from ParameterValidatorBase. Provides methods for validating IList parameters, including IsEmpty, IsNotEmpty, IsNotNull, and IsNotNullOrEmpty.
ObjectValidator	Inherits from ParameterValidatorBase. Provides methods for validating any Object parameter, including IsNotNull and Equals.
ParameterValidatorBase	The base class for all validators. Provides the name of the parameter and a protected property to store the parameter's value.
StringArrayValidator	Inherits from ParameterValidatorBase. Provides methods for validating string arrays, including IsNotNull, IsNotEmpty, and IsEmpty.
StringValidator	Inherits from ParameterValidatorBase. Provides methods for validating strings, including Contains (overloaded), Endswith (overloaded), IsNotEmpty, IsNotNull, IsNotNullOrEmpty, and StartsWith (overloaded).
TransactionValidator	Inherits from ParameterValidatorBase. Provides methods for validating IDbTransaction parameters, including HasValidConnection, HasOpenConnection and IsNotNull.

Other parameter validator types are being added as the need for them arises.

When a call is made to Validate.That(), the framework instantiates an appropriately typed parameter validator object. The caller then invokes one or more of the methods on that object to prove that the parameter is valid. The parameter validator merely tests the condition specified by the method name; if the test evaluates to False, an appropriately typed exception (derived from ApplicationException) is thrown. The parameter validator is responsible for properly formatting the message and passing the name of the parameter; this is spiffy because some of the exception objects are inconsistent about the order in which the name and message parameters are passed to them. The framework standardizes the order through the use of overloads: parameter, name, message.

For example, the source code for the ConnectionValidator class looks like this:

Option Explicit On 
Option Strict On

Imports System.Data

Namespace Validation

   Public Class ConnectionValidator
      Inherits ParameterValidatorBase

      Friend Sub New(ByVal connection As IDbConnection, ByVal name As String)
         MyBase.New(connection, name)
      End Sub

      Public Sub IsClosed()
         If Not Connection Is Nothing Then
            If (Connection.State And ConnectionState.Closed) = 0 Then
               Throw New ArgumentException("Operation requires a closed connection.", Name)
            End If
         End If
      End Sub

      Public Sub IsNotNull()
         If Connection Is Nothing Then
            Throw New ArgumentNullException(Name)
         End If
      End Sub

      Public Sub IsOpen()
         If Not Connection Is Nothing Then
            If (Connection.State And ConnectionState.Open) = 0 Then
               Throw New ArgumentException("Operation requires an open connection.", Name)
            End If
         End If
      End Sub

      Public Sub IsValid()
         IsNotNull()
         IsOpen()
      End Sub

      Private ReadOnly Property Connection() As IDbConnection
         Get
            Return DirectCast(InnerValue, IDbConnection)
         End Get
      End Property
   End Class

End Namespace

The Pros & Cons

I've noticed that I am, indeed, far more likely to validate parameters now with this framework. I'm catching a lot more defects with it as well. The idea behind it seems to be working. However, it does have a few problems:

• It's failing to meet its extensibility requirement. In order to add new types, I have to hand-code a new overload to the That method into the Validate class. I need to find a new way to do that.
  
• It's not properly localized. (You can see the hard-coded English strings in the code sample above.)
  
• It adds overhead to the stack trace, which can be confusing to users who don't know what it is.
  
• It lacks a way to conveniently perform multiple tests on the same parameter validator object (aside from Visual Basic's With...End With block, which just looks unnatural). The current implementation tends to ask you to create multiple objects to work with the same parameter; although the objects maintain very little state (two variables, both object references) it's still more than I'm comfortable with.

Despite its drawbacks, most of which are addressable, its benefits appear to be worthwhile. My code quality is rapidly improving. Defect rates are dropping noticeably (and I feel comfortable attributing a good portion of it to better parameter validation).

The Request for Comments

So there you have it. At this point, I would really like to hear back from the community, and find out what you think of this thing. Should I be doing this? Should I be doing it better? How would you improve on it? What would your concerns be if you were using it or designing it yourself?

Remember, I'm the quintessential vacuum coder here, so your input is valuable to me.

Email: The Productivity Killer

Was I asleep? Did I miss something? I was under the impression that email was something you sent when you wanted to communicate something that was good to know, or needed an answer, but didn't need your direct attention or immediate response this second. If it was something really important, you flagged it as urgent.

But suddenly, every email we send has become a ticking time bomb that has to be answered right now, this very second.

God help you if you don't.

We've got people who send emails, and wait anxiously, huddled over their keyboards, waiting for their response. If they don't get one within a few minutes, they pick up the phone or open an IM window and ask, "Did you get my email? Did you read it?" If you dare answer in the affirmative, they ask you why you didn't respond. If it was that urgent, why didn't you just call? 

We've become a society that hinges on instant communication, leaving instant message services open, cell phones on 24/7, and email applications open and minimized so we know the very second we get an email.

But here's the skinny of it: the vast majority of the email we receive is unimportant garbage. And yet, Microsoft Outlook feels utterly compelled to give me a nice little notification every time I get an email from someone who wants to tell me there's leftover cake in the coffee area, that I can get a lower rate on my mortgage, or that some obscure fellow from offshore wants me to engage in a money-laundering scheme with him.

That kind of distraction is a major productivity killer. It's every bit as bad as an instant message window popping up from your mom, telling you that the cat puked on the kitchen floor.

Why? Because it breaks your concentration, and usually at really critical moments when you've got your groove on.

Face it: You don't really need to know those things. Microsoft Outlook shouldn't alert you to every email that you should receive. It should alert you to those marked urgent or flagged with a read-receipt, and then, only those from people on your approved senders list (usually, your contact list or the Exchange server). But no, that might actually make sense.

I came into work today and decided to conduct an experiment. I'm normally distracted by a ton of little things that I am certain are the primary killers of my productivity. They usually boil down to three things: email, instant messenger, and the compulsive urge to browse the Web. So I set out to eliminate those things. I shut down my instant messaging applications. I checked my mail once, and then exited the application. I didn't minimize it, I closed it.

Then, instead of leaving the defect tracking software open in a browser all day, where I'm tempted to keep checking my feeds and peruse Digg, I exported the list of defects to Excel, and then closed the browser. I then kept the browser closed all morning.

I've gotten more work done this morning than I have in the past two days. Why? Because I can concentrate. My train of thought isn't constantly being derailed by instant messages, email notifications, and that highly tempting little bookmark that mocks me on my Firefox Live Bookmarks toolbar.

These kinds of things are serious productivity killers. They simply pull me away from what I should be doing. So you can bet that I'll be seriously mitigating their impact on my work life in the future. It's a sad fact that in our world, we can't escape them because they've become so entrenched in the way that we work and live.

But, I'll set aside specific times to check my mail each day (first thing in the morning, lunch time, and right before I go home). Instant Messaging software is out. And I'll significantly reduce my use of the browser, constraining myself to those uses for which it was intended.

Maybe, if I do those things, and am able to get my work done at the office, I won't have to take my work home. You know, where there's a completely different set of distractions.

On Self-Control and Software Development

This essay was written months ago, and never posted. I resurrected it today, in light of certain recent entries.

Recently, as I was working to deliver a major release on a product I'm working on, I found myself sidetracked by a little project of my own.

You see, there's this little problem with one of the data fields in the database. It's not major, just an annoyance, like a four year old poking you in the ribs for an hour, asking repeatedly, "Does this bug you?"

Well, it's been bugging me for ages. And I found myself today doing database queries and pasting data into Excel to have Excel build update queries for me using formulas (nice little time saver that is) so that I could include those statements in the SQL script to accompany the next major release.

And then it hit me: no one asked for this. It's not included in the test plan for this release. It's gold plating. I'm doing this because I want to, not because the customer asked me to.

Whoa, there, cowboy. Get a grip on yourself. Set that stuff aside, and focus on what you need to do, and not what you want to do. There are far more important deliverables to worry about, and you don't have time to waste on unauthorized features or fixes. Especially when those fixes are for issues that don't negatively impact the application. (It was a display issue--first name before last name.) It's just fluff.

In reflection, I find myself experiencing these kinds of monumental self-control issues all the time. I get really excited about the things I could do for the customer, and I really want to do them for them. But the truth is that just because I can do something for them, it doesn't mean that I should do it.

Any change that I make to the product has the potential to introduce new defects into the system. That's why every change that I make to it must be tested.  It's why there's so much testing involved in software. (And if there isn't, something's seriously wrong.) And the testing doesn't just occur here, at my desk. It happens at the client. The product undergoes rigorous user acceptance testing. And testing isn't cheap--it consumes precious man hours, which equates to someone's hourly wages. And if I haven't gotten it right, it has to be fixed and retested. It can amount to massive amounts of money in man hours of testing.

Lets not forget the impact that the change has on updating the test plan, the release notes, requirements documentation, and user guides. Plus any associated costs with reprinting and redistributing them.

And what happens if the customer decides that my unauthorized change needs to be taken out? What if its impact on the system is so drastically negative that it must be removed? Can it be easily rolled back? And if it must be removed, what are the costs associated with doing so, and republishing all the updated documentation and builds?

Are you getting my point yet? The cost of a simple change isn't just what it takes me to code and test it at my desk. That's just the tip of a massive iceberg.

It takes a lot of self-control to prevent myself from adding features to a system when those features aren't (1) requested by the customer, (2) included in the project plan, and (3) absolutely critical to the current release.

The problem, I think, is that a lot of developers out there, myself included, don't get sufficient mentoring in the discipline of self-control when it comes to software development.

For example, we're all hailing the virtues of refactoring code to improve its maintainability, and I agree that that's a good and useful thing. But how many developers know that just because you can refactor a piece of code doesn't mean that you should? How many developers are out there bogging down project schedules because they're busy refactoring code when they should be developing software that meets the requirements for the project deadline?

(And here, I will sheepishly raise my hand.)

It occurs to me that before I ever modify a piece of code, before I ever touch that keyboard and write any new class or method, or create any new window or Web page, I should be asking myself, "Is this in the project plan? Is it critical to the current release?" If it doesn't satisfy one of those questions, I shouldn't be doing it.

The key to getting that product out the door on time is staying focused, and not getting sidetracked by fluff. Take it from someone with experience: it's easy to get sidetracked by fluff. Adding cool features is easy to do, because you're excited about it, and motivated to do it. Working on the required deliverables is hard work; it requires discipline and self-control. You have to stay focused and keep your eyes on the target. (You thought I was going to say "ball," didn't you?)

But we, as human beings, don't want to do what we need to do, we want to do what interests us, and what excites us. It takes an act of sheer will to resist that urge, to restrain ourselves, and get the real work done. I would imagine that one of the things that separates a mature developer from a novice developer is quite likely his or her ability to resist that urge to introduce fluff into software.

In the end, I think it might be a good idea if programming courses included curricula on self-control as a discipline for developers. And I mean that quite seriously. We need to have it drilled into our heads that we shouldn't be adding anything to the product that only serves our own sense of what's cool or useful. That's not to say that sometimes developers can't predict useful features before the users do; but they cannot and should not be introduced haphazardly into a product: they should be included as a planned feature as part of a scheduled release, so that they can be adequately tested and documented, and not just suddenly sprung upon someone as an easter egg.

There's a time and a place for everything. Gung-ho initiative has its proper place; software isn't one of them.