Knowing How is Not Enough

There’s an old adage that I heard once, and it’s stuck with me through the years:

He who knows how to do a thing is a good employee. He who knows why is his boss.

I’m also fond of this one:

If you can’t explain it, you don’t understand it.

So I’ve been ramping up on some technology that I’ve really not had an opportunity to really use before, and I’m very excited about it. To make sure I understand it, I’ve decided to go back to the MSDN examples, reproduce them one line at a time, and then document the source code as I understand it. It’s a great way to learn, and sheds a great deal of light on what you think is happening, versus what’s actually happening.

To be perfectly honest, the technology is AJAX. Over the last few years, I’ve predominantly worked for companies that haven’t had any use for Web services, so there’s been no compelling need for it. I’m starting a new job soon and it will rely heavily on Web services, and I really want to make sure I understand them well before I set my foot in the door. It has never been enough for me to know that you just drag a control onto a form or page, set a few properties and press F5. To me, that degree of abstraction is a double-edged sword.

When abstraction reaches the level that it has with Microsoft AJAX, you start to run into some fairly significant issues when it comes time to test and debug the application. The MS AJAX framework is no small accomplishment, and it hides a lot of complexity from you. It makes it so easy to write AJAX applications that you really don’t need to understand the underlying fundamentals of Asynchronous Javascript and XML that make the whole thing work. Consequently, when things go wrong, you could very well be left scratching your head, without a clue, and no idea where to begin looking.

Where, in all of this enormously layered abstraction did something go wrong? Was it my code? Was it the compiler? Was it IIS? Was it permissions? Was it an update? Was it a configuration setting? Was it a misunderstanding of the protocol? Did the Web service go down or move? Was the proxy even generated? If it was, was it generated correctly? Do I even know what a proxy is and why I need it?!

When I started learning about AJAX, we coded simple calls against pages that could return anything to you in an HTTP request using the XMLHTTPRequest object. Sure, it was supposed to be XML, but that was by convention only. The stuff I wrote back then (and I only wrote this stuff on extremely rare occasions, thank the gods), returned the smallest piece of data possible: a single field of data in flat text. It was enough to satisfy the business need, and didn’t require XML DOM parsing.

But even with DOM parsing, the code to make a request and get its data back via XMLHTTPRequest was a lot smaller than all the scaffolding you have to erect now. You might argue that you don’t have create a lot of code now, but that is just an illusion. You’re not writing it, but Microsoft is. Just because you don’t see it doesn’t mean it’s not there. Do you know what that code is doing?

In theory, the Why of Microsoft AJAX, or any AJAX library is to make our lives easier when it comes time to write dynamic Web applications that behave more like desktop applications. To a certain degree, they have. When they work. But when they don’t, I wonder if the enormous degree of abstraction they’ve introduced hasn’t dumbed us down to the point where we’ve ignored essential knowledge that we should have.

If you’re going to write Web services, or consume them, you should, at a minimum, understand what they are, and how they work. You should understand their history, how they evolved, and the problem that AJAX tries to solve. It’s not enough to know how to write a Web service, you have to know why you’re doing it, and why you’re doing it the way you are. That sort of knowledge can be crucial in making the right choices about algorithms, protocols, frameworks, caching, security, and so on.

But this could be true of any technology or practice we learn. AJAX, LINQ, design patterns, TDD, continuous integration, pair programming, and so on. Know why.

Try this simple litmus test. Explain something you think you know to one of your peers. If you can’t explain it clearly without having to pull out a reference or go online, you don’t understand it the way you think you did. Consider relearning it. It’ll only improve your value to yourself, your peers, and your employer.

Technorati Tags: Productivity,Software Development,Technology

The Absurdity of "Don't Reinvent the Wheel"

As developers, we've had this adage drilled into us from the beginning: Don't reinvent the wheel. In short, don't rewrite what's already been written. The idea is sound, in theory. You can save yourself time and money if you'll simply reuse existing code and/or components rather than writing them yourself from scratch. This time and money is saved up front when you write it (or would have written it), and down the road, when you have to maintain your system.

However, I'd like to point out another, equally applicable adage: There's nothing new under the sun. Anyone who's ever tried to write a novel, a short story, a play, a movie, a song, or a piece of software, will know this one simple truth: somewhere, at some point in time, it's already been written.

Every algorithm, every piece of code that we will ever attempt to write has already been written somewhere, at some point in time, by someone. Only the names have been changed. You're not inventing anything that is completely new, that's never been seen before. You should wisely disabuse yourself of that notion as quickly as possible.

In the grand scheme of things, at the application level, you may very well have an idea for a system that is unlike anything that has been done to date. But the algorithms that drive it have already been written. Bubble sorts, hashes, exception handlers, encryption, data access, socket management, shopping carts, entire application frameworks, date management, document management, serialization, port I/O, and all that other stuff has already been done. Further, it's already been done several times over in many different languages to varying degrees of success.

Tragically, if you're using a large application framework, like Java's EE or Microsoft's .NET, the chances are good that the functionality you're looking for is built right into the framework itself. The problem is that the framework is so vast that you'll spend more time looking for it, and determining whether or not it works the way you need it to work than you would just rewriting it yourself.

Application frameworks are stunningly afflicted with feature creep. They must do everything under the sun, must meet every possible need. The problem, then, is that their scope becomes so broad, so vast, that no one in their right mind could possibly grasp the totality of all that they can do. It is inevitable that anyone using them will reinvent some of their functionality. The scale of that functionality might be small (reformatting dates) or it might be substantial (pooled database connections).

In the end, it's absurd to think that we can possibly avoid reinventing the wheel. Of course we're going to reinvent it. Every application we write is a reinvention of someone else's wheel. It just so happens that our wheel is a custom wheel. All this paranoia about reinventing the wheel is blown out of proportion. A proper buy vs. build decision should never be neglected; but don't ever think for one minute that what you're creating hasn't been created before.

Consider the scenario where you're under the gun to get a product out the door. And I mean it's a really tight schedule. And don't act like it's a perfect world, and you have leverage over the schedule. This is reality here. In the real world, the customer controls the schedule, because it's tied to when the product is released, and that's tied to this big, huge monstrosity in another state or another country. The product's delivery schedule is a train barreling down the track at 120mph and no one short of God can stop it. Now, you have a very finite amount of time to work in. You need an algorithm. You know you could write it. Or you could look to see if someone else has written it.

If you do the whole Web search thing, you have to ask yourself a few questions: Is it from a source you trust? Is it in the language you're using, or do you have to convert it? Does it work? Does it need to be tweaked? If any of these fail, you're back to the drawing board. Time's wasting here, and that train's getting closer to its destination. If all the answers pass, you have to make sure you don't run into any copyright or licensing issues with that code. (You are paying attention to that, aren't you?)

If you decide to peruse your application's framework, you'd better hope it's well documented, and very easy to search. Good luck using the search features in .NET. It's not like the ASK.COM interface, where you can ask, "How do I convert a date in DOD format to Gregorian format?" Yeah. Good luck with that. On the other hand, you could ask your coworkers. They might know. Then again, they might not. If they don't, you're off to Google to get the information. Here's hoping you get a timely and accurate response.

Sure, this is an extreme example. But it makes my point: At some point, the work has to get done. You can't afford to spend days or weeks scratching your head about whether or not that wheel's already been invented. Believe me, it has. The problem is, there are a countless number of wheels, and none of them are labeled, and you don't know where to find the wheel you're looking for.

Stop wasting time, and invent your own damned wheel.

After all, whatever code you might reuse, is just someone else's reinvention of the same wheel.

Who's Testing Your Software?

There's a common mistake in software development: trusting the developers to test the software. Historically speaking, developers are the worst kind of testers, because we tend to use the software only as we designed it to be used. It takes a special kind of developer to be able to think outside the box and think like a user with little or no computer savvy.

In the comment thread to the article, Microsoft Admits Vista Update Glitch, one poster made this point:

Beta testing is not getting the bugs out of software because they got the wrong people doing it. Don't use computer savy [sic] people to beta test, use people like my wife who don't have a clue what makes the computer work. She can discover any glitch in software code, guaranteed. Her gift also applies to use of TV remote controls, etc.

To which came this reply (edited for brevity):

This is the best answer I have read for several years. Beta testers are people who do not do things that cause problems, rather, they look for features and bugs that are sometimes not there...The best Beta testers are people who are not knowledgeable and those who don't know the difference of double or single click.

These folks are referring specifically to Microsoft's beta tests for its operating systems (more specifically, for Windows Vista). But the general sentiment is true and universal: users who have never been exposed to your software in the first place, and have had little exposure to technology are frequently the best ones to determine whether or not it actually works. They have a disturbingly accurate ability to ferret out bugs that borders on the psychic.

As developers, we like to believe that our software is rock solid, easy to use, painfully obvious, and bulletproof. A user who can't tell the difference between clicking and double clicking, or why it's a bad idea to keep lots of applications open at once on a machine with limited resources, is the prime candidate for testing your software. If it's a Web application, find someone who's rarely used the Web or who only uses it for the basics: IM and email. One thing that they'll be able to tell you right away is whether or not the user interface is actually usable. And if you think for one minute that you shouldn't be designing clean, minimalist interfaces for the lowest common denominator of user, you've probably never met the average computer user. There are far more of them than there are of us.

We have some pretty interesting users for our Web applications. Some of them are fond of ignoring on-screen instructions. Tooltips, online help, field prompts, clearly written button text, user training...not much of that seems to make a difference. When all of that fails, what does the application do? How robust is it? How gracefully does it handle bad user behavior? For that matter, how gracefully does it recover from bad application, network, or hardware behavior? And does it alert the user to that kind of thing in a clear, friendly, and meaningful way?

You can't determine that sort of behavior by trusting your developers or your unit tests to find them. Inexperienced users will find far more than your tech savvy users will. That's not to say that your testing team shouldn't include tech savvy users; it absolutely should. But make sure that you include novice computer and Web users in your testing team.

Software Release Engineering

In his Coding Horror post titled How Should We Teach Computer Science?, Jeff Atwood blogs about the lack of coverage of release engineering in computer science courses. At best, he points out, it's given cursory coverage in these courses.

Now, I'm a self-taught developer. I started programming computers in 1985 or so, and I've taught myself everything I know. So I can't really comment about what the courses in a college or university are like. But I can say this: experience has taught me that a few things he says are absolutely, undeniably true. So in this article, I'm going to enumerate those things I think are really important, and how I built a software release process at the company I worked for.

The Ugly Truth About Release Engineering

1. Release Engineering is not simply deploying your product. There's a reason it's called engineering. It involves getting the latest version of the build from source code control, building the software, executing unit tests, building installers, and labeling the build if it's correctly built. It may involve pinning or branching the build. It requires a daily "good code" check-in time policy. It requires daily builds to ensure that you have software the compiles every day, and a means of notifying folks when the build is broken, and fixing the build-breaking code right away. It's NOT simple.
2. Consistent, disciplined use of source-code control is the bedrock of release engineering. At any given time, you might need to fix bug in the prior release. That's hard to do if you've already started changing the code for the new release. Branched builds allow you to do that. Also, versioned files in the repository allow you to view the history of changes to a file to recover from unintentional changes. You can also develop for multiple platforms while using many of the same files, sharing them across projects without having to worry if they're out of synch. Labels on files and projects tell you exactly which version of a file was used to create any given build so that you can recreate a project from the repository if you need to.
3. Building for your environment is not enough. You need a test environment that mimics your client's environment as closely as you can make it, down to the OS, the browser, the applications and the add-ons. Just because it runs on your machine when you press F5 from Visual Studio does not mean it's going to run on the client's machine. If you're developing for multiple browsers, install those browsers and test for them.
   (Ugly true story: our company accidentally allowed IE7 through the group policies. We had IE7 deployed everywhere. Our clients don't plan to upgrade to IE7 for another year, at least. Our product must run on IE6. I had to create a separate machine that was safe from IE7 downloads and strictly ran IE6 to be certain the product ran correctly.)
4. F5 is not enough. Every build should be a clean build. Every build. Don't ship files to the customer that aren't required to run the software. Create a build script that does the job. Excluding a file from a Visual Studio project doesn't delete the file from the folder, but does leave it in source code control (a good thing for versioning). To ensure  clean release, have your release script remove files you aren't using prior to shipment.
5. You need a checkin-time policy. All good source code must be checked in by a certain time every day. Code that isn't checked in does not make it into the daily build. This check-in time should be early enough that the release manager can start the daily build (if it's a manual process), or make the rounds and make sure that all code is checked in prior to it. I favor end-of day check-ins (around 4 PM) for nightly builds, but each organization is different.
6. The software must successfully build every day. A successful build is a good sign of project health. An automated build tool can be set up to execute the build in the off hours after everyone has gone home, and after all files are checked in. Once the build is complete, the pass/fail report is sent to your release manager. However, just because it compiles doesn't mean that it's entirely healthy or bug-free. Therefore...
7. Automated unit tests should be executed on every build. If you aren't using automated unit tests, you should be. They're not hard to learn, the tools to create them are freely available and they can improve the stability and quality of your code immeasurably. Incorporate the unit tests into your build script so that they're executed every time you build the software. Correctly written unit tests alert you to build-breaking defects quickly and immediately.
8. Build-breaking defects must be resolved before anything else. This includes any defect that causes the software to fail to compile or any defect that causes a unit test to fail. The team must adopt a "drop everything and fix the build" mentality. In my own personal experience, this view is not easily accepted in the early stages of a project, but during the later stages, when there's typically a "crunch" mode, and the build isn't riddled with build-breaking defects, developers are thankful that those defects simply aren't there.
9. You need a release manager. While you might have many people who contribute to the build, checking in changes and adding new content, you need one person whose primary responsibility is to ensure that the software builds properly every day. That individual is also responsible for your installer, and for identifying the code that breaks the build and ensuring that it gets resolved. The release manager doesn't resolve the defects himself unless he checked in the build-breaking defect (since he doesn't know anything about the defect); rather, he must play the role of the hard-nosed drill sergeant ensuring that the coder who checked it in drops everything to fix the build right now. If you can't build the product, you can't ship the product, and anything else that developer might be working on is a moot point. It's an ugly, painful job, but it's crucial.
10. You need a dedicated build server. This machine is clean, and does nothing but build your software. This guarantees that it injects no artifacts into your final product. It runs the daily build, executes the unit tests, and sends out the notifications when the build passes or fails. It might also house archived copies of each build's source code and binaries. It must be on the network, and should be backed up regularly. The Release Manager should have access to it, but no one else on the development team.

My Own Personal Release Process

It bears noting here that I've done the release process for two different companies. At one, it was for a full team of developers (about twenty of them), and the release process there was a nightmare. At that time, we couldn't get a release out in a month if we tried. So I volunteered to take on the job, and redesigned the process. It took about a week to get the process reengineered and everyone on board, but after two weeks we had daily builds working and everything was going much more smoothly.

I took many of those same principles and applied them to my new job. Clearly, some of them don't apply in a single-developer shop. But the basic principles are the same.

Source Code Control

1. All developers must use source code control.
2. All working, compilable code that does not break the build must be checked in by 3 PM every day. Code that is not checked in at this time does not make it into the daily build.
3. User names and passwords are required for accessing source code control.
4. The admin password is written on a piece of paper, sealed in an envelope, and stored in the CIO's desk. No one else has it.
5. Minimal rights to access the repository based on need are granted.
6. The main tree has the following subprojects: Build, Dev. Each tree's subprojects are mirrors of each other. Build is where the branched and pinned copies of the successful builds are. Mainline development takes place under the Dev tree.
7. Every file that is required to create or ship the project is included in source code control: source files, SQL scripts, Web pages, images, build scripts, unit tests, test plans, requirements documentation, etc.
8. Because we use SourceSafe, every weekend, during off-peak hours, regularly scheduled maintenance is performed on the repository to keep it in tip-top shape.
9. The repository is stored on on the network. This folder is backed up incrementally nightly, and fully weekly.

Build Process

1. Every afternoon, at 3 PM, all developers must have code they want included in the build checked into the repository.
2. The release manager does a final verification at 3:15 to ensure that all code is checked in.
3. An automated script fires off the build at 3:30 PM. It does the following:
   1. Clean the build folders on the build server. This involves deleting all files and folders from the project's build folder, ensuring a clean build.
   2. Get the latest version of the software from the DEV tree in the repository.
   3. Compiles the software and all of its dependencies. If the compilation fails, an email with high importance is sent to the release manager, notifying him of the failure, and the script aborts.
   4. Executes the unit tests. The unit test results are output to a text log file which are then sent to the release manager in an email.
   5. Executes a cleanup batch file that ensures that any files that should not be shipped with the product are removed.
   6. Creates the installer or archives the build into a ZIP file.
   7. Labels the build in the repository.
   8. If the build was successful, sends a "Build success" message to the release manager.
4. Note that step 3 may execute multiple times depending on whether you are targeting multiple platforms or releases (such as Debug and Release, or various browsers, or various OSes).
5. Upon receipt of a build failure email, the Release Manager reviews its contents, and identifies the offending source code. He then determines who checked that code in, and contacts that developer and asks them to resolve the defect as soon as possible.

   Important: Except in the direst of circumstances, the release manager should not attempt to fix someone else's defects. He should ask the developer to fix his own defects. If the release manager takes this task on himself, he'll quickly become inundated trying to fix all the build-breaking defects, and won't have time to do his own work.

6. The developer resolves the build-breaking defect and checks in the change for inclusion in the next daily build.
7. If enough build-breaking defects were present, the Release Manager may choose to manually rebuild the software once defect corrections are checked in.
8. If the build is shipped to the customer, it is labeled, pinned and branched into the BUILD tree in the source code repository.

In closing

We're a Microsoft shop. Although I've worked in Java houses, my limited experiences have largely focused on the Microsoft stack, and the process that I've outlined above is primarily geared for Microsoft Visual Studio and SourceSafe. But the basic principles should be pretty universal. You should be able to take them and apply them to just about any combination of source code repository tools, unit testing tools, and IDE (or text editor).

The primary thing to remember is this: if you can't build it reliably, predictably, and on a moment's notice, you're in trouble. When a development team knows they can't build the software, and when the testing team is sitting around for days or weeks at a time wondering when they're going to get a new release to test, morale suffers, tempers flare, and things rapidly go downhill. I've been there. I've seen it. It ain't pretty.

Every project needs a good, solid release process. I'm tempted to say that any release process is better than no release process, but that wouldn't be entirely true. A release process needs to be trim, make sense, bolster confidence in the project, and help propel the team forward towards success. That's what this process is designed to do.

I'm sure that others have some ideas on how to improve the process above. I'd love to hear those ideas. I'm sure that others have different ways of doing things. I'd love to hear that too. There is no silver bullet, and I'm not anywhere stupid enough to think that this plan is perfect. But I hope it's enough to help someone, somewhere get a little bit closer to a project that gets out the door a bit faster, healthier, and with its developers' sanity in tact.

 

Technorati Tags: Software, Release Engineering, Deployment, SCC, Unit Testing, Computer Science, Release Manager, Software Builds, Compile, Install

Email: The Productivity Killer

Was I asleep? Did I miss something? I was under the impression that email was something you sent when you wanted to communicate something that was good to know, or needed an answer, but didn't need your direct attention or immediate response this second. If it was something really important, you flagged it as urgent.

But suddenly, every email we send has become a ticking time bomb that has to be answered right now, this very second.

God help you if you don't.

We've got people who send emails, and wait anxiously, huddled over their keyboards, waiting for their response. If they don't get one within a few minutes, they pick up the phone or open an IM window and ask, "Did you get my email? Did you read it?" If you dare answer in the affirmative, they ask you why you didn't respond. If it was that urgent, why didn't you just call? 

We've become a society that hinges on instant communication, leaving instant message services open, cell phones on 24/7, and email applications open and minimized so we know the very second we get an email.

But here's the skinny of it: the vast majority of the email we receive is unimportant garbage. And yet, Microsoft Outlook feels utterly compelled to give me a nice little notification every time I get an email from someone who wants to tell me there's leftover cake in the coffee area, that I can get a lower rate on my mortgage, or that some obscure fellow from offshore wants me to engage in a money-laundering scheme with him.

That kind of distraction is a major productivity killer. It's every bit as bad as an instant message window popping up from your mom, telling you that the cat puked on the kitchen floor.

Why? Because it breaks your concentration, and usually at really critical moments when you've got your groove on.

Face it: You don't really need to know those things. Microsoft Outlook shouldn't alert you to every email that you should receive. It should alert you to those marked urgent or flagged with a read-receipt, and then, only those from people on your approved senders list (usually, your contact list or the Exchange server). But no, that might actually make sense.

I came into work today and decided to conduct an experiment. I'm normally distracted by a ton of little things that I am certain are the primary killers of my productivity. They usually boil down to three things: email, instant messenger, and the compulsive urge to browse the Web. So I set out to eliminate those things. I shut down my instant messaging applications. I checked my mail once, and then exited the application. I didn't minimize it, I closed it.

Then, instead of leaving the defect tracking software open in a browser all day, where I'm tempted to keep checking my feeds and peruse Digg, I exported the list of defects to Excel, and then closed the browser. I then kept the browser closed all morning.

I've gotten more work done this morning than I have in the past two days. Why? Because I can concentrate. My train of thought isn't constantly being derailed by instant messages, email notifications, and that highly tempting little bookmark that mocks me on my Firefox Live Bookmarks toolbar.

These kinds of things are serious productivity killers. They simply pull me away from what I should be doing. So you can bet that I'll be seriously mitigating their impact on my work life in the future. It's a sad fact that in our world, we can't escape them because they've become so entrenched in the way that we work and live.

But, I'll set aside specific times to check my mail each day (first thing in the morning, lunch time, and right before I go home). Instant Messaging software is out. And I'll significantly reduce my use of the browser, constraining myself to those uses for which it was intended.

Maybe, if I do those things, and am able to get my work done at the office, I won't have to take my work home. You know, where there's a completely different set of distractions.

Methodology Fundamentalism

If you aren't taking part in Steve McConnel's Software Best Practices forums, you're missing out. It's not very active yet, but it's got tons of potential. By all means, check it out.

In a really interesting thread, a poster suggested that he'd like to ban the use of the term, "Best Practices," given that it's become something of a convenient excuse that IT professionals use to excuse every insane practice under the sun, regardless of its logical suitability to the business or environment. In his particular case, he cites this lovely bit of whimsy:

The IT manager at my last place of employment had set up the network to change passwords every 30 days. His response every single time the multitudes cried out in pain was to hold up a sheet of "best practices". Yahhh!

Now, admittedly, lots of companies have this policy in place. But if the only reason you can explain the 30-day password change policy is because it's on a best practices list somewhere, you've got a problem.

The Chief Software Engineer at one of my previous employers had a staunch policy that Is-A inheritance had to be maintained at all times, and could not be violated at any time whatsoever, even if you had a perfectly good reason for doing so. He would rewrite your code behind your back to make sure it happened. "Best practices, dontcha know." End result: confused developers who had code changed on them, extra code written to support additional classes, angry developers, adversarial relationships, hostile work environment, and, eventually, employee turnover.

Blind adherence to Best Practices is a Very Bad Thing. You need to have a compelling reason to do so. If you're doing it just for the sake of doing it, you're shooting yourself in the foot, and likely pissing people off in the process.

The same is true if you're just blindly grabbing hold of The Next Big Methodology. We're inundated with methodologies in this business. Test Driven Development, Refactoring, Visual Modeling, Big Upfront Design, the Waterfall Method, Use Cases, Agile, Extreme Programming, and on and on and on. Every one of them involves risk. The risk is largely due to the fact that there are people involved, and people tend to clash, especially when they're under pressure. So you can't just foist any old methodology on them; them you have to pick the one that's going to work best for them, given the number of people that you have, the resources at your disposal, the time you have, and the project you're doing.

And let's be very clear, people: There is no Silver Bullet Methodology.

A few months ago, I responded to a thread on the Braidy Tester (a great site, love Michael Hunter's stuff) that blind adherence to "best practices" and adoption of "The Next Big Thing" without adequate analysis of the risks involved was unwise. I contended that sometimes you had to deviate from what everyone else was doing and develop your own customized methodology that might be a hodgepodge in order to get the job done, because what worked for ABC corporation didn't necessarily work for Your Real World Company, Inc.

A Microsoft Developer, who shall remain unnamed, said, in short, that I had an amateur opinion.

But the truth is that I've watched a lot of folks (and entire companies) become Best Practice Zealots, and turn this whole thing into a Fundamentalist Religion. Best practices and methodologies are supposed to help you get your job done better, faster, on time, within budget, and according to specification and customer expectations. They aren't supposed to be an iron rod with which you can bludgeon everyone around you. Too many companies are willing to just snatch up the next Silver Bullet Methodology and apply it without determining whether or not it fits their business model, their project, or their customer.

If you're going to embrace "best practices" remember this: they're guideposts, sitting on the side of the road, pointing you towards success. If you're using methodologies, they're supposed to be the route that gets you where you're going without hitting any tolls, construction sites, congestion points, and so on. Best practices and methodologies are supposed to be used together, to get you where you want to go: to a successful product delivery, with the whole team, the client, and the product intact. If anything is going awry, be willing to reevaluate your selection of practices and methodologies.

Caveat: Don't switch methodologies in the middle of the project. Jeesh. I'm not that stupid. (Though I can see where some people might think so.)

Dew is to Water As Want is to Need

On a recent Coding Horror entry, Jeff Atwood brought up a really interesting point about the power of observing users versus asking them. Paraphrased, what users actually want is typically not what they think they want or tell you they want.

It's funny and sad because it's true.

It took me a long time to understand the difference between want and need. I may want a nice, tall fizzy bottle of Mountain Dew, but my body needs water. It doesn't need Mountain Dew. Sure, Mountain Dew tastes better, and I like the fizz, and I look way more cool when I'm holding it, but I don't need it. I need water to hydrate my body and keep me numbered among the living. There's a big difference.

Similarly, when users tell you they want a piece of software that does X, Y, and Z, what they usually need is something that does A and B. (Usually, A and B are something on the order of "It works well" and "It doesn't corrupt my data.")

Nonetheless, trying to get users to tell you what they need is akin to extracting molars from a chicken. It's nearly impossible. They'll give you something like this:

• It has to look really good. You know, like SILF. (Software I'd Like to @#$%)
• It has to be fast. Really fast. Like, it has to be so fast that I get whiplash when it starts up.
• It can't hoard memory. Cuz I'm using Windows 98. In fact, can you make it use no memory at all?
• It has to be secure. Really secure. Like, Fort Knox secure. Oh, but I want to be able to pass it around on the Internet and share it with all my friends. Or on a USB drive. Or whatever. Ooh! BitTorrent!
• It has to be a Web app to. With Flash. In fact, do it all in Flash. But I have to be able to use it on my cell phone. And on my XBox. And FireFox. FireFox totally pwnz Micro$oft.
• Everything has to be done in my company's colors: Black and brown. I want all the text in this really cool dark brown color, and the background all has to be black! It's bitchin'! And flames everywhere! And I have this cool soundtrack I want to play throughout the whole thing! And every button should be a different color, and they should make machine gun sounds when you click them! And then explode with a giant fireball!
• I've only got a budget of $500.
• I need it by Friday.

Oh, God. The horror. The humanity!

While this list is obviously heavily laden with hyperbole, it's not too far from the truth. A completely user-driven set of application specifications, expressing their wants would likely provide nothing that they needed. At some point, you have to realize that users are, in fact, dreaming about toys.

In the end, software is about enabling users to get their work done faster and easier with a minimal amount of hassle. But users don't know that. Largely, they think it's about fun. Not all software is a first-person shooter or a massively multiplayer online role-playing game. (Sadly.) As a member of a software team, it's our job to identify their needs, and create software that meets those needs without getting sidetracked.

If you're starting a new system from scratch, don't waste your time asking the users what they want; ask them what they need. Explain the difference to them. (Begin by making sure you understand it yourself.) Explain that everything that isn't needed that is added to a product incurs additional cost and delays the delivery of the product, and that someone has to pay for it.

Build systems into your products to monitor which features are being used the most. Don't trust users to tell you. They aren't thinking about it. This kind of information is very useful for determining where to spend your time improving the product. It's also useful for identifying features that you think are critical (in business applications, especially) that should be used, but aren't being used. You can figure out why they aren't being used and target those areas for resolution.

If you write a solid subsystem like that for your software, you won't have to rely on user feedback, which isn't always reliable. A solid monitoring system will not lie to you. And that information will help you make much more intelligent decisions in the future. It's a question of silently observing the users as they actively use the product, instead of asking them about it after they've done so. All of us have a pretty short attention span when it comes to software use. I have no idea what the most used commands are in any given software package based on my own usage scenarios. For that reason, you couldn't ask me to tell you what commands or features I use and get a meaningful answer from me.

So think about whether or not you need it. If you do, invest the time to do it. But don't do it because you want it. Do it because you need it.

Dew? Or water?

Everything I Need to Know about Debugging I Learned from CSI

I started to write this blog entry a few months ago, but it quickly got out of hand. With a little encouragement from a tester at Microsoft I was encouraged to publish it to a magazine, but things at work got out of hand, and I just never found the time. Since I am the quintessential procrastinator, I've decided to just publish it here, so that it will at least get published in some form. So, without further adieu, I present it for your general amusement.

—Mike

Okay, I admit it. I'm a CSI dweeb. No, I don't like CSI: Miami (it blows). I like the original CSI. And although all of those shows are equally implausible and unrealistic (let's face it, no CSI team is that thorough, that precise, or that good), the very premise of crime scene investigation and its parallels to defect resolution hit me like a ton of bricks recently.

At first thought, it's just a corny idea. But then, the more I thought about it, the more I realized that the idea isn't as silly as it sounds. I thought about writing this article as a parody, but when I set out to do so, it didn't turn out that way.

Some folks might look at this and laugh their butts off. But read it, think about it, chew on it, and then, after you're done, if you still think the parallels aren't striking, go ahead and laugh.

The Basics of Crime Scene Investigation and Defect Resolution

We’ve all done the dirty work in software development: defect resolution. In many companies, it’s the first place where new developers are unceremoniously dumped when they are brought on board. The thinking is that it will familiarize them with the product. “You’ll learn the code!” they say. “Then you can move into the development.” These poor saps are armed with reams of source code, an IDE, and a compiler, and sent marching into the battlefield with a stack of defect reports and an order to make progress repairing a system with which they typically have no experience whatsoever.

This thinking is fundamentally flawed. You don’t want someone who doesn’t know a lick about a complex software system trying to resolve its defects. But that’s a subject for another article.

When confronted with a defect report, there are a certain number of predictable responses that tend to flash through every developer’s mind when he hears it:

1. “It’s an ID10T error.” This one’s my favorite. It can’t possibly be a defect in any code that I wrote. The user must have done something wrong. Everyone knows users are st00pid. I mean, just look at them. They’re like, lame. And stuff.

   In the Dark Ages, this might have flown, but this is the 21^st Century. There’s this thing called a presumption of innocence.

2. “We already fixed that.” Another keen insight. If it’s already fixed, why is it still happening? If it’s already been fixed, you’ll have to provide proof that it’s been fixed in a build in your test environment. If you can’t prove that, then what you’ve likely done is fixed the wrong thing and claimed victory. As we’ll see later, this will fall under the novel concept of “Convicting the wrong suspect.”

3. “He did what? You’re not supposed to do that.” Okay, let’s get something straight: just because a developer might not do something doesn’t mean that a user won’t. Users do unpredictable things all the time. And your lack of coding for it doesn’t mean that it’s not a defect. Holding the users at fault for being unpredictable is not an acceptable excuse. Inadequate code coverage in the test plan is a defect. Get used to it.

4. “It’s a known issue and we can’t do anything about it.” Okay, that’s marginally acceptable. Sometimes. But have you made an effort in the software to barricade the users from the effects? Trained them? Documented it? Why are users still running into it?

5. “I know exactly what causes that. Let me fix that right now.” The most fatal of all the answers. This knee-jerk reaction is what leads to reaction #2. This is always a bad response, and only in the rarest of cases is it ever right. I would estimate the chances of it being right as roughly equivalent to those of a stray cosmic ray setting off a nuclear disaster that ended the world within three seconds of your reading this sentence. Okay, that’s extreme. But you get my point.

   Defects resolved this way are rarely documented properly. Test plans are rarely updated to ensure that the fix is correctly tested. They’re just quietly slipped in, like an Easter egg, and no one is any wiser. The only thing that gives them away is that there’s a new version of the file in the source code repository. (You do have a source code repository, right?) And that’s assuming it’s the only change in that version of the file.

6. “Oh God. What now?!” Don’t even tell me this has never crossed your mind. We’re all swamped. Products slip, schedules get crazy, we work overtime, and work piles up. We try to prioritize, but things get missed. Defects get buried in a stack, and some of them just don’t get fixed. We don’t see defects as challenges, we see them as annoyances, burdens, more junk sitting on our plate when we’re already seriously overtaxed.

We all know that there’s more clever and witty responses out there. Some of them I just can’t put in print. But for the purposes of this article, I think we’ve painted a pretty accurate picture of defect resolution as it stands today: it’s viewed as a dull job, one that’s resented, a pain in the neck, and one that no one looks forward to.

Let’s face it. You have to essentially tell the developers that their code is broken. Or, you have to tell the users that they don’t know what they’re doing, or that there’s nothing that can be done, and that they just have to wait. Either way, it’s a no-win scenario for you. You always come out the bad guy. No one wants to cooperate with you, because they know that you’re only going to give someone bad news. If you’re new to the company, you probably don’t even know anything about the product to begin with, so you’re flying by the seat of your pants as well. And if your company is like most, you don’t have the best equipment or software to make finding those defects as easy as it could or should be.

Who the heck would want that job?

Now, turn your attention to another group of individuals who are stuck in the very same situation. Their job is no different. They have to do the same basic thing. They have to wade into a situation that they know nothing about, typically understaffed and underequipped, and determine whether or not a problem occurred. Then they have to accuse someone of being in the wrong, or telling both sides that no wrongdoing took place at all (potentially angering both sides). Through it all, their job is to figure out the who, the what, the where, the why, and the how of it all. Crime scene investigators do this every day. They wade into a new crime scene, knowing only that a crime may have been committed, that one or more suspects are at large, and they have a crime scene to work with. They’re given the evidence, and told to run with it. Sound familiar? It should.

Crime scene investigation is essentially the act of solving a complex problem: finding the truth in a vaguely described problem when you’ve got few hands, little money, a lack of resources, a finite amount of time, and every witness can be a suspect. At the end of each case, they have to render their findings, and simply state the facts, regardless of whether or not the victim or the justice system likes it. Sometimes they’re praised, sometimes they’re despised. But they’re frequently overworked and underpaid, and the amount of care they have to take to get their jobs done is mind-boggling. If they make a mistake that tampers with the evidence, an entire case can get thrown out of court.

The Process

The job of the crime scene investigator is to determine the following:

1. Whether or not a crime was committed.
2. If a crime was commited, what the crime was.
3. If a crime was commited, who committed it.
4. if a crime was commited, how it was commited.

You’ll note that the investigator is not responsible for prosecuting the crime. His job is simply to collect the evidence, analyze it, and form a theory that fits the facts and leads to the perpetrators of the crime (if any).

The crime scene investigator uses the scientific method to arrive at his or her conclusions. The American Heritage Dictionary defines the Scientific Method as:

n. The principles and empirical processes of discovery and demonstration considered characteristic of or necessary for scientific investigation, generally involving the observation of phenomena, the formulation of a hypothesis concerning the phenomena, experimentation to demonstrate the truth or falseness of the hypothesis, and a conclusion that validates or modifies the hypothesis.

In other words, “Prove it, buster.”

Here’s the gist of it: You need to gather the facts, form a hypothesis based on the facts, and then prove your hypothesis. In crime scene analysis, proving the hypothesis leads you to one or more suspects who is or are more than likely guilty of committing the crime. You don’t rely on the “hunch.” Hunches put innocent people behind bars, wasting taxpayer dollars, and getting cases thrown out of court or convictions overturned on appeal.

In defect resolution, the same practice applies. You gather the facts, determine whether or not an actual defect exists, and then review the facts to create a theory. Then you prove the theory. If you can’t prove the theory, you don’t have a case. You’ll likely fix the wrong code, incorrectly mark it as “not reproducible,” “by design,” or “user error,” or fix part of the problem while the other parts that contributed to the problem remain uncorrected.

As a CSI conducts his investigation, certain guiding principles govern the way that the investigation is conducted. These are:

• Humans lie and make mistakes; evidence doesn’t. When you can’t rely on the witnesses, keep going back to the evidence to find the truth.
• You always want to convict the guilty party. You never want to convict the wrong party of the crime. When you do get a conviction, you want it to stick; you never want it to be overturned on an appeal.
• Your first suspect is usually not the right suspect. Knee-jerk reactions tend to be wrong, and based on faulty assumptions. Careful evaluation of the evidence leads you to the right suspect(s).
• You want to convict all of the guilty parties, not just one or some of them.
• Don’t be swayed by your emotions or personal involvement. Always remain detached and objective.
• Expensive tools aren’t always required to analyze the evidence. Sometimes, it’s simple tools that can be found at our fingertips every day that will do the trick.
• Patience and persistence rule the day.
• There ain't nothin' glamorous about this job. It's full of blood, gore, hate, anger, greed, fecal matter, tire tread, and a lot of pavement. No one ever cooperates willingly, but they all want answers now. And no one is ever guilty. Get used to it.

So without further adieu, let's see how the CSI process parallels defect resolution. Hold onto your butts people, it's going to be a bumpy ride.

The Process

Identify the Crime

Any time an alleged defect occurs in your product, treat it as a crime. After all, some part of your code has theoretically failed to meet its contractual obligation to the end user (or, so we’re assuming for the purposes of this article). You’ll first want to know what this alleged crime was. Was data corrupted? Did the software simply vanish off the screen? Did an error message appear? Did the screen lock up? Was sensitive data compromised?

Once you identify the crime, you'll need to categorize it. Its severity helps you to determine how quickly it needs to be resolved.

It's important to note, however, that at this point, you don't know that a defect has actually occurred. All you really know is that something happened. You still have to prove that it's a defect. So you start taking copious notes. This is why you need a defect tracking system. You need a place where you can record as much information about the event as you possibly can--preferably in one place.

Identify the Victim and Witnesses

The victim and witnesses provide valuable insight to what happened when the crime occurred. But it’s important to realize that witness accounts tend to be fuzzy at best.

Crimes and defects tend to catch people by surprise—they’re usually not paying close attention when these things happen, and the panic factor is pretty high, so relevant and often important details tend to escape their notice. You’ll still find their input valuable for recreating the series of events that led up to the event, and certain general information about it; but you will do well to remember that witnesses typically are not an authoritative source of information.

Identify the Crime Scene

When an alleged defect occurs in your product, treat the event as a crime scene.

You’ll want to know where and when the event occurred, what version of your product was being used, what OS it was being used on, what browser was being used, any plug-ins or service packs applied, what the user load was at the time, and so on. Any of these might have a bearing on the crime that was committed. You’ll need to know this information so that you know exactly which version of your software to use when you recreate the "crime scene" later.

Preserve the Crime Scene

It is absolutely imperative that you preserve the state of your software while you are attempting to identify the cause of the defect that occurred. If the environment is changing, someone is tampering with the evidence, and the evidence can no longer be relied upon to point you to the right suspect.

This is why a solid revision control process is critical to defect resolution. Every build must be labeled in your source code repository so that you can recreate it, and test it for defects. You must be able to recreate the environment later, and that means being able to use the same version of the software that the defect occurred in. You’ll hopefully have the means to do it on the same OS, with the same browser and plug-ins that the victim was using, but that’s not always feasible due to cost constraints. But having access to the source code that was used to create the software is absolutely essential. Your suspect may be hiding in there somewhere.

Collect the Evidence

The evidence is what you will base your findings on. Everything else will be ignored, because only the evidence can be relied upon to tell you the truth. Evidence includes the source code for the build in question, a fresh copy of the database, any exceptions that occurred, event log entries, data files, screen shots, and other output from the software generated at the time that the defect occurred.

Do not include email communications as evidence unless they were system-generated; interpersonal communications are testimony, not evidence.

Collect Testimony

Testimony includes emails, voice mails, and oral accounts from users that describe what happened when the defect occurred. It is vitally important to note that testimony is not evidence. Rather, testimony helps you to evaluate the evidence. Testimony is subject to witness credibility and the fallibility of human recollection.

That probably sounds pretty harsh, but it’s a simple statement of fact. As we’ve mentioned before, folks tend to be caught off guard when something goes wrong. They aren’t expecting someone to snatch their purse, jack their car, or corrupt their data. It takes them by surprise. Consequently, they don’t tend to be looking for the vital details that you need from them when you are trying to figure out what happened. They’ll tend to remember vague details, but not the specifics.

There’s also the uncomfortable truth that we simply don’t like to admit that we might have done something wrong. So we’re reluctant to divulge information. And we’re emotional when our data is corrupted or software that we’re required to use doesn’t behave as it’s supposed to and we’ve got tons of work to do. We get angry, even hostile. It’s human nature. We all do it. But the net effect is that our testimony in those situations isn’t always as objective as it might be. It’s subjective, defensive, and guarded.

Finally, witnesses can only tell you what they saw, not what happened internally. If it were a medical condition, we would say that they saw the symptoms, and not the underlying disease. As a professional, you don’t want to treat the symptom; you want to root out the disease. But a witness can’t tell you anything about the disease because she simply can’t see it.

So the witnesses’ testimony helps you to evaluate the evidence, but it isn’t evidence in and of itself.

Be careful, however, that you do not treat witnesses with hostility. Just because witnesses may not be accurate sources of information does not mean that they are dishonest sources of information. Always treat them with respect and understanding. Remember the golden rule when interviewing the witness: You’ll get more with honey than you will with vinegar.

Analyze the Evidence

Once you have all of the evidence, you must analyze it to determine what happened. Sometimes, the crime that was reported turns out not to be the crime that occurred. It turns out to be the wrong crime altogether. Or there was no crime at all. Careful analysis of the evidence determines whether or not a crime occurred at all; if one did, analysis of the evidence determines when it occurred, where, and who the likely suspects are. You can use the testimony to evaluate the evidence, such as to reconstruct the order of events that led up to the crime in question. But, again, do not treat the testimony as evidence.

You should use every available tool at your disposal to evaluate the evidence, including your application’s debugger, tracing tools, the event viewer, query tools, hex viewers, file parsers and viewers, system diagnostic utilities, network utilities, and so on. Expensive tools aren’t always required. A simple text editor is often sufficient for viewing data files, and a baseline graphic editor will suffice for viewing graphics files in most cases. Wherever possible, use the simplest tool that will accurately evaluate the evidence before you. There’s no need to inflate the costs of your investigation.

The outcome of your analysis should be a theory of the crime. You should have one or more suspects: the portion(s) of your code or the external components that caused the defect.

The next step is not to rush off and fix the code. Rather, you need to prove your theory. After all, you want to convict the suspect, and you want to convict the right suspect. And you don’t want to put this suspect in jail, only to find out that the same exact crime is being committed by another suspect that you hadn’t considered. This is especially true if your suspect is the victim.

Recreate the Crime

So now you have a theory. You just have to prove it. And you have to prove it beyond a reasonable doubt. So, you have to recreate the crime scene, and then walk through the crime itself. That means putting the victim and the witnesses back where they were at the scene of the crime, and taking all the steps that lead up to the moment when the crime occurred.

At the end of the recreation, if your theory called for one suspect, there can only be one suspect who contributed to the crime. If there’s more than one suspect at the end of your recreation, you’ve got a problem. If, in the course of recreating the crime you find that some other unexpected entity was involved, you need to go back to the evidence collection step, and start over. You’ve got another suspect out there somewhere that you didn’t know about.

Once you’ve identified all your suspects and you are reasonably sure you can prove they are the cause of the problem, you need to verify that your theory cannot be disproved. Is there any possible way that the crime could have been committed by another suspect? After all, that’s what a defense attorney would claim; the defense is going to do everything in their power to shoot your case full of holes. You want to be absolutely sure that your case is ironclad. As a developer attempting to identify the cause of a defect, you want to make sure that you’ve eliminated all the possible causes of a defect. Is there any other way that this defect might be caused that you haven’t considered? If there are, you need to account for them.

One last caveat: Never make the assumption that the victim is the suspect; conversely, never assume that the product is the suspect. Prove it. Be sure. And make sure that the evidence proves your case. Don’t rely on hunches or speculation. Neither the end-users nor the developers are going to appreciate being accused of being in error. If the suspect is an external entity, the developers are even less likely to be happy, because you’ve just identified extra work for them; be absolutely certain you have the facts to back up your case.

If you cannot disprove your theory, you’re ready to move on and prepare your case.

Prepare Your Case

Document everything. Preserve the evidence. If this case ever presents itself again, you’re going to want to know what you did to research it. A decent defect tracking tool is invaluable in this regard. If you lack one, there’s no reason you can’t keep it in your source code repository (unless there’s a storage limitation on it).

Even if you can’t identify the suspect, and this case is unsolved, you can keep this one in your Cold Case Files. If it rears its ugly head again, you can reopen the case and you’ll have all that evidence from the previous investigation at your disposal.

Obtain a Warrant

Now, with evidence in hand, and a solid provable case, you’re ready to obtain a warrant for the suspect. Up until now, you haven’t had enough to do that. But with the evidence, which doesn’t lie, and a thoroughly documented case, you can make your case to the development team. You’ll have the information to convince them that a defect exists, or it doesn’t.

Carefully lay out the facts, tell them what happened, how you determined that it occurred, and how you eliminated all the other possible suspects. Rely on facts, not conjecture. This is where your personal detachment is critical. You’re not supposed to be on their side or the users’ side. You’re on the truth’s side.

If no defect exists, say so. If one does exist, say so. Don’t make an issue out of it or point fingers. Simply state the facts. Be sure to point out how severe the issue is in terms of data corruption, application downtime, usability and so forth—so long as those pieces of information are based on facts and not opinions. These pieces of information will help the project team decide how quickly the defect should be resolved.

Arrest the Suspect(s)

Once the development team is convinced that the defect is real, the development team will take the information you’ve collected and use it to prioritize and correct the defect. This particular defect should no longer victimize anyone. Once the defect is resolved, the case is closed.

In Closing...

Most developers are constantly burdened with having to research and resolve defects. I know I am. But the problem is that we tend to treat them as annoyances. We don't see them with the weight that they deserve. To us, they're just "something that went wrong" and need to be addressed. So we quickly glance at the code, make a best guess and accuse the first suspect that walks by. All too often, we accuse the wrong suspect. Just as frequently, we take the defect report that’s given to us, add it to the growing list of things to do, and hope to get around to it at a later date when “more pressing concerns” don’t occupy our attention.

Perhaps the problem stems from how we view defects. Perhaps we see them as just another blip on the radar—another defect. I suspect that as developers we tend to think of defects solely in the light of the code base, and rarely in the light of the victim: the end user affected by the defect itself.

But what would happen if we changed our thinking by creating teams that viewed defects as offenses against victims that set out to prosecute or acquit the suspect by collecting the evidence and evaluating the testimony from witnesses? By elevating the perceived seriousness of the defect, perhaps we can increase the desire to get them corrected, and get them corrected correctly the first time. Too farfetched? Corny? Maybe. Maybe not.

Don’t make the mistake of thinking that I’m advocating the creation of a real CSI unit in your software shop or IT department (that would be an absolute disaster and insanity in and of itself). I don’t think you need to treat end-users as hostile witnesses. What I am advocating is the application of the scientific method to the resolution of defects: Know the difference between evidence and testimony and the value of proving your case. It’s likely to be far superior than scratching the first itch that irritates you. Think your way through a defect; get serious about resolving it correctly, deterministically, cost-effectively. There’s nothing in that statement that detracts from code quality. In fact, it enhances it.

I would hope that we all want to write better, more stable software, and that when defects are found, we'd address them quickly and with a minimal amount of cost. Part of minimizing that cost is identifying the right cause of the defect the first time.

What started out as a parody for me led me to rethink my process for defect resolution. There are ways that I can tighten it up, and improve my process. I might get laughed out of the door, but when it comes down to it, the only real thing that matters is whether or not I nailed the real perpetrator, and did so more efficiently than I did before. And isn't that the whole point of this exercise?

Guiding Principles for Software Projects

If you haven't seen this blog post from Patrick Cauldwell, you should. In it, he lays out the guiding principles he's following on the project he's working on. (Credit to Scott Hanselman, where I found the link.)

Take a gander at it. It's a truly excellent and thorough list of principles, and it's one I plan to post near my desk (especially given recent posts I've made). You may not be able to achieve every one of them, but it's not the achieving that's important--it's the effort. It's getting the darned thing started.

Creating My Own Personal Hell

In today's excellent post, Jeff Atwood blogs about the classic development process mistakes. He provides this handy list, which I will blatantly repeat here:

People Mistakes	Process Mistakes	Product Mistakes	Technology Mistakes
Undermined motivation Weak personnel Uncontrolled problem employees Heroics Adding people to a late project Noisy, crowded offices Friction between developers and customers Unrealistic expectations Lack of effective project sponsorship Lack of stakeholder buy-in Lack of user input Politics placed over substance Wishful thinking	Overly optimistic schedules Insufficient risk management Contractor failure Insufficient planning Abandonment of planning under pressure Wasted time during the fuzzy front end Shortchanged upstream activities Inadequate design Shortchanged quality assurance Insufficient management controls Premature or too frequent convergence Omitting necessary tasks from estimates Planning to catch up later Code-like-hell programming	Requirements gold-plating Feature creep Developer gold-plating Push me, pull me negotiation Research-oriented development	Silver-bullet syndrome Overestimated savings from new tools or methods Switching tools in the middle of a project Lack of automated source control

I was horrified when I saw that list. I've seen it before, of course, being a big fan of Steve McConnell's works. But seeing the list put there like that, so succinctly, gives me this sinking feeling in the pit of my stomach. I think the project I'm on is guilty of almost every one of those. The problem? I'm the sole developer on it (admittedly, however, not by choice).

I was thrust into this project from day one with a completely unrealistic schedule, no requirements, no goal, no vision, no contact with the users, no management team, and no freedom of movement in the tool selection. When it comes down to it, I have to fill all the roles in the project: I have to be the DBA, the software engineer, the business analyst, the architect, the web designer, the project manager, the customer support representative, QA engineer, the risk manager, the technical writer, the release engineer, and the software developer.

I get paid to be the software developer. It's my job title. I'm not particularly very good at much else; I slog my way through the rest of the roles. In three years, I've been pulling my hair out to delivery *everything* related to the project, and as a result, every facet of the project has suffered. The code quality has never been what I would have liked it to be. In this particular case, the old adage, "Jack of all trades, master of none," has certainly proved to be true. My inability to specialize in my chosen field has prevented me from being able to deliver code of a superior quality; instead, I've delivered substandard products across the board: code, product, documentation, and customer service.

Our process is haphazard at best. I am constantly rushing to get the builds out according to a quarterly schedule. The customer seems to want to get the builds as quickly as possible. I never have time to fully regression test the builds. The customer wants to constantly shortchange the testing time in order to get the builds in their hands. So, I get them the builds when they want them, giving them the caveat that I *know* there are bugs in it. I ask them to fully test the build from end to end. But they never do. They find a handful of bugs and throw it back to me, insisting that the bugs they've found must be fixed before they will continue testing. The company will not hire a QA or testing staff because they believe that I can do it all. So I have to stop my work, resolve the handful of defects the customer has found, and rush that bug fix build out the door. Again, the customer wants that build as quickly as possible--without sufficient time allocated to me to regression test it. So I give it back to them, and they shortchange their own testing again. And the cycle repeats itself.

The customer seems to think that the entire problem rests at my feet. Perhaps it does. I don't know. I'm just tired of being shortchanged on the testing time, the lack of hands to help with the testing (even if it's temp assistance), and the proper prioritization of testing in the development process. You CANNOT shortchange testing in software development. The number of defects in this project has soared out of control because the customer will not allow me to take the time to test it fully, nor will they fully test it themselves.

Each time they get a build, they insist on getting an update to the test scripts, which I dutifully try to update--but they don't give me enough time to complete them--so they get partially updated test scripts. Even then, I know that they aren't using them. How do I know this? Because when I use the test scripts to test the software, I find defects that aren't reported in our defect tracking system. I know they aren't using them. So why do they demand that I prepare them for them? Are they using them as some sort of ego-bloating paperweight on their desks? I know the work has to be done, but dammit, why isn't sufficient time allocated for it?

I cannot tell you how many weeknights and weekends I've given up for this project. And it's not like I'm getting paid overtime for this stuff. I have no personal life. And "thank you" and "good job" is simply not enough. I don't get paid for all the other hats I wear. I get paid to be a software developer. I don't get paid to be the the DBA, the software engineer, the business analyst, the architect, the web designer, the project manager, the customer support representative, QA engineer, the risk manager, the technical writer, and the release engineer. And management doesn't see that. All they see, every time I bring this up, is that the customer is apparently "happy" and that I can always be relied upon to get the work done.

Of course I can. I'm killing myself to get it done. It's Classic Mistake #4: Heroics. The company thinks I'm freaking Superman.

My God. What have I done?

To a large degree, I led the company to that opinion. I took on too much work. I let them believe that I could do it. But eventually, I started pushing back when I realized my mistake. I set realistic expectations about what could and could not be done. The course was reasonably corrected.

But in every project, in every organization, there is a certain Wall. Beyond that Wall ye shall not pass. It is the Wall of Absolute Resistance, and no amount of pushing, pleading, or reasoning will avail you. The forces on the other side are simply not reachable.

You can preach best practices until you're blue in the face, and some organizations will simply not see the light of day. You can talk about automated unit testing and they won't care. You can talk about investing more time in testing, to reduce the costs of defects as early as possible, and they'll still be more interested in beating a competitor to the market with a bug-riddled application than they will be in sparing users a frustrating experience and corrupting their data. You can tell them that you are going to have to bring on more hands earlier in a project to ensure its success, and they'll be more worried about the costs of the new employees' benefits in the short term than they will be about the rapidly escalating costs of the software's development and maintenance in the long term.

What do you do in an organization like that? What should I have done? The short answer is this: I don't know what I should have done in the past. It's been a valuable learning experience for me, that's for sure. In newsgroups, people have told me they envy me my position, that it's great job security. My response to them these days is this: I can live without this kind of security.

In the future, if a company offers me a job and tells me I'll be the sole developer, I'll walk away from the offer. I'll never do this again.

Some folks have claimed that it presents the great opportunity to establish your own process. In my experience, there is no process in a team of one. There's nothing in place to hold off the torrents of work that come your way. There's no one to correct you when the urge to gold-plate the code comes along. There's no one to review your code. There's no one to ensure that your code is checked in on time, labeled properly, unit tested regularly. There's no one to ensure that you're following a coding standard. There's no one to monitor your timeliness on defect correction. There's no one to verify that you're not just marking defects as "not reproducible" when, in fact, they are. There's no one to double-check your estimates, and call you on it when you're just yanking something out of your ass.

There's no one to pick up the slack when you're sick, or away on a business trip. There's no one to help out when you're overworked, sidetracked with phone calls, pointless meetings, and menial tasks that someone springs on you at the last minute and absolutely must be done right now. There's no one to bounce ideas off of, no one to help you figure your way out of a bind, no one to collaborate with on designs, architectures or technologies. You're working in a vacuum. And in a vacuum, no one can hear you scream.

The insane hours, the stress, the loss of a productive life outside of your job--it's not worth it. I'm willing to accept all that I have done to create this scenario. And you can bet that I'll be doing everything in my power in the future to ensure that it doesn't happen again.

If anyone's reading this, let this be a lesson to you. Think hard before you accept a job as the sole developer at a company. It's a whole new kind of hell. If given the chance, take the job working with other developers, where you can at least work with others who can mentor you and help you develop your skill set, and keep you abreast of current technology.

Take my advice. Don't open that door. It leads straight to Hell.

Fighting Fire with Fire

Today, yet again, someone sent me a chain letter in my email.

I loathe chain letters. I've blogged about them. They're evil. They're vile. They're a waste of time. They clog up mailboxes. They make impossible promises and fill your head with touchy-feely nonsense. I'm sick of receiving them, and I think it's rude and impolite of folks to send them around to their friends out of some superstitious and selfish sense that they're going to improve their own wellbeing. And yet, my cries to have people stop sending them to me have gone unheard, unheeded. I am still receiving them. And I have noticed that the recipient lists only seem to be getting larger.

So I've capitulated. But I've decided to do it with a twist.

Every time someone sends me one of these damned things, I'm going to send them a link to my blog post about chain letters. And I'm going to CC everyone on the list.

So now, admittedly, I've become a hypocrite. But only for a short time. When I'm satisfied they've gotten the message, I'll stop. Until them, it's on, baby. Bring it!

The Evil Wiles of the Chain Letter

My mother recently got a chain letter in her email, and was telling me how she hates it when people send them to her. Certain friends and family members are notorious for forwarding them us, and it really peeves us off, because they're a complete waste of our time. We simply delete them these days. Unfortunately, in many cases, you have to open the email before you can determine whether or not its a chain letter. And by the time you've opened it, you feel utterly compelled to finish reading it. It's like a hypnotic effect.

Once you've opened a chain letter, it's contents are like road kill: you have to look. You can't help it. You don't want to look, but you have to. Your brain screams, "It's not right! Turn the other way! Close your eyes!" But your eyes stare fixedly at it while your mouth drops open in horror and bile rises at the back of your throat. When you've finished reading it, you're left with that feeling that you just got ripped off, like you paid to see Saving Private Ryan and instead got something on the order of Killer Beach Bimbos from Outer Space.

Chain letters have even moved away from the email format and evolved into MySpace comments. (I haven't checked out sites like TagWorld or any of MySpace's competitors, so I can't really comment on that.) Fortunately, MySpace comments are easily ignored. You can just customize the page to not display them.

Now, chain letters pose a number of problems, as I see it:

1. They waste the recipient's time. No one that I know likes to waste time reading them. I'd guess that those who do are also those who like to forward them, thereby propagating them and contributing to the misery of those who don't want to get them in the first place.

2. They clog up mailboxes. Most users have a finite amount of space in their mailboxes. When your mail box runs out of space on the mail server, the mail server refuses to accept mail for you until you start clearing out your mailbox. If your mailbox is full of spam, that tends to tick you off. No one wants to waste a limited amount of space on email that serves no purpose and that they'll just delete anyway. This is especially problematic if the chain letter includes images, audio, or video attachments.

3. They tend to contain deeply nested attachments. Because they are forwarded so many times, and due to the way that many email programs forward emails, the message is forwarded as an attachment: its text is not included in the body. This forces the user to open the attachment in order to read the message, which is often a risky proposition these days, since many casual users either don't use virus software or don't keep it up to date. However, every time the message is forwarded, the message is sent as an attachment, with a new message wrapper. So the new recipients have one more layer to click through in order to get to the actual message. By the time it's been sent ten or twenty times, which happens quite frequently, the real message is so deeply embedded in nested attachments that it's a chore to reach it. This is a highly annoying aspect of these messages, and it wastes lots of time.

   The fact that the messages are included as message headers also bloats the size of the email itself. Instead of simply including the text of the body, and using a single mail header, the email is sent as a file attachment, with the complete email header plus the complete email header for the new email. Now if I send it again, I get to create an email that contains a new email header, plus a new file attachment that contains the email header and text for the email I received plus the original attachment. The file's just gotten much larger. When my recipient forwards the file, it will get larger again. This causes the chain letter's overall size when sent across the Internet to grow every time it is forwarded.

4. They are propagated by exploiting human weakness. Chain letters aren't sent out by a mindless computer sitting in an air conditioned network operations center. Nor are they sent out by some hacker sitting in a dark room in his mom's basement somewhere. They are sent out by people who get them from their friends, and think they're cute, or touching, or emotionally stirring, and forward them on to their friends. They forward them on to the people that they think are most interested in them. And so, unlike spam, which dies when it hits a user's machine and (hopefully) goes to his or her recycle bin or junk email folder, the user keeps the chain letter alive by sharing it with his or her friends and coworkers. Further, the user sends it to multiple individuals, and those individuals do the same. So the chain letter is spread like a biological virus.

5. They are nothing more than personal, selective spam. Don't believe me? Dictionary.com defines spam as "unsolicited e-mail, often of a commercial nature, sent indiscriminately to multiple mailing lists, individuals, or newsgroups; junk e-mail." Now here, by "indiscriminately" we mean that the sender of a chain letter thinks that everyone wants to get the same chain letter that he or she did. "If I like it, certainly all of my friends and coworkers want to get it." So what does that mean, exactly?

   It means that the sender is distributing spam, knowingly and willingly, albeit naïvely. People have been sent to jail and fined for that because it's illegal in certain states. Why is it illegal? Because it annoys users, and wastes bandwidth and the recipients' resources. The biggest reason, however, is that it annoys users. This article shows that the vast majority of people want spam banned simply because it annoys them.

   By definition, spam is not required to be commercial in nature. That means that all those chain letters you get that promise you happiness, the good will of angels, good luck, blessings, prosperity, the love of your life within three days, your wish coming true, the prevention of bad luck, and on and on, are all spam. Most of us just haven't viewed chain letters in that light before.

And those are just the issues I can think of right off the top of my head.

While I'm at it, it isn't just novice users that get sucked into these emails. With address spoofing you can receive an email that looks perfectly legitimate, and you can start clicking on those attachments because it seems perfectly reasonable to do so. I've done it. My mom has done it. My coworkers have done it. You've typically already opened four or five layers before you realize what you've done and think to yourself, "What am I doing? This is obviously junk!"

So please, please, think before you send that next chain letter out. Those promises they make at the bottom of them are untrue. The threats they make are untrue. You control your own destiny, and no email is going to dictate it for you. Do you think that the destiny of the people that to whom you forward the email is going to be changed because you sent it to them?

You aren't helping anyone by forwarding these things. Point of fact, you're likely annoying more people than you're helping.

Do everyone a favor. When you receive a chain letter, delete it immediately. Then contact the person who sent it to you and politely ask them not to send them to you anymore.