Knowing How is Not Enough

There’s an old adage that I heard once, and it’s stuck with me through the years:

He who knows how to do a thing is a good employee. He who knows why is his boss.

I’m also fond of this one:

If you can’t explain it, you don’t understand it.

So I’ve been ramping up on some technology that I’ve really not had an opportunity to really use before, and I’m very excited about it. To make sure I understand it, I’ve decided to go back to the MSDN examples, reproduce them one line at a time, and then document the source code as I understand it. It’s a great way to learn, and sheds a great deal of light on what you think is happening, versus what’s actually happening.

To be perfectly honest, the technology is AJAX. Over the last few years, I’ve predominantly worked for companies that haven’t had any use for Web services, so there’s been no compelling need for it. I’m starting a new job soon and it will rely heavily on Web services, and I really want to make sure I understand them well before I set my foot in the door. It has never been enough for me to know that you just drag a control onto a form or page, set a few properties and press F5. To me, that degree of abstraction is a double-edged sword.

When abstraction reaches the level that it has with Microsoft AJAX, you start to run into some fairly significant issues when it comes time to test and debug the application. The MS AJAX framework is no small accomplishment, and it hides a lot of complexity from you. It makes it so easy to write AJAX applications that you really don’t need to understand the underlying fundamentals of Asynchronous Javascript and XML that make the whole thing work. Consequently, when things go wrong, you could very well be left scratching your head, without a clue, and no idea where to begin looking.

Where, in all of this enormously layered abstraction did something go wrong? Was it my code? Was it the compiler? Was it IIS? Was it permissions? Was it an update? Was it a configuration setting? Was it a misunderstanding of the protocol? Did the Web service go down or move? Was the proxy even generated? If it was, was it generated correctly? Do I even know what a proxy is and why I need it?!

When I started learning about AJAX, we coded simple calls against pages that could return anything to you in an HTTP request using the XMLHTTPRequest object. Sure, it was supposed to be XML, but that was by convention only. The stuff I wrote back then (and I only wrote this stuff on extremely rare occasions, thank the gods), returned the smallest piece of data possible: a single field of data in flat text. It was enough to satisfy the business need, and didn’t require XML DOM parsing.

But even with DOM parsing, the code to make a request and get its data back via XMLHTTPRequest was a lot smaller than all the scaffolding you have to erect now. You might argue that you don’t have create a lot of code now, but that is just an illusion. You’re not writing it, but Microsoft is. Just because you don’t see it doesn’t mean it’s not there. Do you know what that code is doing?

In theory, the Why of Microsoft AJAX, or any AJAX library is to make our lives easier when it comes time to write dynamic Web applications that behave more like desktop applications. To a certain degree, they have. When they work. But when they don’t, I wonder if the enormous degree of abstraction they’ve introduced hasn’t dumbed us down to the point where we’ve ignored essential knowledge that we should have.

If you’re going to write Web services, or consume them, you should, at a minimum, understand what they are, and how they work. You should understand their history, how they evolved, and the problem that AJAX tries to solve. It’s not enough to know how to write a Web service, you have to know why you’re doing it, and why you’re doing it the way you are. That sort of knowledge can be crucial in making the right choices about algorithms, protocols, frameworks, caching, security, and so on.

But this could be true of any technology or practice we learn. AJAX, LINQ, design patterns, TDD, continuous integration, pair programming, and so on. Know why.

Try this simple litmus test. Explain something you think you know to one of your peers. If you can’t explain it clearly without having to pull out a reference or go online, you don’t understand it the way you think you did. Consider relearning it. It’ll only improve your value to yourself, your peers, and your employer.

Technorati Tags: Productivity,Software Development,Technology

The Perfect Villain

I play City of Heroes quite a bit. Okay, so that's an understatement. I play it a lot. After five years in the game, my addiction runs so deep it's threaded its way through my genetic makeup. You'd be hard-pressed to separate me from it.

I've always had a love of comic books. What's not to love? Sure, they're the domain of geeks, nerds, and all their ilk. But let's face it: who wouldn't love to fly? To have virtually god-like power at their disposal? To have a body so ripped you wouldn't think twice about showing up in public in a spandex body suit?

But one thing has started to really bug me about the villains in my game of choice. Now, I laud the writers for COH. As the game has aged, the stories have grown more compelling, more twisted, more convoluted. Sure, we have some dangling plot threats, gaping plot holes, and all that jazz, but they've fleshed out some of the characters quite nicely. In a world full of super-powered beings, that's frequently no mean feat. It's not easy fleshing out characters in a story about normal humans; try fleshing out a character who can incinerate you with a look, or teleport across vast distances, or summon the dead--and justify their reasons for doing what they do. That's a lot easier said than done.

The budding writer in me understands their desire to do just that. I think that the armchair psychiatrist in all of us would tend to agree that any villain who embarks on a life of crime does so for a reason. Greed, revenge, betrayal, power, lust, all the standard reasons are there.

But I've watched those foundation villains in City of Heroes for some time now, and I've discovered something about them that disturbs me.

They are all too distracted.

Give me any villain from the COH Universe and I can likely show you how that villain is too distracted. They all lack focus. A clear vision of what it is that motivates them. What Paragon City, indeed the COH Universe, needs is a villain who is so absolutely single-minded that he (or she) is utterly unstoppable and incapable of being distracted.

Imagine, if you will, a villain whom you never see. He is adept at keeping his identity and location concealed from everyone, including those with psychic and magical abilities. He does not want money, power, fame, or legions of followers. What he does want is to watch people suffer. He does not care, particularly, how he does it. He just cares that the world is filled with pain, anguish, misery, and death, and that it was his hand that brought it to pass.

He is not a god. He is not inhuman. He is not even super-powered. He is simply intelligent, and possessed of a single-minded determination to bring mass chaos to the world around him. He loathes the other villains because they are weakened by the petty squabbles they engage in between themselves. He will kill heroes, villains and civilians with merciless abandon.

While he has no followers of his own, there are those who have heard of him, and rally to his cause. These copy-cats seek to emulate him. They spread his anarchist agenda with a ruthlessness on par with his own. They could be anyone: civilians, members of known factions, heroes teetering on the brink of villainy. But none of them knows who he is or has ever seen him.

The only reason that anyone knows he exists is because he occasionally forewarns people of upcoming tragedies he will inflict, only to incite more terror. His mastery of indirection will sometimes lead heroes to one site in an attempt to avert disaster, only to realize that his real target was some other heavily populated target.

His targets of choice are always large population centers: hospitals, police departments, large office buildings, schools, and so on. Given any choice where he can inflict damage, he will inflict as much damage as is humanly possible.

Yes, this is the kind of villain that the COH Universe needs. A Black Mask, completely unpredictable, untraceable, never brought to justice, enemy to both hero and villain, and incapable of being distracted from the one thing that he seeks the most: the pain and suffering of others.

I highly doubt that this sort of a villain could be brought to pass and maintain the T for Teen rating that the game enjoys in its current incarnation. But one could dream. And if you're going to dream, dream big.

Technorati Tags: City of Heroes

On Legacy Software Maintenance

In the mad, mad, mad, mad world of software development, we are faced with the trying task of maintaining legacy systems. In an ideal world, that wouldn't be the case. We'd all be developing brand new systems from the ground up, writing ground-breaking code that no one has ever seen before, without the hassles that arise when you have to worry about things like backwards compatibility and maximum system uptime.

But this isn't an ideal world. The vast majority of us don't have the luxury of developing completely new systems. Instead, our lives are fraught with the perils of correcting defects and adding new features to systems that have been around for ages and, occasionally, decades. Those systems have usually passed through a number of hands and they tend to be poorly documented. They sometimes have sprawling feature sets, support technologies that have long since fallen by the wayside, bloated with code that doesn't appear to be invoked by anyone, and riddled with obscure and seemingly nonsensical comments.

Your job, as a maintenance developer, is to massage that seemingly horrific beast into a thing of beauty. Real people doing real jobs depend on it to get their work done in a timely manner. As much as we might loathe an ancient codebase, the language it was written in, or the tools we have to use to get the job done, truth is, a legacy application is maintained for a reason: it has intrinsic value to the bottom line of the business. When the folks who depend on that system can get their jobs done on time, in a productive manner, they can continue to draw a decent paycheck. That means that they can continue to pay the rent, put food on the table for their families, afford healthcare, and all those other essentials.

So tread lightly when you delve into the code of a legacy system. It's far more important than you think. We just take it for granted that it's a dirty, loathsome job and someone has to do it. We just happen to be the unlucky bastard who drew the short straw. Not so: you happen to be the lucky one who drew that straw. People depend on you to help them keep their families safe, warm, and well-fed.

My point isn't that every legacy application is manna from heaven. My point is that legacy applications exist for a reason, that they're maintained for a reason. They have long, varied histories for a reason. They have endured because they have value; they've grown beyond their original specification because the company sees real value in them, and doesn't want to lose that investment. The problem for you, as a developer, is in ensuring that you do not destroy that investment.

When we are first introduced to a legacy system, we have a tendency to look at the source code and view it as though it were written by a blind deaf quadriplegic with Tourette's syndrome. No one in their right mind would have written a system that way. What could they possibly have been thinking? You certainly wouldn't have! I certainly wouldn't have.

But then, over time, we start to learn things about it. The code is old; very old. There's been a high turnover rate, and the code has passed through lots of hands. The companies that published third-party components went out of business when the dot-com bubble burst. They used to use Novell for security, and then switched to Active Directory. When this thing was released, Windows 95 was still popular. They upgraded the database about two years ago, and had to make some emergency revisions to the code.

There are reasons that things like this happen in a legacy system that's old enough to qualify for Medicare. Many of those reasons are valid, and many of them are the product of a lack of time and budget. Sometimes, sadly, it's a result of a lack of skilled developers (but that's something for someone else to blog about). The point, in short, is that systems grow from an original vision into large, cumbersome, bloated systems because developers respond to the needs and demands of the business.

Now, here you are, present day, and you're tasked with maintaining that source code. You have two primary responsibilities: 1.) Fix the defects, and 2.) Add new features. Keep in mind that while you are doing both, you must not at any point in time break backwards compatibility or bring down the system. People rely on this system. It's the bread and butter (or part of it) of the business. And it's your baby now.

It is absolutely crucial that you treat legacy software with its due gravity. If you view it like it's some recurring annoyance, stop that. If you leap to hasty conclusions about the cause of problems in the system, stop that immediately. This is a system that many people rely on. Get it into your head that you need to treat this thing delicately, as if it were your pride and joy. Once you fix a defect, once you put a new feature into it, your name is associated with that software. Take pride in it. Do it right. Take your time.

Over time, the legacy system stops being the horrific beast associated with all those who preceded you. It becomes the creature that you have molded it into. And then, people will associate it with you, for better or worse.

Technorati Tags: Quality, Software Development, Legacy Software, Software Maintenance

The Sad State of SQL Refactoring

I love refactoring tools.

The ability to select a variable in code, right-click on it, choose the Rename command from a context menu, and then safely rename that variable in every location in which it appears is a boon to my programming productivity. Similarly, the ability to take a large block of complex logic and extract it to its own method is really handy. Or, the ability to reorder parameters, and know that every piece of code that calls that method will be updated appropriately saves me tons of time and improves my confidence in the quality of the code.

When you refactor code of any kind, you enhance its readability without changing the way it works. But refactoring by hand is frequently difficult, tedious, and error-prone. That's why refactoring tools exist and why the essential service they provide is important. They allow us to improve the maintainability of code, hopefully reducing maintenance costs, quickly and efficiently.

Enter SQL languages.

In theory, there is a standard for SQL languages: the ANSI SQL-92 standard. One would like to think that it would be a simple matter to create refactoring tools for any SQL based on the ANSI standard for SQL. One would be wrong. You can't use the standard as the sole basis of a refactoring tool.

Any given database vender wants to strive to make their product unique, to stand out from the crowd. And so, they don't entirely conform to the standard. They have additional features that separate them from each other. For example: Oracle organizes functions and procedures into packages. Microsoft does not. Microsoft allows a bit data type on columns. Oracle does not. And, although we all loathe to think of Access as a real database, Access provides a boolean data type that you can use in columns, while Oracle does not.

Now, let's also talk about legacy support. Oracle has been around for a very long time. Version 2 came out in 1979, from what I gather. Their legacy join syntax looks nothing like the ANSI standard (and that's not a criticism, just a simple statement of fact):

SELECT Customers.CustomerId, Company, OrderID
FROM Customers C, Orders O
WHERE C.CustomerID (+) = O.CustomerID
UNION
SELECT Customers.CustomerId, Company, OrderID
FROM Customers C, Orders O
WHERE C.CustomerID = O.CustomerID (+)

In ANSI SQL, this is :

SELECT Customers.CustomerId, Company, OrderId
FROM Customers C FULL OUTER JOIN Orders O ON
(C.CustomerId = O.CustomerId)

Now, we could add in all the different SQL variations that we know that are out there:

• ANSI standard
• Interbase/Firebird
• IBM  SQL
• Microsoft Transact SQL
• MySQL
• Oracle PL/SQL
• PostgreSQL
• Access
• FoxPro

...and on and on and on, but you get the point. There are a lot of SQL variants out there. And their goal is to accomplish the same thing:

• Create a data store.
• Occasionally, modify the structure of the data store.
• Get data out of a data store.
• Put data into a data store.
• If supported, execute code within the confines of the data store and (optionally) return a result.

Behold, SQL in a nutshell.

Over the lifetime of these various products, they have added features that they have to maintain for legacy support. Somewhere out there there's a business that absolutely must have that feature in place or their whole process will come crashing down. Don't you dare remove it. It doesn't matter that there are better features (likely based on a standard); there are applications out there for which they don't have the source code anymore, or which no one understands, and they're too afraid to touch.

Now, all these reasons have led us to a scenario where we have vastly different implementations of SQL. Sure, they share a lot in common, but they also have radically different feature sets and syntax. And that situation, in and of itself, has led us to one frightening and sad conclusion:

We will likely never have a tool that is able to connect to any database and be able to correctly refactor its SQL.

And that's just a damned shame. Because there's a lot of SQL out there. Stored procedures, functions, views, triggers, even inline SQL in applications and all that other jazz. But the amount of work it would take to get us to a point where a refactoring tool could recognize any variant of SQL and correctly parse it, refactor it, and not hose the code is enormous.

It's a pity, really. I could see tons of use for a tool like this. In my own office, we work with SQL Server, Oracle, and Atomix databases. What I wouldn't give for a tool that could refactor SQL to enhance its readability without changing the way it worked.

And who knows? Down the road, we may be working with something else.

But this is the world we live in. If we want to refactor SQL, it looks like we'll have to settle for separate refactoring tools for each language. And then each will come with its own quirks. That might be good or bad, but it's likely the best we can hope for for now.

 

Technorati Tags: SQL, Best Practices, Productivity, Code Quality, Refactoring, Software Development

The Absurdity of "Don't Reinvent the Wheel"

As developers, we've had this adage drilled into us from the beginning: Don't reinvent the wheel. In short, don't rewrite what's already been written. The idea is sound, in theory. You can save yourself time and money if you'll simply reuse existing code and/or components rather than writing them yourself from scratch. This time and money is saved up front when you write it (or would have written it), and down the road, when you have to maintain your system.

However, I'd like to point out another, equally applicable adage: There's nothing new under the sun. Anyone who's ever tried to write a novel, a short story, a play, a movie, a song, or a piece of software, will know this one simple truth: somewhere, at some point in time, it's already been written.

Every algorithm, every piece of code that we will ever attempt to write has already been written somewhere, at some point in time, by someone. Only the names have been changed. You're not inventing anything that is completely new, that's never been seen before. You should wisely disabuse yourself of that notion as quickly as possible.

In the grand scheme of things, at the application level, you may very well have an idea for a system that is unlike anything that has been done to date. But the algorithms that drive it have already been written. Bubble sorts, hashes, exception handlers, encryption, data access, socket management, shopping carts, entire application frameworks, date management, document management, serialization, port I/O, and all that other stuff has already been done. Further, it's already been done several times over in many different languages to varying degrees of success.

Tragically, if you're using a large application framework, like Java's EE or Microsoft's .NET, the chances are good that the functionality you're looking for is built right into the framework itself. The problem is that the framework is so vast that you'll spend more time looking for it, and determining whether or not it works the way you need it to work than you would just rewriting it yourself.

Application frameworks are stunningly afflicted with feature creep. They must do everything under the sun, must meet every possible need. The problem, then, is that their scope becomes so broad, so vast, that no one in their right mind could possibly grasp the totality of all that they can do. It is inevitable that anyone using them will reinvent some of their functionality. The scale of that functionality might be small (reformatting dates) or it might be substantial (pooled database connections).

In the end, it's absurd to think that we can possibly avoid reinventing the wheel. Of course we're going to reinvent it. Every application we write is a reinvention of someone else's wheel. It just so happens that our wheel is a custom wheel. All this paranoia about reinventing the wheel is blown out of proportion. A proper buy vs. build decision should never be neglected; but don't ever think for one minute that what you're creating hasn't been created before.

Consider the scenario where you're under the gun to get a product out the door. And I mean it's a really tight schedule. And don't act like it's a perfect world, and you have leverage over the schedule. This is reality here. In the real world, the customer controls the schedule, because it's tied to when the product is released, and that's tied to this big, huge monstrosity in another state or another country. The product's delivery schedule is a train barreling down the track at 120mph and no one short of God can stop it. Now, you have a very finite amount of time to work in. You need an algorithm. You know you could write it. Or you could look to see if someone else has written it.

If you do the whole Web search thing, you have to ask yourself a few questions: Is it from a source you trust? Is it in the language you're using, or do you have to convert it? Does it work? Does it need to be tweaked? If any of these fail, you're back to the drawing board. Time's wasting here, and that train's getting closer to its destination. If all the answers pass, you have to make sure you don't run into any copyright or licensing issues with that code. (You are paying attention to that, aren't you?)

If you decide to peruse your application's framework, you'd better hope it's well documented, and very easy to search. Good luck using the search features in .NET. It's not like the ASK.COM interface, where you can ask, "How do I convert a date in DOD format to Gregorian format?" Yeah. Good luck with that. On the other hand, you could ask your coworkers. They might know. Then again, they might not. If they don't, you're off to Google to get the information. Here's hoping you get a timely and accurate response.

Sure, this is an extreme example. But it makes my point: At some point, the work has to get done. You can't afford to spend days or weeks scratching your head about whether or not that wheel's already been invented. Believe me, it has. The problem is, there are a countless number of wheels, and none of them are labeled, and you don't know where to find the wheel you're looking for.

Stop wasting time, and invent your own damned wheel.

After all, whatever code you might reuse, is just someone else's reinvention of the same wheel.

My First Time: Software Passion

I remember when I was first bitten by the computer programming "Bug."

I was young: still in high school, in fact. At my particular high school (in Fontana, California), we had an Indian Education Department. That small department was lucky enough to have a TRS-80 computer for the kids in the Indian Club to play on. I dare say that the computer itself drew a few kids to the club; we were misfits, outcasts, by and large, but we were drawn to that thing like moths to a flame.

It was nothing to look at really. It was just an old monochrome monitor, with a keyboard, and a tape drive. Our model didn't even have a floppy disk. Everything was on tapes. But we were mesmerized by that thing. I remember watching one of the other kids, Dan, fire up ZORK, and typing, "go north" into the computer. It responded to his simple command, and described the next room to him. It amazed me. I remember sitting there and thinking, "How did they do that?"

I mean, it was just a stupid box, with a keyboard and a cassette drive. It couldn't think. But there it was, responding to him as if it could think. And he could type commands that were, for the day, fairly close to English. "Eat food." "Quaff potion." "Open door."

I remember sitting there, thinking about that and being relentlessly tormented by it. I had to know. How could an inanimate box like that do things like that? How did it know what room he was in? How come the rooms changed every time we played the game? How did it decide if the potion killed him, healed him, or made him sick? How did it decide what color the potion was? How did it decide what was in the room? For a stupid box with no brain, this thing was pretty damned smart.

And then, one day, Dan got stumped by the game. Something, apparently was wrong. A few years later, I'd realize he'd found a bug. So, he fired up a program, and cracked open the game's source code. And there, before my eyes, was the big secret. It was line after line after line of source code: carefully written instructions that told the stupid box exactly what to do. From those cryptic instructions, written in some obscure language called BASIC, you could make that TRS-80 do amazing things!

That was the beginning of the end for me. I had to master that language. I had to know how I, too, could command a stupid, brain-dead box and make it do amazing things. It wasn't long before I had obtained a copy of the language reference for BASIC and taken a computer programming course at our High School. (Yes, we had them, even out in the sticks in Fontana.)

So, in a way, ZORK made me a programmer.

It's been about twenty-five years since I watched Dan crack open the source code to ZORK, and the path of my life was irrevocably altered. Up until that point, I didn't really have any real aspirations. I don't think I really did after that, either. But one thing became very clear: more than any other endeavor to which I applied myself, computer programming proved itself to be my one enduring passion. All these years later, I still have those moments reminiscent of that first day. I'll see a beautiful piece of code, a website design, or an application, and I'll think, "How did they do that?" Any ideas I may have had about leaving software development will be blown away and my passion for software will be rekindled.

It's because I have to know. I can't walk away from these damned stupid boxes without being able to make them do amazing things.

There's a certain, childish delight in figuring out the solution to a problem, or finding a new way to do something. For me, it's like Christmas, and I want to share that joy with others. Sadly, a lot of folks don't understand it--especially if they're not in the same field. But anyone who's done this work, and ever had a EUREKA! moment knows exactly what I'm talking about.

Somewhere, right now, a budding young developer is experiencing his or her first time. They're being bitten by the bug. It's an infection that will take hold and set in for life. For most of us, it's a turbulent ride, filled with ups and downs, and we frequently consider leaving the field. For others, it's pure hell, and we leave it too quickly; for a lucky few, it's nirvana all the way through. I'm not sure I envy the lucky few; I rather like the way my challenges have tempered me over the years.

When you face challenges, think back on what it was about software that caught you in the first place. Think back to your first time. Then think about the many times you've been lured back to it by your own passion. Not because someone offered you money, or material goods, or power, or prestige; think back to those times that your personal passion for software kept you in the game. Then ask yourself why you feel so passionate about software. The answer for me was surprising: I'm not really doing it for anyone else, but because I have to know, and because I have to conquer the stupid box.

For all my noble aspirations, that's a humbling admission.

But that passion is still there. It keeps me in the game. And, in retrospect, it's likely why I feel so passionately about software quality. It's not enough that it works, it has to work well.

What was your first time like?

You are Not the Average Computer User

John Lilly, the CEO of Mozilla, recently blogged about Apple's practice of including a new installation of Safari in Apple's Software Update service, even if you didn't have the application installed in the first place. You can read the full article here. His main point was this: As a matter of trust, update software should update previously installed applications, and not install new applications. Apple pretty much violated that trust when it presented users with this handy little dialog box:

The main issue here is that Safari is not already installed on the end-user's machine. So, the option is not an update, but a fresh download of brand new software. Further, the option is checked by default, and the button in the lower right hand corner clearly says "Install 2 items".

Now, I'm not going to rehash the pros and cons of Apple's tactics in this matter, because that argument has been debated endlessly on John's blog and on Reddit. What I am going to take issue with is the arrogant presumption that many commenters take when they make these sorts of statements:

"I don’t see what the problem is here. If you don’t want the software, you uncheck the box. The product description is listed very clearly in the window, no extra clicking required."

Omar

"I don’t see the big deal. They are promoting their software through their software update program. It’s automatically checked…ok, so? Lots of update programs automatically check everything anyway, not just apple.

"If FF is better then people will use FF. If they like safari then they will switch. These browser “loyalty” wars are getting old. IE came with windows by default and FF is still gaining ground. It is gaining ground because it is better. Just keep making a better browser and stop worrying about this. ppl will flock to the best. We’re not stupid."

Chris

"Oh fer heaven’s sake, uncheck the box and get over it. Are you saying the majority of Windows users of iTunes are too clueless to look and see what they’re downloading? OK, I’ll admit it’s a bit pushy of Apple but beyond that I fail to see what all the fuss is about."

Anne

These are knee-jerk responses. The last one, in particular, is an exemplary case of a poster who clearly doesn't understand the idea that users who read or post to tech blogs or forums are not typical computer users. If you're reading this blog, you're not a typical computer user. (I'm not sure what you are, exactly, but you're not typical.)

Apple's case is interesting because of the enormous success of the iPod, and the vast number of iPod owners who use Windows. Those users will download iTunes so that they can use their iPod with their computer to purchase music and manage their playlists. However, the vast majority of those people are not what we would classify as tech savvy users. Rather, I'd call them click-through users, who implicitly trust the software vendor to make decisions for them. Think about your mom, your dad, your sister, your brother, your aunt, your uncle, the kids at school, the clerks at the nearest retail outlet or fast food joint, your fellow students, or your nontechnical coworkers.

Those people represent the average computer user. They are click-through users.

A couple times a year, I get calls from my family members about their computers. Inevitably, they'll tell me that the computer is suddenly horrifically slow, and that they need me to fix it. So they bring it to me, and I look at it, and it has tons of mystery software on it. I like to have them sit with me when I'm going through it, so that I don't remove anything that they might actually need or use. Nine times out of ten, they'll tell me, "I don't know where that came from." Apple's software update for Safari is likely going to produce an awful lot of these scenarios, because the the average computer user will have just clicked through the dialog, trusting that Apple knew what was best for them.

A tech savvy user isn't likely to just click through that dialog box because they know what can happen, and they're pretty darned picky about what goes on their machine. They don't blindly trust the vendor to make those decisions for them. But the number of users like that is relatively small, and is hardly representative of the world's population.

But the world is full of click-through users. There are far more of them than there are of us. Thinking for one minute that everyone thinks and/or behaves as we do is naive, shortsighted, arrogant and presumptuous.

Again, my point here isn't that Apple was right or wrong. My point is this: never assume for one minute that YOU represent the average computer user. You don't.

• If you're smart enough to competently read or post an a technical blog or forum, you're not an average computer user.
• If you know how to correctly fix someone else's machine after they've borked it, you're not an average computer user.
• If you know the difference between a hash table, a binary tree, and a linked list, you are not an average computer user.
• If you know what recursion is, you are not an average computer user.
• If you know how to safely overclock your machine, you're not an average computer user.
• If you read technical books like they're gripping, fast-paced murder mysteries, you're not an average computer user.

This list is undoubtedly incomplete, but I haven't had enough coffee yet. But you get the point.

So, enough with this arrogant presumption. Stop assuming that all users behave as we do. Because the simple truth is that the vast majority of users do not behave or think as we do. They trust; we suspect.

Certified at Last!

Okay, so it's not my Microsoft certification yet. But it's something:

 

Being a big fan of measurable, quantifiable scores, I'd say it's nice that when someone asks, I can give them hard numbers to justify my claims when I say "Yes, I am, in fact, a nerd. Thank you very much for asking."

This ought to look really good in my email signature:

Michael Hofer
High Nerd
One More Pointless Blog

I can't wait to use it!

Heath Ledger Has Died

I was stunned to read that Heath Ledger has passed away while cruising the City of Heroes forums.

I'm stunned. As many know, Heath had just wrapped up his portrayal of the Joker in The Dark Knight. His passing will make it very eery to watch him on the screen when the movie comes out.

For those who don't know much about him, check out his IMDB profile. Heath wasn't just another pretty face around Hollywood. He picked his projects carefully to avoid being pigeonholed. He'll be missed.

Heroes, Villains, and Software

I'm a big fan of comics, and I'm also a big fan of software development. Coincidentally, I see metaphors and analogies for software development just about everywhere I look.

Comic books have come a long way from the days when it was just about the guy wearing the mask or the brightly colored costume who rushed in to save the day. In the early days, the stories were pretty straightforward: the bad guys were always easily identified, and so were the good guys. But stories these days have become more complex, more adult, and they tackle tough issues. But one common element persists: the heroes and villains tend to wear costumes or masks that exemplify their heroic or villainous persona.

The attire of a hero or villain isn't simply done for theatrical reasons (although there is certainly a large and undeniable element of theatricaility to it). The mask grants the wearer anonymity; those who behold the wearer have no idea who the wearer is, and with that anonymity the wearer is emboldened to do things he or she might never do or have the courage to do if people knew who he or she was.

Anonymity grants the wielder power. What separates the heroes from the villains is a sense of responsibility and what they choose to do with it. A villain might have a sense of responsibility but choose to ignore it, and use his power for personal gain.

When a civilian in a comic book views a mask-clad individual for the first time, they don't really know whether that individual is a hero or a villain, or just some bloke in a costume. They might be able to infer some things from the costume, but they have to trust their instincts. A great failing of classic villains seemed to be that they went out of their way to look villainous; they were so easily identified that they were easy to avoid or locate and defeat.

But the greatest villains actually looked like one of the good guys, or just a regular Joe. They managed to garner the trust of those around them, and then used that trust against those who placed it in them. You see, sometimes, wearing no costume at all is a mask in and of itself.

So now we come to the part where we apply this to software.

All software wears a mask. It's called the user interface. It doesn't matter if it's a desktop application, a Web application, a framework, a Web service, a device driver, or anything else. The mask is the face that your software presents to the public, how the public perceives the software when they first look at it.

Most software presents itself as the next silver bullet to any given problem domain. After all, what software doesn't want to be perceived as a Hero? A user, looking at your software for the first time, is forced to place a certain amount of trust in it because all they can see is its mask. And the mask grants the software anonymity and power.

Unless you're giving the user the source code, and the user knows how to read it, and can build it themelves, they're at your mercy. They're trusting you to be heroic on their machine: to have a sense of responsibility, and to use the power you've been granted wisely.

Sadly, much precompiled software relies on its anonymity to conceal its true state: that it is miserable, ugly, unmaintainable, untestable, bloated, unreliable, and unsafe. Software of this nature is hardly heroic; rather, it's villainous. It might handle user data with reckless disregard for integrity, security, or consistency. It might crash frequently, and for no apparent reason. It might install spyware or adware. It might log keystrokes. It might be riddled with bugs, be poorly documented, and have no consistent coding model or standard. It might use an inconsistent user interface paradigm, or hog system resources. It might break other software. It might do all of these things.

The question we are compelled to ask is this: What is hiding behind the mask of the software that we're writing, right now, at this moment? Is it heroic? Or is it villainous? Despite claims by certain advertisers, it isn't really enough that "it just works." It has to work well, and responsibly.

Some of the greatest heroes in comic book lore were brave enough to take their masks off so that people could see them and know who they were. Would you be willing to open your code base up to let the world see what you're doing? Would you feel comfortable letting everyone see the code that you write and what it's doing? Would you be confident or proud of your work? Or would you feel shame? Embarrassment? Even guilt?

Not all software can be heroic. There are always mitigating circumstances. But we can all strive to be heroic. It's when we strive to be villainous that we need to be worried. Or when we're writing villainous software, and we know it, and we choose not to do anything about it. We all have a choice. We can choose to make great software that behaves responsibly, and does what it says its going to do, or we can abuse the trust placed in us and serve ourselves.

Take stock of the software you're writing. Find out what's behind its mask. If it's not software you'd be prepared to unmask in front of its users (aside from intellectual property issues), chances are, something somewhere has gone horribly awry. What are you going to do about it?

A Terser Visual Basic?

There’s a rather interesting post going on at Panopticon Central right now, dealing with the terseness of Visual Basic.NET. In a rather obtuse attempt at reducing the “terseness” of the language, Paul Vick hacks a VB.NET compiler to coerce all the keywords to lowercase. While it’s a nice look, it doesn’t solve the common complaint that Visual Basic is inherently verbose.

Visual Basic’s verbosity problem likely stems from the fact that we VB developers commonly have to tell the compiler lots of things that it already knows or should be able to tell from the syntax. To see what I’m talking about, let’s take a piece of the original NValidator code that was written in VB and see what it looks like:

Namespace Validation

  Public Class ConnectionValidator
    Inherits ParameterValidatorBase

    Friend Sub New(ByVal connection As IDbConnection, ByVal name As String)
      MyBase.New(connection, name)
    End Sub

    Public Function IsClosed() As ConnectionValidator
      If Not Connection Is Nothing Then
        If (Connection.State And ConnectionState.Closed) = 0 Then
          Throw New ArgumentException("Operation requires a closed connection.", Name)
        End If
      End If
      Return Me
    End Function

    Private ReadOnly Property Connection() As IDbConnection
      Get
        Return DirectCast(InnerValue, IDbConnection)
      End Get
    End Property

  End Class

End Namespace

Now, it bears noting that every blue word in there is a keyword. That's a lot of keywords. So yes. Visual Basic is verbose. I suspect that the Visual part of Visual Basic has ceased to be "visual" as in "visual design" and has somehow become "visual keywords." But I digress.

First, lets give Paul the benefit of a doubt. Let’s see what it looks like with everything converted to lower case.

namespace Validation

  public class ConnectionValidator
    inherits ParameterValidatorBase

    friend sub New(ByVal connection as IDbConnection, byval name as string)
      mybase.New(connection, name)
    end sub

    public function IsClosed() as ConnectionValidator
      if not Connection is nothing then
        if (Connection.State and ConnectionState.Closed) = 0 then
          throw new ArgumentException("Operation requires a closed connection.", Name)
        end if
      end if
      return me
    end function

    private readonly property Connection() as IDbConnection
      get
        return directcast(InnerValue, IDbConnection)
      end Get
    end property

  end class

end namespace

Well, it does look a lot more like C#. And yes, the lower case does help to reduce the visual WHAM! of the keyword volume. But it’s the same number of words, so we haven’t addressed the terseness. To address the terseness, you actually have to reduce the number of words. So how do we do that? You get rid of the words that aren’t actually doing you any good.

First and foremost, get rid of all the words that accompany the End keyword. They’re superfluous. With proper indentation, that’s pretty obvious. So in this case, that’s End Namespace, End Sub, End Property, End If, End Function, and End Get. Believe me, if the End doesn’t have a corresponding block statement, the compiler will let you know.

Next comes the As keyword. Let’s accept the fact that identifiers can’t have spaces in them. Therefore, anything that follows an identifier in a variable, property, or method declaration must, by definition, be a data type. The As keyword isn’t getting us anywhere. Drop it.

The Then keyword on If blocks is equally useless. It needs to go.

The Sub, Function, and Property keywords are completely useless. They’re used to tell the reader whether or not a method returns a type. They certainly don’t do anything for the compiler: the compiler gleans this information from the method’s signature (it’s arguments and return type). If the method has a getter and a setter, it’s a property. So these could be removed as well.

Finally, the ReadOnly keyword is useless on our property, since it doesn’t have a setter on it. So let’s get rid of that, too.

Okay, I think that’s enough for now. We’re off to a good start. What does the fruit of our labor look like?

namespace Validation 

  public class ConnectionValidator 
    inherits ParameterValidatorBase 

    friend new(byval connection IDbConnection, byVal name string) 
      mybase.New(connection, name) 
    end 

    public IsClosed() ConnectionValidator 
      if not Connection is nothing 
        if (Connection.State and ConnectionState.Closed) = 0 
          throw new ArgumentException("Operation requires a closed connection.", Name) 
        end 
      end 
      return me 
    end 

    private Connection() IDbConnection 
      get 
        return directcast(InnerValue, IDbConnection) 
      end 
    end 

  end 

end 

So there you go. Now you’ve addressed the problem of the verbosity. But the result doesn’t really look all that much like Visual Basic anymore. It’s almost a different language.

You could likely get around all this by making all the keywords we removed optional, but what’s the point? A large body of the folks who use Visual Basic have an ardent devotion to keeping the language just as it is; messing with it in any way runs the risk of stirring their ire and starting a war of religious proportions. Then you have to consider the problems that arise from one developer who uses the keywords, and one who doesn’t, and both of them working on the same code base.

And what happens when those old keywords become the goto of the 21^st century?

Just leave it alone. If you want a terser version of Visual Basic, then make a new language. Personally, I’d love to be able to make the keywords lowercase, and not have to pamper the compiler any more than I have to. But I’d rather not waste any more time than I have to debating what’s sure to become a religious war when people realize what’s really involved in addressing the terseness of Visual Basic.

It’s not about the case of the characters. It’s about eliminating the keywords that don’t help the compiler or the developer. And the end result of that effort is a radically different language.

Why Does Software Cost So Much?

So my sister-in-law recently discovered that the computer a "friend" built for her has an illegal copy of Windows XP installed on it. (Big surprise, that.) When she asked me what she should do about that, I told her, quite simply, that she'd need to get a legal copy of Windows. When I told her what it cost, she was flabbergasted. She looked at me with disgust on her face and demanded, "I don't get it! Why is software so expensive?"

I looked her dead in the eye and said, "Do you really want me to explain it to you?"

Now, she knows that I write software for a living, and she still, in her naiveté, had the temerity to ask that question of me. She's well aware of how constantly stressed out and tired I am from the work I do. So she backed up a second, swallowed, and said, "No, I guess not."

In hindsight, however, I think she posed a fair question. I hear it alot. It's probably a fair statement that most folks have no real grasp of why software costs as much as it does. So I will try to point out some of the reasons why I believe that it is. These are my opinions, mind you; your mileage may vary.

Software is expensive for one very important reason: the people who make it have to live, pay their bills, and feed their families. Beyond all other reasons, that's probably why it's so darned expensive. It's a job, just like any other. And, like any other field, it's not just a simple task: it's a very complex task that requires lots of people, lots of resources, and lots of time.

(There seems to be this preconception out there that writing software is easy because using it is easy. It's just a game, right? How hard can it be?)

I can't speak for the open-source community who do everything over the Web, and develop software on a volunteer basis. But then again, this article doesn't necessarily apply to them. But certain things must be in place in order to develop software in a corporate environment:

• You have to have somewhere to do it. If you're a freelancer, working out of your house, you have rent or a mortgage to pay. On the other hand, if you're a business, it means you need a lease.
• For businesses with a building, you need insurance, so that you're covered in case your employees are injured on your premises. You need flood and fire insurance as well.
• You have to pay utility bills (heating, cooling, electricity, water, etc.). 
• You have to pay your local, state and federal taxes.
• You have to have office equipment. Depending on how many people are involved, you may need LOTS of it: desks, chairs, bookshelves, computers, printers, fax machines, extra toner, phones, paper, notebooks, file folders, filing cabinets, trash cans, pens, and all that other jazz. You'll probably need whiteboards for collaboration. If you don't think that stuff is expensive, start browsing your local office supply store web sites. You'll be in for a big surprise.
• You have to employees. Employees get paid. Depending on how complex your product is, it can take anywhere from a handful to hundreds of employees to develop your product.
• You have to provide your employees with benefits (medical, dental, 401k, workmen's compensation, etc.), unless they're contractors. If they're contractors, they have to cover their own benefits which means they get paid more than W-2 employees.
• You have to have lawyers to ensure that your intellectual property is protected from copyright infringement.
• You have to spend money and invest time to develop a plan for what the product will do. This is often a long, arduous process that involves a lot of arguing, shouting, frayed nerves, and confusion.
• You have to spend money and time creating mock-ups of what it will look like. This mockup goes through several revisions as people heatedly debate why they hate it, what they want changed, and what a crappy job you did in the first place. They'll want to know why you didn't understand that by "circle" meant an ellipse and that by "square" they meant a beveled rectangle with a shadow in the lower right-hand corner. They'll send you back to the drawing board. Repeatedly. What one user demands another will despise. No one will ever be completely satisfied.
• You have to spend money to finally start developing the actual software. You'll do this with a severe shortage of time, money, and hands, and the developers will want to kill the management. They'll be unhappy because the specs will likely be hastily put together, the time will be insufficient, and they'll be grumbling about the likelihood of "crunch time scheduling" that requires overtime that they won't be getting paid for.
• You have to spend LOTS of money testing the solution once it's put together. This is the hardest part of the whole thing. You will have made the mistake of thinking that it works. It doesn't. It never does. Just when you think you know what you're doing, a user is right there to prove you wrong. The system will have to be ripped apart, rewritten, retested, over and over and over again. And all of that takes time. An adversarial relationship will develop between the testers and the developers. Finger-pointing will rule the day. Some people will quit, costing you money as you have to replace them unexpectedly and retrain them. The schedule will slip, costing you further. And as long as you're doing that, you're paying for rent, utilities, taxes, payroll, benefits, office supplies, insurance, and everything else that goes with it.
• You have to spend money to develop a marketing campaign for the product.
• You have to spend money to package the product (put it on disk, print the materials that go inside the box, put it in the box, shrink wrap it).
• You have to spend money to ship it to the retailers who will carry it, or negotiate deals with online resellers who will carry it for you.
• You have to employ a customer support staff after the product ships. (Benefits, payroll, etc.)
• You have to pay to support the customers when they call to complain about the product after it ships, or when they can't figure out how to install it or find the icon to start it up or find the CD ROM drive on their computer. You have to deal with irate customers who think it's your fault that the software has too many features when they're the ones that demanded them. You have to deal with customers who are angry and seem to think that it's okay to scream at you because they made a mistake and spilled soda on their keyboard, or the dog yanked the laptop onto the floor, or they didn't pay their cable bill and they want to know why your Web-based software doesn't work anymore. And you have to smile through it all while the bills keep piling up.

These are just a few of the reasons that software is so expensive.

And let's bear in mind one very important fact: writing software is not like writing an essay, or building a house, or fixing your car. Writing software is damned hard to do. Hell, I seriously suspect that disarming bombs is easier.

From where I sit, the computer is just a stupid box armed with the vocabulary of a three year old. It will do exactly what you tell it to do, very quickly, and very efficiently. If you tell it to do the right thing, it will do it very quickly, and very efficiently. Tell it to do the wrong thing (like, erase all files in the My Documents folder) and it will do it very quickly, and very efficiently. This is why I get irritated when someone says, "Oh, the computer hiccupped!" or "The computer screwed up." No, the computer did not. The computer did exactly what some programmer told it to do.

You have to teach a computer everything. Consider a simple task that we take for granted, such as "Transfer $100 from my savings account into my checking account." In its simplest state, a computer is completely incapable of doing that. It doesn't know what a transfer is, what an account is, what savings is, or what checking is. It doesn't know what a dollar is, or a bank is. It has to be taught everything. Software is about teaching that dumbass box with a 3-year-old's vocabulary how to do extremely complex tasks without making any mistakes.

In short, a computer programmer is an individual who sits down at a computer every day, and figures out how to handhold a mechanical idiot through tasks simple and complex, mastering the task of tutoring that idiot through those tasks until the idiot becomes a savant. It's grueling, frustrating, and infuriating. It takes time, and it's expensive. Depending on how big the task is, it might take lots of programmers to teach the idiot how to do the task.

So there you have it. That's why I think software is expensive.

Coincidentally, I also think it's why I and a lot of my peers in this field are psychotic (or very close to it).

Refactor Yourself

Take a moment to stock of where you are now. What skills do you have? How can you improve them? Every day of your career, you should be learning something, improving something, refining something. Your skill set should be undergoing constant refactoring. This can only make you more efficient. If the stuff you're learning isn't making you more efficient, discard it.

At some point, you have to have the guts to go against the grain. Just because a "best practice" works for someone else at some other company doesn't necessarily make it a "best practice" for you and your company. A "proven methodology" isn't necessarily going to be a "proven methodology" for you. Have the guts to challenge the status quo. If it's not making you more efficient, it's likely hindering you. Refactor it out.

If your team doesn't have the funds to learn some new technique, seek that knowledge personally. There is no reason that your company's inability to fund team education should hold you back. Buy books and read. Search the Internet. Read blogs and programming newsgroups. Experiment with code. Ask your peers. Never stop seeking knowledge. Never stop learning.

Take some of your old code, copy it, and then refactor the hell out of it. You'll be surprised what you can learn by simply refactoring code: more efficient ways to implement things that you did before (and will likely do again), better algorithms that work faster, use less resources, and are easier to maintain. Refactoring improves your skill set. Refactoring your own code, on your own time, is a personal competition against yourself to improve your own skill set.

You don't need to compete against anyone else. Coding cowboys, platform fanboys, methodology purists, conspiracy theorists...you shouldn't be worrying about them. You should worry about yourself. Make yourself as good as you can possibly be. Every day, ask yourself this essential question: "How can I improve myself today?" Find that way, and then do it. Set aside a little time every day to refactor your skill set.

Each day is an opportunity to make yourself a little bit better, a little more efficient than you were the day before. With each passing day, you have the opportunity to become smarter, faster, wiser, more valuable. But that means taking care to constantly revise your skill set. Have the wherewithal to discard habits and ideas that simply don't work. If you suspect you're doing something one way simply because you've always done it that way, or because that's the way everyone else does it, question it. If you can't see a tangible benefit to it, refactor it out.

Look, I'm not Gandhi or anything. But I can tell you this: I firmly believe that the key to success in this field is a personal commitment to growth. Don't trust anyone to just hand you knowledge, or to stumble across the skills you'll need. You have to actively reach out and take the skills and knowledge you need to be successful. It's an active task. It's not going to be something you just acquire through osmosis.

We all have to get to a point where we realize that we're not as efficient, not as smart, not as skilled, and nowhere near as good as we could be. There's always someone out there who's better than we are.

Our goal isn't to compete with them. Our goal is to constantly aspire to be better than we are right now, at this very moment.

Batman: Software Ninja

Scott Hanselman painted this lovely picture today:

...It's rarely a good idea for management to go get "The Smart Guy" and have him come crashing down through a stained glass on a zipline ready to save the day.

[snipped!]

A crappy project can't be fixed by a line by line code inspection, no matter how good a ninja one is. Sez me.

I disagree. Batman's a ninja, and I'd bet real money he has *something* on his utility belt or in the Batcave that can fix any project. Even mine.

That is the ninja to whom you were referring, right, Scott? The one who comes crashing down through stained glass on a zip line to save the day?

Man, I would totally love to see that. Batman, crashing into a meeting for a doomed project, knocking some heads around and pointing his gloved finger in someone's terrorized face. Speaking in that low growl, slowly, authoritatively, so that there's no doubt that you know what he wants you to do.

Who could resist team leadership like that? Who would dare to waste our time with pointless agendas and pissing contests? People would get in, say what they needed to and get out. No one would dare to tick him off. And if they did, the consequences would be dire. He'd have an aggressive plan for dealing with risks. If he didn't, he'd get one fast.

Then again, you know there'd just have to be a Joker, a Two-Face, and a Riddler out there somewhere.

/emote sigh

I so need a new life.

Usability Observation of the Day

Imagine a person who's trying to quit smoking. That person is already in a foul mood. On the one hand, he wants to quit. On the other, he doesn't.

He's going through the pangs of withdrawals. It's pretty serious stuff. He's shaking, sweating, eating everything in sight. He's driving his car like he's Mario Andretti, and he's not even out of the driveway. Anything could set him off at any moment. A simple "Hello" is returned with "WHAT THE F#$>@ DO YOU WANT?!" He's ready to do violence.

So what do you, as a nicotine patch vendor, do?

You seal the nicotine patch in a vacuum-sealed packet that requires the addict to use a pair of scissors or a knife to get the thing open.

Usability Advice: DO NOT advise an addict to arm himself with a sharp object in order gain access to the one thing that will quell his cravings.

<Queue up the music from Psycho right here.>

Rather, perforate the packets, and make them easy to tear open. When that guy needs that patch, he needs it NOW and he doesn't want to spend forty minutes on a mad search for something to open it with. By the time he finds something suitable, he'll likely want to stick it in someone's eye first, and then just sink giggling into a corner while he sucks the nicotine out of the patch.

Now, if you'll excuse me for a moment, I have to find a pair of scissors.

Feature Creep Accelerates Entropy

Many folks cruising around Digg recently have seen the numerous articles pointing to the claims by Steorn that their Orbo device can produce "free" energy, in direct violation of the Second Law of Thermodynamics. Basically, the device claims to produce more energy than it uses.

Scientists everywhere are choking on their degrees, and with good reason. Such a device would be either (a) a ridiculous grab for attention based completely on fantasy, or (b) a radical shift in our understanding of thermodynamics.

But this post isn't about the Orbo device. It's about a key principle of the Second Law of Thermodynamics: Entropy. Specifically, we're interested in how entropy affects software. To see where I'm coming from, however, let's review:

A measure of the amount of energy in a physical system not available to do work. As a physical system becomes more disordered, and its energy becomes more evenly distributed, that energy becomes less able to do work. For example, a car rolling along a road has kinetic energy that could do work (by carrying or colliding with something, for example); as friction slows it down and its energy is distributed to its surroundings as heat, it loses this ability. The amount of entropy is often thought of as the amount of disorder in a system.

entropy. Dictionary.com. The American Heritage® Science Dictionary. Houghton Mifflin Company. http://dictionary.reference.com/browse/entropy (accessed: July 06, 2007).

 

The car, rolling down the road, only has so much energy to work with. Some of that energy is kinetic (forward motion), and some is being dissipated as friction on the tires and by the air blowing past the car. But there's a finite amount of energy involved. If the driver taps the brakes, more energy is transferred to the brakes, but it's the same amount of energy. The car simply slows down, which has the net effect of reducing the severity of an impact with a tree.

So what the heck does this have to do with software?

All software suffers from entropy. You can only cram so many features into it before it has become so large that it has more unusable features than it has usable features. The code itself, at that point, becomes a chaotic mish-mash of bits and pieces of work cobbled together by people who came and went over time, riddled with cracks where their styles, philosophies, and standards didn't quite mesh well. The quality of the source code itself begins to degrade, becoming less and less maintainable, sprinkled with routines, variables, classes, and modules that are never used. The older the product gets, the more pronounced the entropy effect will be, until someone either cans the project, or demands a full rewrite.

Users tend to have intense, often emotional bonds with their favorite products. Those feelings tend to border on the fanatical when those products are easy and simple to use. They get really upset when you unnecessarily complicate them.

Not surprisingly, the products that seem the most resistant to entropy are the ones with the fewest features. They just work. Light switches are a wonderful example of a device that resists the effects of entropy. You just flip the switch, and the light comes on. But then, down the road, someone fancified them with dimmers. Now the knob pops off. Sometimes, the dimmer breaks. You need special bulbs. People cry out for plain switches. So the dimmer rarely gets used. Entropy claimed it.

Toasters, similarly, are a simple device that resist the effects of entropy. Insert bread; push a lever; electrical current is applied to the heating element; the rack pops up, and you have toast. It's simple. But when we complicated it with toaster ovens, microwaves, and bagel toasters, things got tricky. More features, like programmable times, and self-cleaning, and defrosting, and popcorn timers became complex and we couldn't figure out how to use them. So they became unusable. Entropy claimed them.

In software, similar things happen all the time. We toil for hours to get our features just right. We labor over their design, making sure that this feature we've envisioned is perfect, because we know that users will love it, they need it, they absolutely have to have it. But the truth is that users love simplicity. And the more features you add to a system, the sooner entropy will claim it.

Microsoft Word, for instance, has more features than anyone could possibly use in their lifetime. Yet an entire army of people labored for weeks and months to get them just right. Microsoft Office 2007 is a shining example of unusable energy outweighing the usable energy. It's the poster child for the entropy effect among software products.

When you're designing products, as I do for a living, think carefully about feature creep. The fewer features your product has, the better the chances are that those features will be used. Increasing the number of features increases the chances that most of them will be unused. It's entropy at its best.

Email: The Productivity Killer

Was I asleep? Did I miss something? I was under the impression that email was something you sent when you wanted to communicate something that was good to know, or needed an answer, but didn't need your direct attention or immediate response this second. If it was something really important, you flagged it as urgent.

But suddenly, every email we send has become a ticking time bomb that has to be answered right now, this very second.

God help you if you don't.

We've got people who send emails, and wait anxiously, huddled over their keyboards, waiting for their response. If they don't get one within a few minutes, they pick up the phone or open an IM window and ask, "Did you get my email? Did you read it?" If you dare answer in the affirmative, they ask you why you didn't respond. If it was that urgent, why didn't you just call? 

We've become a society that hinges on instant communication, leaving instant message services open, cell phones on 24/7, and email applications open and minimized so we know the very second we get an email.

But here's the skinny of it: the vast majority of the email we receive is unimportant garbage. And yet, Microsoft Outlook feels utterly compelled to give me a nice little notification every time I get an email from someone who wants to tell me there's leftover cake in the coffee area, that I can get a lower rate on my mortgage, or that some obscure fellow from offshore wants me to engage in a money-laundering scheme with him.

That kind of distraction is a major productivity killer. It's every bit as bad as an instant message window popping up from your mom, telling you that the cat puked on the kitchen floor.

Why? Because it breaks your concentration, and usually at really critical moments when you've got your groove on.

Face it: You don't really need to know those things. Microsoft Outlook shouldn't alert you to every email that you should receive. It should alert you to those marked urgent or flagged with a read-receipt, and then, only those from people on your approved senders list (usually, your contact list or the Exchange server). But no, that might actually make sense.

I came into work today and decided to conduct an experiment. I'm normally distracted by a ton of little things that I am certain are the primary killers of my productivity. They usually boil down to three things: email, instant messenger, and the compulsive urge to browse the Web. So I set out to eliminate those things. I shut down my instant messaging applications. I checked my mail once, and then exited the application. I didn't minimize it, I closed it.

Then, instead of leaving the defect tracking software open in a browser all day, where I'm tempted to keep checking my feeds and peruse Digg, I exported the list of defects to Excel, and then closed the browser. I then kept the browser closed all morning.

I've gotten more work done this morning than I have in the past two days. Why? Because I can concentrate. My train of thought isn't constantly being derailed by instant messages, email notifications, and that highly tempting little bookmark that mocks me on my Firefox Live Bookmarks toolbar.

These kinds of things are serious productivity killers. They simply pull me away from what I should be doing. So you can bet that I'll be seriously mitigating their impact on my work life in the future. It's a sad fact that in our world, we can't escape them because they've become so entrenched in the way that we work and live.

But, I'll set aside specific times to check my mail each day (first thing in the morning, lunch time, and right before I go home). Instant Messaging software is out. And I'll significantly reduce my use of the browser, constraining myself to those uses for which it was intended.

Maybe, if I do those things, and am able to get my work done at the office, I won't have to take my work home. You know, where there's a completely different set of distractions.

On Self-Control and Software Development

This essay was written months ago, and never posted. I resurrected it today, in light of certain recent entries.

Recently, as I was working to deliver a major release on a product I'm working on, I found myself sidetracked by a little project of my own.

You see, there's this little problem with one of the data fields in the database. It's not major, just an annoyance, like a four year old poking you in the ribs for an hour, asking repeatedly, "Does this bug you?"

Well, it's been bugging me for ages. And I found myself today doing database queries and pasting data into Excel to have Excel build update queries for me using formulas (nice little time saver that is) so that I could include those statements in the SQL script to accompany the next major release.

And then it hit me: no one asked for this. It's not included in the test plan for this release. It's gold plating. I'm doing this because I want to, not because the customer asked me to.

Whoa, there, cowboy. Get a grip on yourself. Set that stuff aside, and focus on what you need to do, and not what you want to do. There are far more important deliverables to worry about, and you don't have time to waste on unauthorized features or fixes. Especially when those fixes are for issues that don't negatively impact the application. (It was a display issue--first name before last name.) It's just fluff.

In reflection, I find myself experiencing these kinds of monumental self-control issues all the time. I get really excited about the things I could do for the customer, and I really want to do them for them. But the truth is that just because I can do something for them, it doesn't mean that I should do it.

Any change that I make to the product has the potential to introduce new defects into the system. That's why every change that I make to it must be tested.  It's why there's so much testing involved in software. (And if there isn't, something's seriously wrong.) And the testing doesn't just occur here, at my desk. It happens at the client. The product undergoes rigorous user acceptance testing. And testing isn't cheap--it consumes precious man hours, which equates to someone's hourly wages. And if I haven't gotten it right, it has to be fixed and retested. It can amount to massive amounts of money in man hours of testing.

Lets not forget the impact that the change has on updating the test plan, the release notes, requirements documentation, and user guides. Plus any associated costs with reprinting and redistributing them.

And what happens if the customer decides that my unauthorized change needs to be taken out? What if its impact on the system is so drastically negative that it must be removed? Can it be easily rolled back? And if it must be removed, what are the costs associated with doing so, and republishing all the updated documentation and builds?

Are you getting my point yet? The cost of a simple change isn't just what it takes me to code and test it at my desk. That's just the tip of a massive iceberg.

It takes a lot of self-control to prevent myself from adding features to a system when those features aren't (1) requested by the customer, (2) included in the project plan, and (3) absolutely critical to the current release.

The problem, I think, is that a lot of developers out there, myself included, don't get sufficient mentoring in the discipline of self-control when it comes to software development.

For example, we're all hailing the virtues of refactoring code to improve its maintainability, and I agree that that's a good and useful thing. But how many developers know that just because you can refactor a piece of code doesn't mean that you should? How many developers are out there bogging down project schedules because they're busy refactoring code when they should be developing software that meets the requirements for the project deadline?

(And here, I will sheepishly raise my hand.)

It occurs to me that before I ever modify a piece of code, before I ever touch that keyboard and write any new class or method, or create any new window or Web page, I should be asking myself, "Is this in the project plan? Is it critical to the current release?" If it doesn't satisfy one of those questions, I shouldn't be doing it.

The key to getting that product out the door on time is staying focused, and not getting sidetracked by fluff. Take it from someone with experience: it's easy to get sidetracked by fluff. Adding cool features is easy to do, because you're excited about it, and motivated to do it. Working on the required deliverables is hard work; it requires discipline and self-control. You have to stay focused and keep your eyes on the target. (You thought I was going to say "ball," didn't you?)

But we, as human beings, don't want to do what we need to do, we want to do what interests us, and what excites us. It takes an act of sheer will to resist that urge, to restrain ourselves, and get the real work done. I would imagine that one of the things that separates a mature developer from a novice developer is quite likely his or her ability to resist that urge to introduce fluff into software.

In the end, I think it might be a good idea if programming courses included curricula on self-control as a discipline for developers. And I mean that quite seriously. We need to have it drilled into our heads that we shouldn't be adding anything to the product that only serves our own sense of what's cool or useful. That's not to say that sometimes developers can't predict useful features before the users do; but they cannot and should not be introduced haphazardly into a product: they should be included as a planned feature as part of a scheduled release, so that they can be adequately tested and documented, and not just suddenly sprung upon someone as an easter egg.

There's a time and a place for everything. Gung-ho initiative has its proper place; software isn't one of them.

Traffic School for Software Projects

Ever wonder if there's a correlation between the way some folks develop software and the way they drive their cars?

The Tailgaters

These are the ones who are constantly on your ass, going 90 MPH, and flipping you the bird because you are holding them up. You might even be going faster than the guy to your right, but that's apparently not fast enough for the maniac behind you. If you get out of this guy's way, he will inevitably zoom past you, race right up on the next guy, and do the same thing to him. These guys are an accident waiting to happen. They aren't thinking about the safety of others, or about the possible parking lot ahead of them on the road. All they care about is getting where they're going as fast as possible. If some poor slob gets angry enough, he's going to slam on his brakes, and there's going to be a loud screeching sound followed by a crash, car parts flying everywhere, sirens, flashing lights, and lawsuits.

In the software world, these are the folks who are constantly hounding you, demanding, "Is it done yet? Is it done yet? Come on! Hurry up!" The quality of the project takes a back seat to the speed of its delivery. They will likely sacrifice people, resources, process, features, and quality and not worry about the risks that await them down the road. It's all about meeting the deadline, and God help those who get in their way.

Advice to you Tailgaters: SLOW THE HELL DOWN! It's better to get there alive than in a body bag. You aren't saving that much time by going so fast you make Mario Andretti look like an old man with a walker. You're risking lives, and you will eventually crash and burn. I'll wager that alot of you think you've never been in and accident. Well, that might be true. But I bet you've caused a lot of them.

Same goes for you software tailgaters. Slow it down. You want the project to get there in one piece, with as much of its promised feature set and team in-tact as possible. Your constant pressure isn't helping anyone. A healthy dose of encouragement is one thing. Unhealthy pressure (tailgating) is quite another; it kills morale, and results in high turnover, which leads to increased costs and loss of employee buy-in. So knock it off; let your foot off the accelerator!

The Weavers

Weavers like to move in and out of traffic, zipping from one lane to another, often unexpectedly, and rarely using their turn signals. Their lane changes are unpredictable, haphazard, unannounced, and a source of frustration and terror to those around them. They are the reason that defensive drivers exist. You can't predict where they're going, when they're going to change directions, or what direction they're going to move in. The best you can do is hope you have power steering, anti-lock brakes, and really good peripheral vision.

Weavers drive like they're the only ones on the road; changing lanes doesn't require a turn signal because, well, they don't care if anyone's in harm's way. All that's important is that they get into the lane they want to be in. They just yank that wheel to the left or right and fwoop! there they are where they want to be, while your brakes squeal, your car lurches, and heart hammers in your chest. We won't talk about the string of expletives that explodes from your mouth.

In the software world, these are the guys on the project who like to suddenly shift direction midstream. It might be a change in the process, in the coding standard, in the project's vision. It could be a change in how the teams are set up, in the reporting structures, in the UI's design, in the tool selection, in anything. Anything. But they thrive on change, and they don't really care about the impact that those changes have on the people around them. They're oblivious to the others around them. Change is exciting! It's good! It's cutting...no, it's bleeeeeeeeding edge!

Weavers work like they're the only ones on the project, with complete freedom to do whatever they like. No warning is necessary; asking for someone's advice or opinion about it's suitability is pointless because it doesn't matter to them. They want to do it, so they're going to do it. Who care's if anyone else likes it?

Advice to the Weavers: Learn to use your mirrors and turn signals! That means you actually have to become aware of the other drivers on the road. Yes, I know. That means you sadly have to realize that you are not the center of the universe, and that there are others who share the road with you.

And for those of you who think you can just change anything about a project midstream, learn to think differently. Change introduces risk into a project. Unannounced risk is unmanaged risk. Unmanaged risk increases the chances of project failure. (Crash and burn, Mav. Crash and burn.)

The Gawkers

Gawkers are a special breed of road hazard. These folks will be traveling down the road and spot something they find of interest, and slow down to a crawl to scrutinize it. In the process, the people behind them must slow down as well. This creates a nifty domino effect in traffic. Interestingly enough, what these folks find of interest is usually nothing of interest at all. But the ensuing traffic jam becomes a subject of much heated debate.

Software projects are commonly plagued by Gawkers: individuals who have no real business being involved in the project, but who nonetheless feel compelled to attend every meeting and inject their two cents, suggest things, and add "filler material" to waste everyone's time. They want to be CC'd on every email, take part in every hallway conversation, and stop by the developers desks to "check-in" and provide "moral support." They pepper us with questions and phone calls, spot-checks and emails, instant messages, and reminders, additions to meeting agendas, and interjections about meaningless and completely unrelated minutia in the middle of our meetings when we'd rather be back at our desks doing real work that matters.

Advice to the Gawkers: For God's sake people, stop gawking and drive! Pay attention to the road ahead of you, ignore that pigeon crapping on the rail, or that car parked on the side of the road, and just drive! You're holding up traffic for miles and leaving a string of accidents behind you.

And all you busybodies who seem to think you have to be involved in projects that have nothing to do with you, "Move along people, nothing to see here." We have work to do, and you're holding us up. Just leave us alone so we can get it done.

The Gadgeteers

We've all seen them. Some of us have been them. Driving down the road, accelerator hammered to the floor, while trying to operate some electronic gadget, be it a cell phone, Blackberry, iPod, radio, or even something as innocuous as the car radio. These geeks are so distracted by what they're doing that they'll careen recklessly from side to side within their lane, as if they're using the Braille Method to stay within the lines. They'll speed up, slow down, and be so engrossed by the spectacular fabulousness of their toys that they have become a danger to everyone else around them.

With the Gadgeteer, it's all about the toys. These devices aren't required to operate their vehicle: they're distractions that make it unsafe to operate the vehicle, especially at high speed, because they make it impossible for the driver of the vehicle to focus on what he or she should be focusing on: the road, and the other vehicles on it.

Software professionals are often distracted by the shiny, flashy appeal of the latest tools. But the truth of the matter is, you can write software in Notepad or any simple text editor and compile it with a command-line compiler. You don't need the flashy, glitzy tools to get the job done. (For the difference between need and want see here.) They might make your job easier, more comfortable, and less of a nightmare, but you don't truly need them. So don't get all obsessed about them and put other people at risk over them. Certainly don't become obsessed about them.

And whatever you do, don't let them constrain you or the project.

Advice to the Gadgeteer: First off, shut up and drive. If you're so compulsive about yakking on a cellphone 24/7, get an implant with a neural interface. Oh wait, we don't have that technology yet. Probably because NO ONE NEEDS TO BE THAT CONNECTED! So disconnect, relax, and enjoy the drive. Stop fiddling with all those gadgets, put them down, and pay attention.

If you're a software Gadgeteer, learn to do without for a while. Learn what it is to use primitive tools. You might actually learn to appreciate the tools that you do have. All those big, flashy programs you use are likely overkill for what you do, anyway. How many of the features do you actually use? Does the system really make you more productive, or does it spend a lot of time distracting you with countless configuration options, window placement options, font and color options, and so on? Wouldn't it be nice if you could actually focus on the code for a change?

The Lost Navigators

There's not a lot of hope for a Lost Navigator. Drifting slowly through the highways, byways, and side-streets of the country, looking for their exit or side street, hopelessly lost, and without a clue, these poor folks present the rest of us with a horrible choice: to be patient, or to pass. Our frustration knows no bounds. They drive slowly, obviously looking for something; sometimes, you can see them, scanning a mapbook, or consulting a crumpled set of MapQuest directions. In the worst case, they have no directions whatsoever to go by. Inevitably, they never seem to stop and ask for directions. They are lost, they are slow, and they are frequently oblivious to the line of cars piling up behind them as they look desperately for the place that they should be going.

Sometimes, in a moment of relief, they see their exit, and swerve to enter it. But frighteningly, this is all too often done from the left lane of the highway, and they must cross two or three lanes of traffic at the last moment to reach their offramp. And so they do. Cars swerve to get out of their way, slamming on brakes, honking horns, and nearly going off the road themselves.

Their problem, usually, is that they simply lack a decent navigational system, a decent set of instructions that tell them how to get where they want to go. It's not always their fault. Sometimes, however, you get the pig-headed kind who decide they can find their way through anything and can't be bothered to stop and ask for directions.

Often, in software development, there are folks who are placed in roles for which they are not well equipped. They are assigned a task, given little to go on, and told to do it. They have the little bit of information they're given, and they set out. So they move slowly, trying to find their way, and they might stop occasionally to ask for help. But their progress is slow, and as a result, their piece of the project moves slow. That's not so bad, actually; it's the guys who refuse to ask for help that cause the problem. By refusing to seek help, they're making a bad situation worse.

Advice to the Lost Navigators: Get better instructions. MapQuest lies. Frequently. Improve your situation! And for you blockheads who won't ask for directions, get off your high horse. Although I'll guess that your problem's a genetic defect of some sort, so I won't push that issue any further.

If you're in a software project, and you're struggling due to a lack of information, get that information! You can't be expected to do your job without it. If you need information about process and such, seek out excellent sources of information from places like the Software Best Practices forums maintained by Construx software. Interact with other developers in the Google newsgroups (drop me a line and I'll point you to them). Check out The Code Project and the programming topics at reddit. Read MSDN. Ask other developers where they go for information online. There's no reason for you to suffer from a lack of information when there's a wealth of free information from experienced developers providing it online.

And there are Others...

We know there are more. Let's not forget the Makeup Artist, the Conversationalist, the Belligerent Road Hog, and so on. But I'm sure you all can figure those out for yourselves.

And I'm sure you all have ideas of your own.