Showing posts with label Software Development. Show all posts
Showing posts with label Software Development. Show all posts

Wednesday, June 30, 2010

Knowing How is Not Enough

There’s an old adage that I heard once, and it’s stuck with me through the years:

He who knows how to do a thing is a good employee. He who knows why is his boss.

I’m also fond of this one:

If you can’t explain it, you don’t understand it.

So I’ve been ramping up on some technology that I’ve really not had an opportunity to really use before, and I’m very excited about it. To make sure I understand it, I’ve decided to go back to the MSDN examples, reproduce them one line at a time, and then document the source code as I understand it. It’s a great way to learn, and sheds a great deal of light on what you think is happening, versus what’s actually happening.

To be perfectly honest, the technology is AJAX. Over the last few years, I’ve predominantly worked for companies that haven’t had any use for Web services, so there’s been no compelling need for it. I’m starting a new job soon and it will rely heavily on Web services, and I really want to make sure I understand them well before I set my foot in the door. It has never been enough for me to know that you just drag a control onto a form or page, set a few properties and press F5. To me, that degree of abstraction is a double-edged sword.

When abstraction reaches the level that it has with Microsoft AJAX, you start to run into some fairly significant issues when it comes time to test and debug the application. The MS AJAX framework is no small accomplishment, and it hides a lot of complexity from you. It makes it so easy to write AJAX applications that you really don’t need to understand the underlying fundamentals of Asynchronous Javascript and XML that make the whole thing work. Consequently, when things go wrong, you could very well be left scratching your head, without a clue, and no idea where to begin looking.

Where, in all of this enormously layered abstraction did something go wrong? Was it my code? Was it the compiler? Was it IIS? Was it permissions? Was it an update? Was it a configuration setting? Was it a misunderstanding of the protocol? Did the Web service go down or move? Was the proxy even generated? If it was, was it generated correctly? Do I even know what a proxy is and why I need it?!

When I started learning about AJAX, we coded simple calls against pages that could return anything to you in an HTTP request using the XMLHTTPRequest object. Sure, it was supposed to be XML, but that was by convention only. The stuff I wrote back then (and I only wrote this stuff on extremely rare occasions, thank the gods), returned the smallest piece of data possible: a single field of data in flat text. It was enough to satisfy the business need, and didn’t require XML DOM parsing.

But even with DOM parsing, the code to make a request and get its data back via XMLHTTPRequest was a lot smaller than all the scaffolding you have to erect now. You might argue that you don’t have create a lot of code now, but that is just an illusion. You’re not writing it, but Microsoft is. Just because you don’t see it doesn’t mean it’s not there. Do you know what that code is doing?

In theory, the Why of Microsoft AJAX, or any AJAX library is to make our lives easier when it comes time to write dynamic Web applications that behave more like desktop applications. To a certain degree, they have. When they work. But when they don’t, I wonder if the enormous degree of abstraction they’ve introduced hasn’t dumbed us down to the point where we’ve ignored essential knowledge that we should have.

If you’re going to write Web services, or consume them, you should, at a minimum, understand what they are, and how they work. You should understand their history, how they evolved, and the problem that AJAX tries to solve. It’s not enough to know how to write a Web service, you have to know why you’re doing it, and why you’re doing it the way you are. That sort of knowledge can be crucial in making the right choices about algorithms, protocols, frameworks, caching, security, and so on.

But this could be true of any technology or practice we learn. AJAX, LINQ, design patterns, TDD, continuous integration, pair programming, and so on. Know why.

Try this simple litmus test. Explain something you think you know to one of your peers. If you can’t explain it clearly without having to pull out a reference or go online, you don’t understand it the way you think you did. Consider relearning it. It’ll only improve your value to yourself, your peers, and your employer.

Saturday, June 26, 2010

Generics, Value Types, and Six Years of Silence

It’s been a long time since I worked on the code for NValidate, but in a fit of creative zeal I decided to dust it off and take a look at it.

As one of the posters on the old NValidate forums pointed out, it was full of duplicate code. Granted, that was a conscious design choice at the time: it was written well before generics came out, and performance was a key consideration. I didn’t want a lot of boxing and unboxing going on, and since a lot of the tests dealt with value types, the only way I could see to get that stuff done was to duplicate the test code for specific types.

Well, time marches on and languages advance. .NET 2.0 came out, and along with it came generics. I figured that they would provide a fantastic way for me to eliminate a lot of the duplicate code. I mean, it would be awesome if we could just write a single validator class for all the numeric data types and be done with it. And that’s where I hit the Infamous Brick Wall&tm;.

It turns out that generics and ValueType objects do not play well together. At all. Consider the following piece of code:

public class IntegralValidatr<T> where T : ValueType
{
}

This, as it turns out, is forbidden. For some reason, the compiler treats ValueType as a “special class” that cannot be used as the constraint for a generic class. Fascinating. The end result is that you cannot create a generic class that requires that its parameters are derived from ValueType. You know: Boolean, Byte, Char, DateTime, Decimal, Double, Int, SByte, Short, Single, UInt, and ULong. Types you might actually want to work with on a daily basis, for all kinds of reasons.


The workaround, they say, is to specify struct. The problem is that struct is a pretty loose constraint. Lots of things are structs but aren't necessarily the types you want. I assume that's why they call it a workaround and not a solution.


So, anyway, here I am with a basic class definition. I can at least admit to myself that I can build the class outline as follows:


public class IntegralValidator<T> where T : struct
{
public T ActualValue { get; internal set; }
public string Name { get; internal set; }
public IntegralValidator (string name, T actualValue)
{
this.Name = name;
this.ActualValue = actualValue;
}
}

But now it’s time to create a test. The problem is determining how to perform basic comparisons between value types when you can’t seem to get to the value types now that they’ve been genericized. Understand that NValidate needs to be able to do the following with numeric values:



  • Compare two values for equality and inequality

  • Compare a value to a range, and fail if it falls outside that range

  • Compare a value to zero, and fail if it isn’t zero

  • Compare a value to zero, and fail if it is zero.

  • Compare a value to the Max value for its type, and fail if it is or isn’t equal to that value.

  • Compare a value to the Min value for its type, and fail if it is or isn’t equal to that value.

  • Compare a value to a second value, and fail if it is less than than the second value.

  • Compare a value to a second value and fail if it is greater than the second value.

You get the picture.


The problem I’m experiencing is that it’s become clear to me that it’s really very difficult to convert a genericized value type back to its original value type. Consider the following code:


private byte ToByte(){
if (ActualValue is byte>)
// The as operator must be used with a reference type or
// nullable type ('byte' is a non-nullable value type)
return ActualValue as byte;
if (ActualValue is byte)
// Cannot convert type 'T' to 'byte'
return byte ActualValue;
}

So, if neither of these approaches works, how do I get to the original values? Generics appear to demand an actual object, which would, in turn, demand boxing and unboxing of value types (which I’m staunchly opposed to for performance reasons).


So, we go back to the drawing board, and eventually we discover that you can, in fact, get to the type through a bit of trickery with the System.Convert class:


private byte ToByte() {
if (ActualValue is byte)
// The as operator must be used with a reference type or
// nullable type ('byte' is a non-nullable value type)
return Convert.ChangeType(ActualValue, TypeCode.Byte);
}

Well, the problem I’m faced with now, upon careful reflection is this: If I’m drilling down to the original data type, I’m kind of defeating the whole point of generics in the first place. And that brings us to the whole point of this article.


I should be able to write a line of code like this:


Demand.That(x).IsNonZero().IsBetween(-45, 45);

And that code should be handled by generics that correctly infer the type of x and select the right code to execute, but I can’t. And the reason I can’t is because (1) you can’t use ValueType as a constraint for generics and (2) there is no common interface for numeric types in the BCL.


This is an egregious oversight in the Framework. Worse, it’s been an outstanding complaint on Microsoft Connect since 2004. Through multiple iterations of the Framework, despite numerous postings on the site and clamorings for the oversight to be corrected, Microsoft has yet to do anything about it. For some reason, they seem to think it’s less important than things like Office Integration, UI overhauls, destroying the usability of online help, and making Web services as difficult and unpredictable to use as possible.


It baffles me that Microsoft’s own responses to the issue have been questions like “How would you use this functionality?” Are they kidding me? There are so many uses for this it’s not even funny.


  • What happens when you have an array or list of heterogenous numeric types and need to work with them in some meaningful way using a generic (possibly a delegate)?

  • What happens when you want to write a program that converts 16-bit data to 32-bit data, or floating point data to Long data, or work with both at the same time using a common algorithm?

  • What happens when you need to work with graphics algorithms common to photo processing software?

  • What happens when you need to work with the many different types of value types and convert them back and forth quickly and efficiently as you would in, say, an online game?

  • Or, as in my case, what happens when you’re writing a reusable framework for validation and boxing and unboxing are simply not an option, and a generic solution would handily solve the problem, but you can’t because there’s no common interface – no One Ring that binds them all together?

It’s about time this issue was resolved. And this isn’t something the open source community can fix. This is something Microsoft has to fix, in the BCL, for the good of all humanity. Six years is far too long to leave something this painfully obvious outstanding.


Thursday, July 16, 2009

InternalsVisibleTo and Chasing Down Public Keys

Sometimes, just getting assemblies to cooperate the way you want them to is a real pain in the neck.

Today, I wanted to make one assembly (which we’ll call Foo) to be able to access the internal (Friend for all us VB geeks) members of another assembly (which we’ll call Bar). Now, the documented way to achieve this is by adding the InternalsVisibleTo attribute to your project. The code sample on MSDN looks like this:

[assembly:InternalsVisibleTo("AssemblyB, PublicKey=32ab4ba45e0a69a1")]

For a Visual Basic application, that statement goes into the AssemblyInfo.vb file, and looks like this:

<Assembly:InternalsVisibleTo("AssemblyB, PublicKey=32ab4ba45e0a69a1")>

No big deal. It’s not rocket science. You just need a strong name. To generate a strong name, you fire up the Visual Studio Command Prompt, and execute the Strong Name tool (sn.exe) and execute it as follows:


C:\Program Files\Microsoft Visual Studio 9.0\VC>sn.exe -k bar.snk

Microsoft (R) .NET Framework Strong Name Utility  Version 3.5.30729.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Key pair written to bar.snk

As you can see, this creates a .SNK file, which contains a public and a private key. However, its contents are not human readable. Further, the InternalsVisibleTo attribute requires that you provide the public key in the constructor. (Don’t even think about trying it without it.)


Here’s where things get tricky: The code samples on MSDN do not provide a public key to the constructor; they provide a public key token. There’s a huge difference between the two, and it’s very misleading. A strong name token is much shorter than a public key; the former is only about 16 characters long, the other well in excess of 140 characters. If you rely on the code sample from MSDN to get you where you want to be, you’ll be pulling your hair out in no time.


But how do you get the public key token?


You can retrieve the public key token as follows:


sn –p foo.snk barpublic.snk


This creates a new file that contains only the public key. It removes the private key information from the file.


sn –tp > barkey.txt


This creates a text file that contains the full dump of the key information, including the public key. We redirect it to a text file so that you can open it in the editor of your choice (because you’ll have to do some cleanup to get the key onto one line). You’ll want to select the public key and paste it into the PublicKey portion of the constructor for the InternalsVisible attribute.


So here’s everything we did at the command prompt:


C:\Program Files\Microsoft Visual Studio 9.0\VC>sn -k bar.snk

Microsoft (R) .NET Framework Strong Name Utility  Version 3.5.30729.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Key pair written to bar.snk

C:\Program Files\Microsoft Visual Studio 9.0\VC>sn -p bar.snk barpublic.snk

Microsoft (R) .NET Framework Strong Name Utility  Version 3.5.30729.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Public key written to barpublic.snk

C:\Program Files\Microsoft Visual Studio 9.0\VC>sn -tp barpublic.snk > bar.txt

C:\Program Files\Microsoft Visual Studio 9.0\VC>

And here’s the contents of our text file:


Microsoft (R) .NET Framework Strong Name Utility  Version 3.5.30729.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Public key is
0024000004800000940000000602000000240000525341310004000001000100e13cb392af5437279736fc3c33fe237242d0f6301fafb01c5cbc719d84102c2d8b30a148600997ed53d99624b5d0eab37fd6b24cca3ce7f7b62ae99f961e148d5421576bade0ac8ab1187a3eee318ca20026ffe9b56b8a63156f817cef49998633867ae547684e8e59c0fe0b68ab29dffa749340dc6cfdd18071f1b69c6772ac

Public key token is db218359dd8997df

So, when we finally add that attribute to our AssemblyInfo.vb file, it looks like this:


<Assembly:InternalsVisibleTo("Bar, PublicKey=0024000004800000940000000602000000240000525341310004000001000100e13cb392af5437279736fc3c33fe237242d0f6301fafb01c5cbc719d84102c2d8b30a148600997ed53d99624b5d0eab37fd6b24cca3ce7f7b62ae99f961e148d5421576bade0ac8ab1187a3eee318ca20026ffe9b56b8a63156f817cef49998633867ae547684e8e59c0fe0b68ab29dffa749340dc6cfdd18071f1b69c6772ac")>

Once this stuff is in place, Bar should be able to access any members in Foo that are marked internal/Friend. It should be smooth sailing from there.

Good luck!


Tuesday, July 14, 2009

On Legacy Software Maintenance

In the mad, mad, mad, mad world of software development, we are faced with the trying task of maintaining legacy systems. In an ideal world, that wouldn't be the case. We'd all be developing brand new systems from the ground up, writing ground-breaking code that no one has ever seen before, without the hassles that arise when you have to worry about things like backwards compatibility and maximum system uptime.

But this isn't an ideal world. The vast majority of us don't have the luxury of developing completely new systems. Instead, our lives are fraught with the perils of correcting defects and adding new features to systems that have been around for ages and, occasionally, decades. Those systems have usually passed through a number of hands and they tend to be poorly documented. They sometimes have sprawling feature sets, support technologies that have long since fallen by the wayside, bloated with code that doesn't appear to be invoked by anyone, and riddled with obscure and seemingly nonsensical comments.

Your job, as a maintenance developer, is to massage that seemingly horrific beast into a thing of beauty. Real people doing real jobs depend on it to get their work done in a timely manner. As much as we might loathe an ancient codebase, the language it was written in, or the tools we have to use to get the job done, truth is, a legacy application is maintained for a reason: it has intrinsic value to the bottom line of the business. When the folks who depend on that system can get their jobs done on time, in a productive manner, they can continue to draw a decent paycheck. That means that they can continue to pay the rent, put food on the table for their families, afford healthcare, and all those other essentials.

So tread lightly when you delve into the code of a legacy system. It's far more important than you think. We just take it for granted that it's a dirty, loathsome job and someone has to do it. We just happen to be the unlucky bastard who drew the short straw. Not so: you happen to be the lucky one who drew that straw. People depend on you to help them keep their families safe, warm, and well-fed.

My point isn't that every legacy application is manna from heaven. My point is that legacy applications exist for a reason, that they're maintained for a reason. They have long, varied histories for a reason. They have endured because they have value; they've grown beyond their original specification because the company sees real value in them, and doesn't want to lose that investment. The problem for you, as a developer, is in ensuring that you do not destroy that investment.

When we are first introduced to a legacy system, we have a tendency to look at the source code and view it as though it were written by a blind deaf quadriplegic with Tourette's syndrome. No one in their right mind would have written a system that way. What could they possibly have been thinking? You certainly wouldn't have! I certainly wouldn't have.

But then, over time, we start to learn things about it. The code is old; very old. There's been a high turnover rate, and the code has passed through lots of hands. The companies that published third-party components went out of business when the dot-com bubble burst. They used to use Novell for security, and then switched to Active Directory. When this thing was released, Windows 95 was still popular. They upgraded the database about two years ago, and had to make some emergency revisions to the code.

There are reasons that things like this happen in a legacy system that's old enough to qualify for Medicare. Many of those reasons are valid, and many of them are the product of a lack of time and budget. Sometimes, sadly, it's a result of a lack of skilled developers (but that's something for someone else to blog about). The point, in short, is that systems grow from an original vision into large, cumbersome, bloated systems because developers respond to the needs and demands of the business.

Now, here you are, present day, and you're tasked with maintaining that source code. You have two primary responsibilities: 1.) Fix the defects, and 2.) Add new features. Keep in mind that while you are doing both, you must not at any point in time break backwards compatibility or bring down the system. People rely on this system. It's the bread and butter (or part of it) of the business. And it's your baby now.

It is absolutely crucial that you treat legacy software with its due gravity. If you view it like it's some recurring annoyance, stop that. If you leap to hasty conclusions about the cause of problems in the system, stop that immediately. This is a system that many people rely on. Get it into your head that you need to treat this thing delicately, as if it were your pride and joy. Once you fix a defect, once you put a new feature into it, your name is associated with that software. Take pride in it. Do it right. Take your time.

Over time, the legacy system stops being the horrific beast associated with all those who preceded you. It becomes the creature that you have molded it into. And then, people will associate it with you, for better or worse.

Monday, July 13, 2009

The Sad State of SQL Refactoring

I love refactoring tools.

The ability to select a variable in code, right-click on it, choose the Rename command from a context menu, and then safely rename that variable in every location in which it appears is a boon to my programming productivity. Similarly, the ability to take a large block of complex logic and extract it to its own method is really handy. Or, the ability to reorder parameters, and know that every piece of code that calls that method will be updated appropriately saves me tons of time and improves my confidence in the quality of the code.

When you refactor code of any kind, you enhance its readability without changing the way it works. But refactoring by hand is frequently difficult, tedious, and error-prone. That's why refactoring tools exist and why the essential service they provide is important. They allow us to improve the maintainability of code, hopefully reducing maintenance costs, quickly and efficiently.

Enter SQL languages.

In theory, there is a standard for SQL languages: the ANSI SQL-92 standard. One would like to think that it would be a simple matter to create refactoring tools for any SQL based on the ANSI standard for SQL. One would be wrong. You can't use the standard as the sole basis of a refactoring tool.

Any given database vender wants to strive to make their product unique, to stand out from the crowd. And so, they don't entirely conform to the standard. They have additional features that separate them from each other. For example: Oracle organizes functions and procedures into packages. Microsoft does not. Microsoft allows a bit data type on columns. Oracle does not. And, although we all loathe to think of Access as a real database, Access provides a boolean data type that you can use in columns, while Oracle does not.

Now, let's also talk about legacy support. Oracle has been around for a very long time. Version 2 came out in 1979, from what I gather. Their legacy join syntax looks nothing like the ANSI standard (and that's not a criticism, just a simple statement of fact):

SELECT Customers.CustomerId, Company, OrderID
FROM Customers C, Orders O
WHERE C.CustomerID (+) = O.CustomerID
UNION
SELECT Customers.CustomerId, Company, OrderID
FROM Customers C, Orders O
WHERE C.CustomerID = O.CustomerID (+)

In ANSI SQL, this is :

SELECT Customers.CustomerId, Company, OrderId
FROM Customers C FULL OUTER JOIN Orders O ON
(C.CustomerId = O.CustomerId)

Now, we could add in all the different SQL variations that we know that are out there:

  • ANSI standard
  • Interbase/Firebird
  • IBM  SQL
  • Microsoft Transact SQL
  • MySQL
  • Oracle PL/SQL
  • PostgreSQL
  • Access
  • FoxPro

...and on and on and on, but you get the point. There are a lot of SQL variants out there. And their goal is to accomplish the same thing:

  • Create a data store.
  • Occasionally, modify the structure of the data store.
  • Get data out of a data store.
  • Put data into a data store.
  • If supported, execute code within the confines of the data store and (optionally) return a result.

Behold, SQL in a nutshell.

Over the lifetime of these various products, they have added features that they have to maintain for legacy support. Somewhere out there there's a business that absolutely must have that feature in place or their whole process will come crashing down. Don't you dare remove it. It doesn't matter that there are better features (likely based on a standard); there are applications out there for which they don't have the source code anymore, or which no one understands, and they're too afraid to touch.

Now, all these reasons have led us to a scenario where we have vastly different implementations of SQL. Sure, they share a lot in common, but they also have radically different feature sets and syntax. And that situation, in and of itself, has led us to one frightening and sad conclusion:

We will likely never have a tool that is able to connect to any database and be able to correctly refactor its SQL.

And that's just a damned shame. Because there's a lot of SQL out there. Stored procedures, functions, views, triggers, even inline SQL in applications and all that other jazz. But the amount of work it would take to get us to a point where a refactoring tool could recognize any variant of SQL and correctly parse it, refactor it, and not hose the code is enormous.

It's a pity, really. I could see tons of use for a tool like this. In my own office, we work with SQL Server, Oracle, and Atomix databases. What I wouldn't give for a tool that could refactor SQL to enhance its readability without changing the way it worked.

And who knows? Down the road, we may be working with something else.

But this is the world we live in. If we want to refactor SQL, it looks like we'll have to settle for separate refactoring tools for each language. And then each will come with its own quirks. That might be good or bad, but it's likely the best we can hope for for now.

 

Saturday, July 11, 2009

Defects and the Scientific Method

Let's leap back a few years to that frightening time in Junior High School. Remember those years? September, possibly October. It was still hot, and the air conditioning didn't work. Your science teacher was standing in the back of the room, with an overhead projector and had a slide up for you to copy down. On it, he had this information:

THE SCIENTIFIC METHOD

  • Ask and define the question.
  • Gather information and resources through observation.
  • Form a hypothesis.
  • Perform one or more experiments and collect and sort data.
  • Analyze the data .
  • Interpret the data and make conclusions that point to a hypothesis.
  • Formulate a "final" or "finished" hypothesis.
  • Ah, remember those days? Remember how boring it was? Remember thinking to yourself, "I'll never use this?"

    Well, as a software developer who is tasked with maintaining software that is virtually guaranteed to contain defects, you can be certain that you need to be intimately familiar with The Scientific Method.

    The Scientific Method provides a clear roadmap for defect isolation. In fact, anyone who has any real experience isolating defects without disturbing the rest of the system has (whether he's aware of it or not) used the Scientific Method to do so. Here's how it breaks down:

    1. Ask and define the question. The software should behave in this manner, but it does not. What is the cause of this problem, and how do we fix it?
    2. Gather information and resources through observation. In a controlled environment that mimics production as closely as possible, reproduce the defect. If possible, step through the code and observe its behavior.
    3. Form a hypothesis. The defect is caused by this behavior in the system (or by the behavior of this external system).
    4. Perform one or more experiments and collect and sort data. Implement a code fix; attempt to reproduce the defect using the fixed code. Observe the results.
    5. Analyze the data. Did the code fix have the desired effect? If so, how?
    6. Interpret the data and make conclusions that point to a hypothesis. Was the code that was modified the cause of the defect, or was it merely a symptom of an underlying problem requiring further resolution?
    7. Formulate a "final" or "finished" hypothesis. If the defect is fully repaired, check all code into the repository. Otherwise, continue the analysis until you have rooted out the underlying cause of the defect.

    Simply put, there's no guesswork in defect resolution. It is a rational, thinking process, much like a game of Sodoku. If you approach any defect and just yank an answer out of thin air, You're Doing It Wrong.

    Instant answers to defects are a dangerous game. Your first, instinctive answer to any problem is likely to be wrong; the chances of this being true will only rise as your code base grows in size. As your product gains features, you'll want to take greater care to make sure that you have taken the time to disturb absolutely nothing outside of the defect you're trying to correct. In that case, take some advice: Keep your grubby fingers to yourself. Touch only the code in the defect domain. The best way to do that is to have a plan for defect resolution, and I strongly encourage you to apply The Scientific Method.

    Developing software is a task for those who can think. It is not a task for the simple-minded, the lazy, or the inattentive. You have to be willing to pay attention to the details, and to invest the time it takes to hunt down a defect in painstaking detail to get to the root of a problem.

    A good software developer knows the difference between a symptom and a disease, and how that correlates to software defects. Sure, you have a NullReferenceException, and your code is missing a handler for that. But is the problem the missing exception handler, or is the problem that a null somehow got into a table that should never have had it, or that a stored procedure in the database returned nulls when they were never expected? Which one is the symptom? Which is the disease? Make sure you're fixing the right defect. Don't just prescribe aspirin when the software needs invasive surgery. To find that out, you need to think critically. You need to apply the Scientific Method.

    Sunday, April 6, 2008

    The Absurdity of "Don't Reinvent the Wheel"

    As developers, we've had this adage drilled into us from the beginning: Don't reinvent the wheel. In short, don't rewrite what's already been written. The idea is sound, in theory. You can save yourself time and money if you'll simply reuse existing code and/or components rather than writing them yourself from scratch. This time and money is saved up front when you write it (or would have written it), and down the road, when you have to maintain your system.

    However, I'd like to point out another, equally applicable adage: There's nothing new under the sun. Anyone who's ever tried to write a novel, a short story, a play, a movie, a song, or a piece of software, will know this one simple truth: somewhere, at some point in time, it's already been written.

    Every algorithm, every piece of code that we will ever attempt to write has already been written somewhere, at some point in time, by someone. Only the names have been changed. You're not inventing anything that is completely new, that's never been seen before. You should wisely disabuse yourself of that notion as quickly as possible.

    In the grand scheme of things, at the application level, you may very well have an idea for a system that is unlike anything that has been done to date. But the algorithms that drive it have already been written. Bubble sorts, hashes, exception handlers, encryption, data access, socket management, shopping carts, entire application frameworks, date management, document management, serialization, port I/O, and all that other stuff has already been done. Further, it's already been done several times over in many different languages to varying degrees of success.

    Tragically, if you're using a large application framework, like Java's EE or Microsoft's .NET, the chances are good that the functionality you're looking for is built right into the framework itself. The problem is that the framework is so vast that you'll spend more time looking for it, and determining whether or not it works the way you need it to work than you would just rewriting it yourself.

    Application frameworks are stunningly afflicted with feature creep. They must do everything under the sun, must meet every possible need. The problem, then, is that their scope becomes so broad, so vast, that no one in their right mind could possibly grasp the totality of all that they can do. It is inevitable that anyone using them will reinvent some of their functionality. The scale of that functionality might be small (reformatting dates) or it might be substantial (pooled database connections).

    In the end, it's absurd to think that we can possibly avoid reinventing the wheel. Of course we're going to reinvent it. Every application we write is a reinvention of someone else's wheel. It just so happens that our wheel is a custom wheel. All this paranoia about reinventing the wheel is blown out of proportion. A proper buy vs. build decision should never be neglected; but don't ever think for one minute that what you're creating hasn't been created before.

    Consider the scenario where you're under the gun to get a product out the door. And I mean it's a really tight schedule. And don't act like it's a perfect world, and you have leverage over the schedule. This is reality here. In the real world, the customer controls the schedule, because it's tied to when the product is released, and that's tied to this big, huge monstrosity in another state or another country. The product's delivery schedule is a train barreling down the track at 120mph and no one short of God can stop it. Now, you have a very finite amount of time to work in. You need an algorithm. You know you could write it. Or you could look to see if someone else has written it.

    If you do the whole Web search thing, you have to ask yourself a few questions: Is it from a source you trust? Is it in the language you're using, or do you have to convert it? Does it work? Does it need to be tweaked? If any of these fail, you're back to the drawing board. Time's wasting here, and that train's getting closer to its destination. If all the answers pass, you have to make sure you don't run into any copyright or licensing issues with that code. (You are paying attention to that, aren't you?)

    If you decide to peruse your application's framework, you'd better hope it's well documented, and very easy to search. Good luck using the search features in .NET. It's not like the ASK.COM interface, where you can ask, "How do I convert a date in DOD format to Gregorian format?" Yeah. Good luck with that. On the other hand, you could ask your coworkers. They might know. Then again, they might not. If they don't, you're off to Google to get the information. Here's hoping you get a timely and accurate response.

    Sure, this is an extreme example. But it makes my point: At some point, the work has to get done. You can't afford to spend days or weeks scratching your head about whether or not that wheel's already been invented. Believe me, it has. The problem is, there are a countless number of wheels, and none of them are labeled, and you don't know where to find the wheel you're looking for.

    Stop wasting time, and invent your own damned wheel.

    After all, whatever code you might reuse, is just someone else's reinvention of the same wheel.

    Thursday, March 27, 2008

    My First Time: Software Passion

    I remember when I was first bitten by the computer programming "Bug."

    I was young: still in high school, in fact. At my particular high school (in Fontana, California), we had an Indian Education Department. That small department was lucky enough to have a TRS-80 computer for the kids in the Indian Club to play on. I dare say that the computer itself drew a few kids to the club; we were misfits, outcasts, by and large, but we were drawn to that thing like moths to a flame.

    It was nothing to look at really. It was just an old monochrome monitor, with a keyboard, and a tape drive. Our model didn't even have a floppy disk. Everything was on tapes. But we were mesmerized by that thing. I remember watching one of the other kids, Dan, fire up ZORK, and typing, "go north" into the computer. It responded to his simple command, and described the next room to him. It amazed me. I remember sitting there and thinking, "How did they do that?"

    I mean, it was just a stupid box, with a keyboard and a cassette drive. It couldn't think. But there it was, responding to him as if it could think. And he could type commands that were, for the day, fairly close to English. "Eat food." "Quaff potion." "Open door."

    I remember sitting there, thinking about that and being relentlessly tormented by it. I had to know. How could an inanimate box like that do things like that? How did it know what room he was in? How come the rooms changed every time we played the game? How did it decide if the potion killed him, healed him, or made him sick? How did it decide what color the potion was? How did it decide what was in the room? For a stupid box with no brain, this thing was pretty damned smart.

    And then, one day, Dan got stumped by the game. Something, apparently was wrong. A few years later, I'd realize he'd found a bug. So, he fired up a program, and cracked open the game's source code. And there, before my eyes, was the big secret. It was line after line after line of source code: carefully written instructions that told the stupid box exactly what to do. From those cryptic instructions, written in some obscure language called BASIC, you could make that TRS-80 do amazing things!

    That was the beginning of the end for me. I had to master that language. I had to know how I, too, could command a stupid, brain-dead box and make it do amazing things. It wasn't long before I had obtained a copy of the language reference for BASIC and taken a computer programming course at our High School. (Yes, we had them, even out in the sticks in Fontana.)

    So, in a way, ZORK made me a programmer.

    It's been about twenty-five years since I watched Dan crack open the source code to ZORK, and the path of my life was irrevocably altered. Up until that point, I didn't really have any real aspirations. I don't think I really did after that, either. But one thing became very clear: more than any other endeavor to which I applied myself, computer programming proved itself to be my one enduring passion. All these years later, I still have those moments reminiscent of that first day. I'll see a beautiful piece of code, a website design, or an application, and I'll think, "How did they do that?" Any ideas I may have had about leaving software development will be blown away and my passion for software will be rekindled.

    It's because I have to know. I can't walk away from these damned stupid boxes without being able to make them do amazing things.

    There's a certain, childish delight in figuring out the solution to a problem, or finding a new way to do something. For me, it's like Christmas, and I want to share that joy with others. Sadly, a lot of folks don't understand it--especially if they're not in the same field. But anyone who's done this work, and ever had a EUREKA! moment knows exactly what I'm talking about.

    Somewhere, right now, a budding young developer is experiencing his or her first time. They're being bitten by the bug. It's an infection that will take hold and set in for life. For most of us, it's a turbulent ride, filled with ups and downs, and we frequently consider leaving the field. For others, it's pure hell, and we leave it too quickly; for a lucky few, it's nirvana all the way through. I'm not sure I envy the lucky few; I rather like the way my challenges have tempered me over the years.

    When you face challenges, think back on what it was about software that caught you in the first place. Think back to your first time. Then think about the many times you've been lured back to it by your own passion. Not because someone offered you money, or material goods, or power, or prestige; think back to those times that your personal passion for software kept you in the game. Then ask yourself why you feel so passionate about software. The answer for me was surprising: I'm not really doing it for anyone else, but because I have to know, and because I have to conquer the stupid box.

    For all my noble aspirations, that's a humbling admission.

    But that passion is still there. It keeps me in the game. And, in retrospect, it's likely why I feel so passionately about software quality. It's not enough that it works, it has to work well.

    What was your first time like?

    Monday, March 24, 2008

    You are Not the Average Computer User

    John Lilly, the CEO of Mozilla, recently blogged about Apple's practice of including a new installation of Safari in Apple's Software Update service, even if you didn't have the application installed in the first place. You can read the full article here. His main point was this: As a matter of trust, update software should update previously installed applications, and not install new applications. Apple pretty much violated that trust when it presented users with this handy little dialog box:

    The main issue here is that Safari is not already installed on the end-user's machine. So, the option is not an update, but a fresh download of brand new software. Further, the option is checked by default, and the button in the lower right hand corner clearly says "Install 2 items".

    Now, I'm not going to rehash the pros and cons of Apple's tactics in this matter, because that argument has been debated endlessly on John's blog and on Reddit. What I am going to take issue with is the arrogant presumption that many commenters take when they make these sorts of statements:

    "I don’t see what the problem is here. If you don’t want the software, you uncheck the box. The product description is listed very clearly in the window, no extra clicking required."

    Omar

    "I don’t see the big deal. They are promoting their software through their software update program. It’s automatically checked…ok, so? Lots of update programs automatically check everything anyway, not just apple.

    "If FF is better then people will use FF. If they like safari then they will switch. These browser “loyalty” wars are getting old. IE came with windows by default and FF is still gaining ground. It is gaining ground because it is better. Just keep making a better browser and stop worrying about this. ppl will flock to the best. We’re not stupid."

    Chris

    "Oh fer heaven’s sake, uncheck the box and get over it. Are you saying the majority of Windows users of iTunes are too clueless to look and see what they’re downloading? OK, I’ll admit it’s a bit pushy of Apple but beyond that I fail to see what all the fuss is about."

    Anne

    These are knee-jerk responses. The last one, in particular, is an exemplary case of a poster who clearly doesn't understand the idea that users who read or post to tech blogs or forums are not typical computer users. If you're reading this blog, you're not a typical computer user. (I'm not sure what you are, exactly, but you're not typical.)

    Apple's case is interesting because of the enormous success of the iPod, and the vast number of iPod owners who use Windows. Those users will download iTunes so that they can use their iPod with their computer to purchase music and manage their playlists. However, the vast majority of those people are not what we would classify as tech savvy users. Rather, I'd call them click-through users, who implicitly trust the software vendor to make decisions for them. Think about your mom, your dad, your sister, your brother, your aunt, your uncle, the kids at school, the clerks at the nearest retail outlet or fast food joint, your fellow students, or your nontechnical coworkers.

    Those people represent the average computer user. They are click-through users.

    A couple times a year, I get calls from my family members about their computers. Inevitably, they'll tell me that the computer is suddenly horrifically slow, and that they need me to fix it. So they bring it to me, and I look at it, and it has tons of mystery software on it. I like to have them sit with me when I'm going through it, so that I don't remove anything that they might actually need or use. Nine times out of ten, they'll tell me, "I don't know where that came from." Apple's software update for Safari is likely going to produce an awful lot of these scenarios, because the the average computer user will have just clicked through the dialog, trusting that Apple knew what was best for them.

    A tech savvy user isn't likely to just click through that dialog box because they know what can happen, and they're pretty darned picky about what goes on their machine. They don't blindly trust the vendor to make those decisions for them. But the number of users like that is relatively small, and is hardly representative of the world's population.

    But the world is full of click-through users. There are far more of them than there are of us. Thinking for one minute that everyone thinks and/or behaves as we do is naive, shortsighted, arrogant and presumptuous.

    Again, my point here isn't that Apple was right or wrong. My point is this: never assume for one minute that YOU represent the average computer user. You don't.

    • If you're smart enough to competently read or post an a technical blog or forum, you're not an average computer user.
    • If you know how to correctly fix someone else's machine after they've borked it, you're not an average computer user.
    • If you know the difference between a hash table, a binary tree, and a linked list, you are not an average computer user.
    • If you know what recursion is, you are not an average computer user.
    • If you know how to safely overclock your machine, you're not an average computer user.
    • If you read technical books like they're gripping, fast-paced murder mysteries, you're not an average computer user.

    This list is undoubtedly incomplete, but I haven't had enough coffee yet. But you get the point.

    So, enough with this arrogant presumption. Stop assuming that all users behave as we do. Because the simple truth is that the vast majority of users do not behave or think as we do. They trust; we suspect.

    Thursday, March 13, 2008

    NValidate: Misunderstood from the Outset

    Occasionally, I will post questions about the design or feature set of NValidate on Google Newsgroups. More recently, I posted a question about it to LinkedIn. Almost immediately, I got this response:

    I'd suggesting looking at the Validation Application Block portion of the Enterprise Library from the Microsoft Patterns and Practices group.

    Now, I'm not belittling the response, because it's perfectly valid, and the Validation Application Block attempts to solve essentially the same problem. But when I talk about NValidate, which I find myself doing a lot as I interview for jobs (it's listed on my résumé), people often ask me questions like it:

    1. How is that any different from the Validator controls in ASP.NET?
    2. Why don't you just use the Validation Application Block?
    3. Why didn't you go with attributes instead?
    4. Why didn't you use interfaces in the design?
    5. Why not just use assertions instead of throwing exceptions?

    These days, I find myself answering these questions with alarming frequency. It occurs to me that I should probably get around to answering them, so I'm going to address them here and now.

    It helps, before starting, to understand the problem that NValidate is trying to solve: Most programmers don't write consistent, correct parameter validation code because it's tedious, boring, and a pain in the neck. We'd rather be working on something else (like the business logic). Writing parameter validation code is just too difficult. NValidate tries to solve that problem by making it as easy as possible, with a minimal amount of overhead.

    Q. How is NValidate any different from the Validator controls in ASP.NET?

    A. The Validator controls in ASP.NET can only be used on pages. But what if I'm designing a class library? Isn't it vitally important that I make sure I test the parameters on my public interface to ensure that the caller passes me valid arguments? If I'm not, I'm going to fail spectacularly, and not in a pretty way. You can't use the Validator controls (RangeValidator, CompareValidator, and so on) in a class library you're writing that's intended to be invoked from your Web application.

    Q. Why don't you just use the Validation Application Block?

    A. This one's pretty easy to answer. NValidate is designed to accommodate lazy programmers (like me).

    Here's the theory that essentially drives the design of NValidate: Developers don't write parameter validation code with any sort of consistency because it's a pain in the neck to write it, and because we're in a big hurry to get to the business logic (the meat and potatoes of the software). Let's face it: if the first chunk of the code has to be two to twenty lines of you checking parameters and throwing exceptions, and doing it all over the place, you'd get tired of doing it, too. Especially if that code is extremely repetitive.

    if(null == foo) throw new ArgumentNullException(foo);
    if(string.Empty == foo) throw new ArgumentException("foo cannot be empty.");
    if(foo.length != 5) throw new ArgumentException("foo must be 5 characters.");

    We hate writing this stuff. So we skip it, thinking we'll come back to it later and write it. But it never gets done, because we get all wrapped up in the business logic, and we simply forget. Then we're fixing bugs, going to meetings, putting out fires, reading blogs, and it gets overlooked. And the root cause is because it's tedious and boring.

    I'm not making this up, folks. I've talked to lots of other developers and they've all admitted (however reluctantly), that it's pretty much the truth. We're all guilty of it. Bugs creep in because we fail to erect that impenetrable wall that prevents invalid parameter values from slipping through. Then, we have to go in after the fact and add the code after we've got egg on our face and fix it, at increased cost.

    So, if you want to make sure that developers will write the parameter validation code, or are at least more likely to do it, you have to make it as easy as possible to do so. That means writing as little code as possible.

    Now, if we look at the code sample provided by Microsoft on their page for the Validation Application Block, we see this:

    using Microsoft.Practices.EnterpriseLibrary.Validation;
    using Microsoft.Practices.EnterpriseLibrary.Validation.Validators;
    public class Customer
    {
        [StringLengthValidator(0, 20)]
        public string CustomerName;
        public Customer(string customerName)
        {
            this.CustomerName = customerName;
        }
    }

    public class MyExample
    {
        public static void Main()
        {
            Customer myCustomer = new Customer("A name that is too long");
            ValidationResults r = Validation.Validate<Customer>(myCustomer);
            if (!r.IsValid)
            {
                throw new InvalidOperationException("Validation error found.");
            }
        }
    }

    A couple of things worth noting:

    1. You have to import two namespaces.
    2. You have to apply a separate attribute for each test.
    3. In your code that invokes the test, you need to do the following:
      1. Declare a ValidationResults variable.
      2. Execute the Validate method on your ValidationResults variable.
      3. Potentially do a cast.
      4. Check the IsValid result on your ValidationResults variable.
      5. If IsValid returned false, take the appropriate action.

    That's a lot of work. If you're trying to get lazy programmers to rigorously validate parameters, that's not going to encourage them a whole lot.

    On the other hand, this is the same sample, done in NValidate:

    using NValidate.Framework;
    public class Customer
    {
        public string CustomerName;
        public Customer(string customerName)
        {
            Demand.That(customerName, "customerName").HasLength(0, 20);
            this.CustomerName = customerName;
        }
    }

    public class MyExample
    {
        public static void Main()
        {

            try
            {

                Customer myCustomer = new Customer("A name that is too long");

            }
            catch(ArgumentException e)
            {
                throw new InvalidOperationException("Validation error found.");
            }
        }
    }

    A couple of things worth noting:

    1. You only have to import one namespace.
    2. In the property, you simply Demand.That your parameter is valid.
    3. In your code that invokes the test, you need to do the following:
      1. Wrap the code in a try...catch block.
      2. Catch the exception and handle it, if appropriate.

    See the difference? You don't have to write a lot of code to validate the parameter, and your clients don't have to write a lot of code to use your class, either.

    Q. Why didn't you go with attributes instead?

    A. I considered attributes in the original design of NValidate. But I ruled them out for a number of reasons:

    1. Using them would have meant introducing a run-time dependency on reflection. While reflection isn't horrendously slow, it is slower than direct method invocation, and I wanted NValidate to be as fast as possible.
    2. I wanted the learning curve for adoption to be as small as possible. I modeled the public interface for NValidate after a product I thought was pretty well known: NUnit. You'll note that Demand.That(param, paramName).IsNotNull() is remarkably similar to NUnit's Assert.IsNotNull(someTestCondition) syntax.
    3. In NValidate, readability and performance are king. Consequently, it uses a fluent interface that allows you to chain the tests together, like so:

      Demand.That(foo, "foo").IsNotNull().HasLength(5).Matches("\\d5");

      This is a performance optimization that results in fewer objects created at runtime. It also allows you to do the tests in a smaller vertical space.

    My concerns about attributes and reflection may not seem readily apparent until you consider the following: it's conceivable (in theory) that zealous developers could begin validating parameters in every frame of the stack. If the stack frame is sufficiently deep, the costs of invoking reflection to parse the metadata begins to add up. It may not seem significant yet, but consider the scenario where any one of those methods is recursive; perhaps it walks a binary tree, a DOM object, an XML document, or a directory containing lots of files and folders. When that happens, the costs of reflection can become prohibitively expensive.

    In my book, that's simply not acceptable. And since, as a framework developer, I cannot predict or constrain where a user might invoke these methods, I must endeavor to make it as fast as possible. In other words, take the parameter information, create the appropriately typed validator, execute the test, and get the hell out as quickly as possible. Avoid any additional overhead at all costs.

    Q. Why didn't you use interfaces in the design?

    A. I go back and forth over this one all the time, and I keep coming back to the same answer: Interfaces would tie my hands.

    Lets assume, for a moment, that we published NValidate using nothing but interfaces. Now, in a subsequent release, we decided we wanted to add new tests. Now we have a problem. We can't extend the interfaces without breaking the contract with clients who are built against NValidate. Sure, they'll likely have to recompile anyway; but if I add new methods to interfaces, they might have to recompile lots of assemblies. That's something I'd rather not force them to do.

    On the other hand, abstract base classes allow me to extend classes and add new tests and new strongly typed validators fairly easily. Further, it eliminates casting (because that's handled by the factory). If, however, the system is using interfaces, some methods will return references to an interface, and some will return references to strongly typed validators, and some casting will have to be done at the point of call. I want to eliminate manual casting whenever I can, to keep that call to Demand.That as clean as possible: the cleaner it is, the more likely someone is to use it, because it's easy to do.

    Q. Why not just use assertions instead of throwing exceptions?

    A. This should be fairly obvious: Assertions don't survive into the release version of your software. Additionally, they don't work as you'd expect them to in a Web application (and rightly so, since they'd kill the ASP.NET worker process, and abort every session connected to it. [For a truly educational experience, set up a test web server, and issue a Visual Basic Stop statement from a DLL in your Web App. You'll kill the worker process, and it will be reset on the next request. Nifty.]).

    Wisdom teaches us that the best laid plans of mice and men frequently fail. Your most thorough testing will miss some points of your code. The chances of achieving 100% code coverage are pretty remote; if you do it with a high degree of frequency, I'm duly impressed (and I'd like to submit my resume). But for the rest of us, we know that some code never gets executed during testing, and some code gets executed, but doesn't get executed under the precise conditions that might reveal a subtle defect. That's why you want to leave those checks in the code. Yes, it's additional overhead. But wouldn't you rather know?

    In Summary

    Sure, these are tradeoffs in the design. But let's keep in mind who I'm targeting here: lazy programmers who are typically disinclined to write lots of code to validate their parameters. The idea is that we want to make it so easy that they're more likely to do it. In this case, less code hopefully leads to more, which (I hope) leads to fewer defects, and higher quality software.

    Tuesday, January 29, 2008

    SOX Compliance and the Waterfall Method

    So here I am, working away at my job, coding in a vacuum as always, a development team one. Some things never change. But other things, inevitably, do.

    Our company was recently purchased. The new company has grand plans to eventually go public. With its eyes set on that prize, they have hired a consulting firm to help them achieve SOX compliance. This firm (who shall remain nameless) is busily churning out reams of process drafts to help us in that endeavor and submitting them to us for approval. It was only a matter of time before the SDLC for software development arrived on my desk for review.

    Now, for those not familiar with how things run at our company, I'll simply refer you to this post, which rather succinctly sums it up. While some things have changed, most things, by and large, remain status quo. I have managed to convince them of the value of hiring temp testers prior to releasing builds, so we've shown moderate improvement there. But I'm still wearing tons of hats, and completely driving the entire development process single-handedly. And the company adamantly refuses to hire any other developers to help out. It's also worth noting that since the acquisition, the number of new software projects piling up on my to-do list is rapidly approaching the double-digits. So any process that these guys throw at me is going to affect me and it's going to affect me pretty damned profoundly.

    You can imagine my utter shock and amazement when the plan that was presented to me for review and acceptance clearly stated that we were to implement, in excruciating detail, the Waterfall Method.

    This presented several problems to me right off the bat:

    1. Whoever presented this plan is clearly unaware of the fact that the waterfall method clearly doesn't work. The very man who initially described it (Winston Royce), pointed out quite clearly that it doesn't work, and suggested an iterative model as a clearly superior alternative. See the actual article for proof.
    2. There is no way that we'd be able to implement that process with a development staff of one person. The process outlined requires that the roles are separately defined and filled by distinct individuals. We don't have individuals to fill those separate roles, and the company refuses to hire them.
    3. Even if we did implement the process, the timeline to implement software solutions for our customers would become so bloated that the customers would drop us like a rock. Our biggest customer demands a release every three months. If we adopted the Waterfall Model as it's spelled out in the SOX compliant process they submitted, it would take three months just to spec out the iteration. Not that the model would permit the iteration.

    Consider this quote, dated in 2004, for crying out loud:

    Asked for the chief reasons project success rates have improved, Standish Chairman Jim Johnson says, “The primary reason is the projects have gotten a lot smaller. Doing projects with iterative processing as opposed to the waterfall method, which called for all project requirements to be defined up front, is a major step forward.”

    In his blog entry, Waterfall Method: A Colossal Blunder, Jeff Sutherland points out the following interesting tidbits in his comments:

    The Waterfall process is a "colossal" blunder because it has cost 100s of billions of dollars of failed projects in the U.S. alone. Capers Jones noted 63% failure rates in projects over 1M lines of code in 1993. By the late 1990's, military analysts were documenting a 75% failure rate on billions of dollars worth of projects. In the U.K. the failure rate was 87%.

    ...

    Let me reiterate, for projects over $3M-$5M, the Waterfall has an 85% failure rate. For those projects that are successful, an average of 65% of the software is never used. The Waterfall is a collosal blunder. The most successful Waterfall company I have worked with had a 100% Waterfall project success rate with on time, on features, and on budget. This led to a 100% failure rate in customer acceptance because the customer's business had changed or because the customer did not understand the requirements.

    In his article, Improve Your Odds of Project Success hosted to SAP NetWeaver Magazine, David Bromlow provides the following chart that shows how the Waterfall Method makes it difficult to start effectively managing project risk until much later in the project compared to more agile methodologies:

    In their article, From Waterfall to Evolutionary Development (EVO), Trand Johnson and Tom Gilb had this to say:

    After a few years with the Waterfall model, we experienced aspects of the model that we didn’t like:

    • Risk mitigation was postponed until late stages;
    • Document-based verification was postponed until late stages;
    • Attempts to stipulate unstable requirements too early: change of requirements is perceived as a bad thing in waterfall;
    • Operational problems discovered too late in the process (acceptance testing);
    • Lengthy modification cycles, and much rework;
    • Most importantly, the requirements were nearly entirely focused on functionality, not on quality attributes.

    Others have reported similar experiences:

    • In a study of failure factors in 1027 IT projects in the UK, scope management related to Waterfall practices was cited to be the largest problems in 82% of the projects. Only approximately 13% of the projects surveyed didn’t fail (Taylor 2000);
    • A large project study, Chaos 2000 by The Standish Group showed that 45% of requirements in early specifications were never used (Johnson 2002).

    Finally, I'll offer this, from the article Proof Positive by Scott Ambler in Dr. Dobb's Journal:

    Agility’s been around long enough now that a significant amount of proof is emerging. Craig Larman, in his new book Agile and Iterative Development: A Manager’s Guide (Addison-Wesley, 2003), summarizes a vast array of writings pertaining to both iterative and incremental (I&I) development, two of agility’s most crucial tenets, noting the positive I&I experiences of software thought leaders (including Harlan Mills, Barry Boehm, Tom Gilb, Tom DeMarco, Ed Yourdon, Fred Brooks and James Martin). More importantly, he discusses extensive studies that examine the success factors of software development. For example, he quotes a 2003 study conducted by Allen MacCormack and colleagues, to be published in IEEE Software, which looked at a collection of project teams of a median size of nine developers and 14 months’ duration. Seventy-five percent of the project teams took an iterative and incremental approach, and 25 percent used the waterfall method. The study found that releasing an iteration’s result earlier in the lifecycle seems to contribute to a lower defect rate and higher productivity, and also revealed a weak relationship between the completeness of a detailed design specification and a lower defect rate. Larman also cites a 2003 Australian study of agile methods, in which 88 percent of organizations found improved productivity, 84 percent experienced improved quality, 46 percent had no change to the cost of development, and 49 percent lowered costs. He also cites evidence that serial approaches to development, larger projects and longer release cycles lead to a greater incidence of project failure. A 2001 British study of 1,027 projects, for example, revealed that scope management related to waterfall practices, including detailed design up-front, was the single largest factor contributing to failure, cited by 82 percent of project teams.

    So, with all this overwhelming information at our disposal (which is just the little bit I could scrape up with Google in about an hour), and years of historical evidence that proves empirically that Waterfall doesn't work, why on earth would you wield impose it as the one and only process to be used for all projects, regardless of size or complexity across your entire organization?

    It's like voluntarily picking up a cursed +4 Vorpal Sword of Mighty Cleaving: it chops your own head off the moment you touch it.

    It's sheer, absolute lunacy. Particularly in our case, where we lack the time, the resources, or the desire to acquire the resources to properly implement it as it's written. We'll be bogged down in a bureaucratic quagmire of Dagoban proportions.

    You'll have to forgive me if this seems like a rant. But that's exactly what it is.

    It might be time to brush off that resume. Sometimes, enough lunacy just piles up that you start to realize that there's no one behind the wheel who has any firing synapses in the brain.

    Friday, January 25, 2008

    Is VB.NET vs. C# Really Just Syntactic Sugar?

    I recently read somewhere, as I have read before, that there aren’t any really compelling differences between C# and VB.NET. As has often been repeated, the differences all really boil down to “syntactic sugar.” C# is nice and terse, deriving its tight syntax from C and C++, while Visual Basic uses verbose language in an attempt to achieve greater clarity. Once the compilers get the code, though, it’s all supposed to be the same MSIL that gets generated, because you’re targeting the same .NET Framework.

    So, it’s been asked, why would you choose one over the other?

    That’s a fairly intriguing question. I’ve been working with VB and VB.NET for a really long time, and I’ve also had the opportunity to work with C, C++, Java, and C#. I like them all. I’d have to say that you can’t really beat VB for getting something up and running really damn fast.

    But I’ve started to get this really deep-seated gnawing in the pit of my gut about what kinds of bad habits I’ve picked up over the years. VB has a reputation for doing things “automagically” to ease your life for you. Implicit type casting, dynamic variable allocation, case insensitivity, and a host of other little time-savers are designed to shield you from the nitty-gritty details of brain-cramping compiler complexities.

    As a thought experiment, I took a large code base here at the office and ran it through C-Sharpener, a utility that converts VB.NET code to C#. Now, as a rule, I figure I write fairly safe code. I try to avoid reliance on the Microsoft.VisualBasic namespace, and use the Framework code instead. I always use Option Strict On (except for one particular class that used Reflection), and always explicitly define my variables. I’m a huge fan of type safety, so that wasn’t a concern to me.

    What I didn’t expect to find were the things that C# complained about that I’d been doing and it told me, in no uncertain terms, were foolishness.

    For instance, in Visual Basic, this is perfectly acceptable:

    Imports System.Diagnostics
    Dim log As New EventLog("Application", Environment.MachineName, "MyApp")
    log.WriteEntry("MyApp", "Message Text", EventLogEntryType.Information)

    (Ignore the crappy code. Just focus on the point I’m making.)

    This will get you a wrist-slap from the C# compiler. Why? Because that particular overload of the WriteEntry method is static. You can’t invoke static methods from an instance variable in C#. The compiler flatly refuses to let you do so. Visual Basic, on the other hand, thinks that’s just fine and dandy; it resolves the issue on the fly for you.

    Does that sound like syntactic sugar to you?

    In Visual Basic, this is just fine and dandy:

    If CInt(txtQuantity.Text) Then
       ' Do something spectactular
    End If

    Visual Basic helpfully converts the result of CInt to a Boolean to help evaluate the If...Then statement. If it’s nonzero, you get True and something spectacular happens.  In C#, you get a lovely compiler error about not being able to cast an int to a bool. Why? Because an int isn’t a bool, stupid!

    "Yeah, yeah. So what? But I always want to do that." Good. So prove it. Explicitly cast, and for Pete’s sake work with the right data type. It shows me that you’ve thought about that when you wrote it. Visual Basic doesn’t force you to make your intent clear. C# does.

    if ( 0 != int.Parse(txtQuantity.Text) ) {
      // do something spectactular
    }

    Again, does that sound like syntactic sugar to you? Remember, intent != syntax.

    In Visual Basic, you can do this for days and the compiler will pat you on the back while you do it:

    Public Function FooBar() As Integer
       Dim result As Integer
       Return
    result  
       ' Do some more real work--that's unreachable
       Return result
    End Function

    Does the compiler care? Nope. Not a peep. C#, on the other hand, gives you this nifty warning: “Unreachable code detected.” Then it gives you the file name and line number where it’s at. It’s like your best friend saying, “Hey man, you really don’t want to do that.”

    There’s no way that’s just syntactic sugar.

    So here I am, looking at this project that I’ve converted, and I’m both pleased and shocked. Pleased because the number of conversion issues and errors is relatively minor. Shocked because I found myself doing things that I didn’t think I was doing. They just creeped up on me and seeped into me like bad habits all to often do.

    I’ve been wanting to make the switch from VB to C# for some time now. Doing this conversion turned out to be a good thing for one very compelling reason: it opened my eyes to the mistakes I’ve been making, the bad habits I’ve adopted. I’m sold on C# now as my full time language. I’ll miss the speed of development of VB, but if slowing down means I write higher quality code that contains fewer bugs that have to be squashed later at higher cost, isn’t that worth it?

    In the end, the point of this post is this: C# and VB are not simply different by the syntactic sugar that distinguishes them. The power of their compilers and the strictness of their adherence to OO principles also separates them. I’d wager a guess that it’s C#’s strictness that makes its compiler so much more powerful than VB’s. I certainly don’t see messages about unreachable code, expressions never being of the provided type, static method invocation, and so forth from VB. 

    So please. Don’t over-simplify the issue. View the languages for what they are, and use the one that’s appropriate for what you’re doing, and how you work. For me, I’m making the switch. It makes sense for me. It won’t for everyone. But I am, by definition, an obsessive-compulsive control freak. I demand to know what I’m doing wrong and then I want to ruthlessly correct it. And I can’t reasonably ask for a harsher taskmaster at this point than an unforgiving, absolutist object oriented compiler.

    Monday, January 14, 2008

    Curious Perversions in Usability

    We've all seen them, and we've all used them: applications foisted upon us by the well-meaning management masses who wanted us to conform to the standard in order to boost our productivity, or the latest whiz-bang website promising to revolutionize its niche market. While the product itself might actually solve a unique problem, or offer a plethora of enticing, well designed features under the hood, its user interface frustrates, confuses, obscures, and clutters.

    When it comes to user interfaces, drill this one simple idea into your mind: Simple, clean interfaces will win out over flash, pomp and circumstance every single time. Why? Because a user interface should get out of a user's way, it should not impede them, confuse them, obscure the information they need to find, or be cluttered with crap they're really not interested in.

    We all have different views about what makes any given piece of software more usable. But certain things tend to peeve users fairly consistently. These tend to be mine when it comes to Web pages:

    • Don't make me wait. Don't make the mistake of thinking that performance isn't a part of your user interface. There are always numerous ways to display the same piece of information, and some are faster than others. Given a choice between having to wait for a Flash or PDF download of a pretty picture, or a flat GIF/JPEG, which do you think most users would prefer? (And if you think Flash hasn't been proposed for this, think again.) Use the smallest, most compact presentation format that will get the job done right.
    • Don't make me scroll. Especially not horizontally. Smaller pages work better. Horizontally scrolling pages are counter-intuitive, and people tend to have a hard time shifting into a mode where they're comfortable scrolling in that direction. Sometimes, you simply can't get around it, and that's the exception rather than the rule. In those cases, under no circumstances should you move the main navigational controls off the screen. In Web applications, embed the scrolling content in scrolling DIVs (or other suitable controls) to ensure that users can still reach your navigational controls without having to page through the document.
    • Don't make me squint. Use a reasonable font size that even the visually impaired can comfortably read. Better yet, use a font size that scales when the user chooses a different size in the browser. Don't force your font size on the user, simply because not everyone has 20/20 vision.
    • Don't make me guess what language you've written that document in. Use a clear, legible, font. You may like your decorative fonts, but they're not suitable for body text, forms, or general deployment on Web pages. Most users won't have them, and they won't look the same. Use standard fonts.
    • Don't make me wonder if there's something written in any area of the page or screen. Don't use dark text on a dark background. Don't use light text on a light background. Strong contrast enhances legibility.
    • Don't hide important information from me. Place important information at eye level. Use font weights, color, and styles to emphasize important information. Place this information prominently on the page, where I can easily see it. Don't obscure it in the page.
    • Don't hyperlink everything on the page. A hyperlink should indicate that there's something worth investigating. If everything on the page is hyperlinked, the hyperlink loses its value, and I'll tend to ignore them. Hyperlink the important topics. If you need to hyperlink lots of topics, provide a section at the bottom of the page called See Also or References and include those links there.
    • Don't obscure or complicate hyperlinks. If you change the style of a hyperlink so that I don't know it's a hyperlink, I won't know what to look for. Don't overly complicate them. Hyperlinks are an established navigational paradigm for the Web (and even desktop software) and everyone knows what they are and how they work. Leave them alone. Users already know how to use them.
    • Don't make me jump through hoops to find the commands or features I need to use. Don't invent an entirely new way of navigating your web site or application. There are a number of existing navigational paradigms that are well established and with which users are very familiar: drop down menus, tree views, bread crumbs, tabs and commands, and so forth. Don't confuse users by making them learn something completely new.
    • Don't surprise me by reconfiguring the user interface when I do something. If I click a button or a menu command and the entire user interface changes, or entire menus disappear, we've got a problem. The user interface, and the navigational system in particular, needs to be consistent and predictable. If it's not, users will be playing a constant guessing game about what they can and should do next. Users playing a guessing game are dangerous users.
    • Don't baffle me with technical jargon or confusing messages.  When something goes wrong, or when I've done something wrong, communicate it clearly and concisely. Tell me what I can do about it. Recover gracefully. Don't just throw up some message box that announces "An error occurred. Press OK to continue." Duh. What should I do next? Should I tell someone? If so, whom? Is my data safe? Do I need to start over?
    • Use consistent language. Don't call it Cancel on one screen and Abort on another. Don't use Logon Name on one screen and Sign In on another. Be consistent. Establish a vocabulary and stick to it.
    • Don't waste my time prompting me in an intrusive way to take part in your survey. I'm not interested in taking part in your survey. Put the offer to take part in a prominent place in your site or program that isn't intrusive. If I'm interested, I'll take you up on the offer. Otherwise, I'm going to close the DIV because you were rude enough to cover up the content that I was looking for with your intrusive popup. The same goes for popup ads. (But we all know how well that's going to go over.)
    • Don't play sound or streaming video as soon as the page loads. If I want to see it, I'll start it myself. You're chewing up my bandwidth, thank you very much. If I'm from an area where that's a precious commodity, that's the height of rudeness. Give me the opportunity to start the sound or video when I want to and if I choose to do so. This includes all forms of linked and embedded media, including Flash.
    • Don't order me to get a better browser. You don't know what browser is best for me. I may like FireFox, IE, Safari, Opera, Navigator, or some as yet unnamed browser still emerging. You may be able to say that your site doesn't support browsers outside a certain set, but it is gauche to insist that your browser of choice is the one and only true browser, whichever browser that may be. Competition is actually good for the industry.
    • Don't assume that my monitor is as big as your monitor. Just because your company has a standard video configuration that supports 1024×768 doesn't mean that's what your users are configured for. A vast amount of users are still set at 800×600. This resolution isn't a matter of laziness, but of simple visual acuity: they can't see anything if it's at a higher resolution. Design for 800×600. Ensure your pages fit on a monitor at that resolution. Doing so means users don't have to scroll horizontally. It also means that your pages will print properly if the user hits the Print button from the browser and is printing in landscape mode.

    Yes, this is surely opinionated. Yes, I'm sure I'll take heat for it. But here's where I'm coming from: I've both used lots of Web sites, and I've had to design lots of Web pages for The Average Computer User(tm). For those users, all of these things have turned out to be true. Ask yourself why Google's search engine is insanely popular. It's not just that their search engine covers the vast majority of the Internet; most people don't know how to effectively use the search engine to get the results they really want. It's because their search page is so simple that it's almost pristine. It's foolproof. Type what you want in the box and click Search. The results pages come up and show you what matches your search criteria. It's simplicity defined.

    Apple's computers have always been lauded as a breathtaking departure from the technical complexity inherent in Windows. Their user interfaces are simple, clean, and easy to use. They're the hallmark of Apple's software. One could argue that the simplicity of Apple's user interfaces is what defines them more than their hardware. Again, simplicity prevails, because the user interface gets out of the user's way, and lets the user get her job done.

    This is what we should be striving for. Design a Web page that is simple, clean, and gets out of the user's way. Don't confuse them. Be predictable in the way you behave. Be forthright and clear in the way you communicate. Use strong contrasting colors, legible fonts and sizes, don't reinvent the Web navigation paradigm, keep the navigation system where users can reach it, and avoid technologies that will degrade the user's experience.

    A Web site can do all that and still be beautiful. CSS allows us to do that. There's no reason you can't be clean, predictable, communicative, unobtrusive, and beautiful all at the same time. You just have to choose to do so. And you have to put your users' needs above your own desire to use the latest flashy, slow, whiz-bang technologies that don't really get you anything more than older, stable, less impressive technologies that accomplish the same thing.

    Who's Testing Your Software?

    There's a common mistake in software development: trusting the developers to test the software. Historically speaking, developers are the worst kind of testers, because we tend to use the software only as we designed it to be used. It takes a special kind of developer to be able to think outside the box and think like a user with little or no computer savvy.

    In the comment thread to the article, Microsoft Admits Vista Update Glitch, one poster made this point:

    Beta testing is not getting the bugs out of software because they got the wrong people doing it. Don't use computer savy [sic] people to beta test, use people like my wife who don't have a clue what makes the computer work. She can discover any glitch in software code, guaranteed. Her gift also applies to use of TV remote controls, etc.

    To which came this reply (edited for brevity):

    This is the best answer I have read for several years. Beta testers are people who do not do things that cause problems, rather, they look for features and bugs that are sometimes not there...The best Beta testers are people who are not knowledgeable and those who don't know the difference of double or single click.

    These folks are referring specifically to Microsoft's beta tests for its operating systems (more specifically, for Windows Vista). But the general sentiment is true and universal: users who have never been exposed to your software in the first place, and have had little exposure to technology are frequently the best ones to determine whether or not it actually works. They have a disturbingly accurate ability to ferret out bugs that borders on the psychic.

    As developers, we like to believe that our software is rock solid, easy to use, painfully obvious, and bulletproof. A user who can't tell the difference between clicking and double clicking, or why it's a bad idea to keep lots of applications open at once on a machine with limited resources, is the prime candidate for testing your software. If it's a Web application, find someone who's rarely used the Web or who only uses it for the basics: IM and email. One thing that they'll be able to tell you right away is whether or not the user interface is actually usable. And if you think for one minute that you shouldn't be designing clean, minimalist interfaces for the lowest common denominator of user, you've probably never met the average computer user. There are far more of them than there are of us.

    We have some pretty interesting users for our Web applications. Some of them are fond of ignoring on-screen instructions. Tooltips, online help, field prompts, clearly written button text, user training...not much of that seems to make a difference. When all of that fails, what does the application do? How robust is it? How gracefully does it handle bad user behavior? For that matter, how gracefully does it recover from bad application, network, or hardware behavior? And does it alert the user to that kind of thing in a clear, friendly, and meaningful way?

    You can't determine that sort of behavior by trusting your developers or your unit tests to find them. Inexperienced users will find far more than your tech savvy users will. That's not to say that your testing team shouldn't include tech savvy users; it absolutely should. But make sure that you include novice computer and Web users in your testing team.