User Activity Measurements - Part One

Feedback where it matters - Production

I am a big fan of closing feedback loops. Doing business with open feedback loops just feels like running around with untied shoes and being oblivious as to what is causing you to continue to fall on your face. There are plenty of feedback loops in development and there is much effort expended on closing and optimizing them. Compilers, automated tests, and code metrics are a few examples of most developer's favorite feedback loops. But if we step back and look at the applications we are building from a broader perspective we can see that a feedback loop that might get neglected is how our application is used in production.

Optimizing for the real world

The dirty secret of software development is that people use the systems we build. Yuck. Our gleamingly shiny and perfectly logical and organized system interfacing with a fleshy pink human? What kind of sadistic person thought this would be a good idea? The horror. At least that's the impression I get from some developers. But it's not a slam on developers it's just a condition that occurs when complex systems are built with open feedback loops. Let's look to the automotive industry for an example.

Lab Tests

You work as an engineer for the Swedish Fjord car company. You and your team of crack mechanical engineers are tasked with building the suspension system for a new SUV. You design and calculate and build. You hook the suspension up to a dummy car chassis in the lab and bounce the chassis up and down. You learn some things about O-rings and hydraulic pressure and take that new information back to re-design and re-calculate and re-build. You repeat until the O-rings stop leaking and the hydraulic pressure stabilizes. Well done, high fives and pizzas all around.

Test Track

After your successes you bolt the suspension onto the rest of the prototype model and have the test driver take a few laps around the test track. The test driver eases into the rigorous test first making sure nothing explodes when he turns the key. With each passing test the driver's confidence increases. Pretty soon they are running through the test track so fast and so smoothly that the test driver is hooting and hollering as the tires squeal.

Ship It!

With such great results on the test track you start production on the car and customers start placing orders thanks to your amazing marketing. Your engineers have all moved onto other projects since their work is done for this model. Then the complaints start pouring in to the support engineers. Tip-overs, inability to stop, and other nightmarish events are occurring with your brand new SUV. Amid the maelstrom the support engineering team digs into the details and finds out what happened. The suspension was never designed to handle the real world usage. It turns out that the real world has roads that are bumpy, full of potholes, and slippery spots that cause less than optimal results when trying to stop, turn or avoid a bicyclist.

It Must Be Laziness

If you are blaming the engineers for a less than rigorous test process or the company for not investing in quality then you're missing the important bit.

Let

me

slow

down

and

say

...

the lab is not the real world.

Well Mr. Smarty Pants, Where Do We Go From Here?

There are many differences between automotive engineering and software engineering. I'll save the discussions on accountability, consequences of failure, and the fault investigation and findings reports which the software industry lacks at the moment for another day. Unfortunately in software it's all too common that the original engineers are off on a new project and have no idea what the impact of their decisions were that caused nightmares for the customers, business, and support engineers. Many times mistakes continue to be made and history repeats itself leaving a wake of semi-useless systems and legacy applications.

Therein lies the problem and the solution - closing the feedback loop between the designers and builders and the real world.

We have a couple of options for closing the loop though. We can make our lab and test track as realistic as the real world. With enough money and time we could simulate darn near every possibility and potentially add a bit of the random chaos that the real world seems so full of. Another option is to extend our lab into the real world. This isn't a plea to let me code in production and remove all the quality checks and discipline...although...no that's not it. It's more of a shift in viewpoint. Instead of looking at the release of software as a "throw it over the wall and walk away" task we need to look at it as the start of getting more feedback about our software. The code is compiled, unit tested, integration tested, performance tested, acceptance tested and all manner of tested and then we add the happy mess that is people into the mix and walk away? Hmmm, must be selective logic on the development team's part.

All of this sets the stage for some ideas I have rattling around that I would like to explore. Tune in next time for a ground breaking discovery. Or another jQuery plugin. It's like 50-50 at this point.

---