Monday, November 25, 2013

What is your problem? – Part 2: Real versus imaginary problems

To ask "What is my contribution?" means moving from knowledge to action.
The question is not: "What do I want to contribute?"
It is not: "What I am told to contribute?"
It is: "What should I contribute?"
Peter F. Drucker

In the first part I covered the general aspects of identifying and reporting problems. Now it is time to apply those concepts to my domain of choice: software engineering (minor apologies to the software gardeners out there, I will come back for you in a future posting) .

I use the Agile method as the backdrop because it has completely overtaken the field to the point of erasing debate on alternatives (defeated waterfall proponents are still called through the backdoor to fill the gaps with valuable contributions, but I digress) .

As a short Agile method recap, work is delivered in small iterations called stories and executed over relatively short intervals, called sprints. If you can tell a story while on the run (sprints, running, see what I did there?) , you know sprint durations can range from ear-bleeding single-week sprints to waterfall-bordering 8 weeks. Beyond 8 weeks, there be dragons and T-virus, walk backwards slowly towards the nearest door and avoid eye contact at all costs.

The drudgery of tasks…

A fundamental tenet of Agile method is that stories be written in the form of “As a [role name] I need [a feature] so that [I can achieve a goal]”.

I personally prefer to replace “achieve a goal” with “solve [one of my] problems”. There are far more people in this world facing immediate problems than people who have goals, and even for the people who have both, the immediate problems tend to grind one’s will and resources to execute on long term vision.

That is not to mean Agile should be cast as a reactionary method that can only tackle situations after they have become a problem, but that any complex project can be mapped to a mind-map of tasks, where each node can be represented as a problem to be solved.

… is no match for the challenge of a problem

The payoff for such mental gymnastics is that a problem statement engages, whereas a task dehumanizes, and success in software development hinges on engagement: on engagement between developer and customer, on engagement between user and solution, and as a more recent phenomenon, engagement amongst users.

It is part of many professions to walk into an engagement where the customer knows exactly what they need, are willing to pay for those services, watch you walk out of the door after completion and then deal with the next task in a master project plan.

That is invariably not true of software development for two main reasons: (1) our largely INTJ subtlety-loving personality is prone to invent dozens of different ways of achieving the same goal with dozens of distinct advantages and disadvantages for each choice, (2) we collectively get bored of those solutions more often than we should, reinventing the field every couple of years in a way that is utterly incompatible with what we once thought to be a good idea.

With all that said, as a prospective customer to a solution requiring software, whenever you engage a developer or a development shop, the first part of your homework is to be absolutely sure the problem you want solved is one of your top-most problems, lest you (or your company) may not sustain the motivation to see it through. As the person (or company) experiencing the problem, and this may seem outrageous since you are about to pay for contracted services, you will be integral part of the solution: there will be hard questions to be answered at some cost of time, intermediary solutions to be attempted, final validations to sign off, all activities that will require your attention and resources.

Know your problems…

As a prospective software provider, accepting a task at face value is a recipe for disaster.
A good software architect must know how to artfully act as a devil’s advocate while engaging a customer, not to blindly question motivations, but to understand why the customer needs a solution. In other words, a good architect will ask the customer “what is your [his] problem?”
I often hear from peers disgruntled with the fact a great idea was not accepted by a potential customer, at the same time failing to recognize the problem solved was not all that important to the customer.

… know thyself

In a previous project I joined a team which had developed an internal tool for analyzing log data from hundreds of products. At the time I was the enablement lead for the technology and really got behind it. We persevered for a long while to make our world-wide support team adopt the(internal) product. After a relatively extended period of … err…lukewarm responses, we changed our approach, meeting frequently with these support teams and also with another internal development team who was already successfully supplying tools to the support organization.

It became clear that our log analyzer tool, however sophisticated in what it could do with log files, required memory capacity that, although readily available to our development team, was unthinkable to a support engineer. This tool also had the ability, developed at great expense, to shift the memory requirements to a relational database to cut down on memory usage, but deploying and maintaining a relational database was equally unthinkable for an audience which had no expertise in managing such systems.

The question no one asked…

At the same time, meeting with the more successful internal development team revealed their key selling point to the support organization: their solution was based on a SaaS model and the support teams could access most of its function through web interfaces, avoiding the need for high-end systems and the costs of installing and maintaining new tools. Their tooling also integrated with another SaaS offering where customers submitted all the supporting information, including the all important log files, for any problem reported in the field.

In the end, building (or selling) a log analyzer to a support team which routinely performed log analysis seemed like a success story in the making, but it failed to recognize two key aspects:
  1. Their most common activity related to log analysis was to isolate error entries in log files, then use snippets of the log entry on Internet searches (incredibly effective) , a feature absent in our tool.
  2. All the information used by the support teams resided on virtual services, requiring only a web browser on their machines, which side-stepped the need for high-end systems.
At the time, and this was a remedial approach to not having asked the “what is your problem” question first, we decided to harvest the analysis internals from the tool, put it under the web-based interface already being used by the support team and surface the analysis results through a page that only contained warning and error messages, with a quick link to an Internet search based on the error message.

…is the problem no one had

Had we asked the harder questions first, the answer would have been a SaaS version of the patent we filed a few months later, where log files submitted to customers were automatically analyzed, cross-searched on websites, and results ranked according to their rate of incidence in results. At that point I left the team for other reasons, but I am told they kept on delivering on that vision.

Something fantastically useful eliminates a problem that is really at the center of someone’s attention, and not something you set out to improve, however successfully. Success also involves far more than technique and technology. In fact, too much technology may just put the solution out of reach for the target audience, in requiring system upgrades, training, and adaptation.

In closing, I end with a quote that symbolizes the consequences of offering a solution on the basis of vision without sufficient understanding of the problem, resulting in one of the costliest mistakes of the kind in recorded history:

We could hardly dream of building a kind of Great Wall of France, which would in any case be far too costly. Instead we have foreseen powerful but flexible means of organizing defense, based on the dual principle of taking full advantage of the terrain and establishing a continuous line of fire everywhere.André Maginot
December, 1929