Monday, December 18, 2006

Fog of war, and colleagues turned enemies

US troops negotiate snow and ice during the battle on Attu in May, 1943."Fog of War" is a military expression used to represent the uncertainty in a theater of operations. The uncertainty, caused by incomplete information, obfuscates the judgment preceding important decisions.

A good leader is expected reduce the fog of war before making a decision and later to live with the consequences of that decision.

Using a WWII episode as an example, we can look at the Japanese invasion the Aleutian islands of Attu and Kiska. After suffering thousands of casualties while retaking Attu, the Allies did not realize that the Japanese had retreated from Kiska under dense fog formation. Without the information, the Allies mustered a costly force of 40,000 American and Canadian soldiers to land on the island. Eventually, some of those forces turned on each other, but more on that later.

In the office environment, similar situations occur when decisions must be made and little data is available. In the absence of data, people invariably resort to the next best thing, which are opinions based on previous experiences.

Just like in the Aleutian Islands, even when the situations may seem similar and cause the illusion of absolute certainty, the technologies underpinning our collective discussions change by the minute. The changes manifest themselves in different ways. Sometimes, a technology may have matured enough to tip the balance used in previous decisions; or new understanding of the consequences of a previous decision may have changed the thought process for solving those kinds of problems. Just think of how many people declared the Java programming language a toy when it first came out on 1997 or derided Ethernet networks as theoretically inferior to Token Ring networks. Those assertions had their merits, but lost their ground as these technologies matured.

One may argue that discussions can always happen with civility, but for as long as one's opinion is based on one's experience, that opinion is indelibly associated with its author. The proponent of an opinion is indirectly putting his reputation behind it. Often, proving an opinion wrong means proving its author wrong. Given enough time and a protracted argument with opposite opinion, colleagues may soon finding themselves defending their ideas and reputations at the same time.

And it can happen to any of us. Creating a new model and collecting specific data to make every single decision in our professional lives is impractical. We will offer opinions and be wrong. We must make peace with the fog of war as it is everywhere. The office version of the fog of war cannot be entirely avoided, but can be reduced to the point where people can still see each other as allies. Negotiation and decision making are sufficiently mature disciplines and some techniques go a long way towards thinning the fog of war.

When facing a choice that requires consensus, the first step is to have all parties to agree on the factors that are important for that decision. The second step is to collect as much data about the alternatives. That collective activity achieves two purposes, (1) it creates a shared framework of perspectives as each party explains why certain factors are important to them; and (2) the newly collected data validates their frames of mind, separating opinion from fact.

There may be conflicts of interest but they will show up early in the process as areas to be worked out instead of manifesting themselves in meeting standoffs.

No matter what technique suits you, it is important to walk into every situation accepting that your experiences may no longer be valid and to be prepared to force a collective back down when diverging opinions start to dominate the discussion. That may be a good point to get the parties to agree on the lack of a model to explain the situation and the lack of data to justify either position.

Walking in armed to the teeth amidst a dense fog without enough information about the situation is extremely dangerous as can attest the troops that landed on Kiska. The Allies faced terrible weather with low visibility, still reeling from the grievous losses incurred in Attu and carrying the chilling memories of hand-to-hand combat with Japanese soldiers. When the situation, experiences, and memories came together, a few of the nervous troops eventually mistook their allies by Japanese platoons and opened fire on each other.

Over 300 lives were lost in a fog of lack of data and bad assumptions.

Wednesday, December 06, 2006

Office politics, ethics, and the darker shades of gray

PlatoA recurring theme in conversations with friends in the industry is a dislike for politics mixing into personal evaluation systems used in large corporations. In this entry, I wanted to explore the subject under rational lenses to separate real ethics from emotional perceptions.

Many good people have willfully curbed their career potential for fear of losing their immortal souls in a sea of corrupt office mates. Many more good people have inadvertently incurred in the same mistake by not making others aware of their presence. Somehow I felt the majority in both camps have made such decisions without analyzing office politics, at the same time causing more harm than benefit damaging to the business.

There are many equally good people who have entered office politics and hardened their souls to withstand the pressures. Not to be left alone in their pain, people in this group will often shed portions of that pain onto the first group. More frequently than not, the 'sharing' comes courtesy of a rather cynical statement of politics as an unavoidable fixture in the office scene.

There are good politics and bad politics, and it has been my experience that good people practice good politics. They have to become bad people before they start practicing bad politics and that transformation may take a long time. For instance, reviewing a 200-page specification for an architect in your area may be good politics (unless I have lost my ways and can no longer tell the difference); after all you are helping improve the quality of someone's work and may rely on his support later on. The returned favor may be a note to his peers endorsing a reasonable proposal you have in mind. I often think of it as creating time in someone's calendar for some task I need from them.

As a counter example, bad politics would be to negatively influence the reviewers of a proposal that could make your project unnecessary. It takes a lot of selflessness to make the right decision and working with author of the proposal instead; but if one's job solely depends on others being unaware of its uselessness, that may be a job not worth having.

PlatoThe world of politics has a way of pushing away the good people, and I am in good company while saying so. But it is the responsibility of the good people to resist the push and inject good politics into the system. Staying away from politics and proudly resenting others who don't is an act of passive-aggression. Going in and deriding those who choose to stay away is equally unproductive.

Now, for the difficult part, what about the situation where one compromises his integrity to benefit those under his responsibility?

A few years ago, in a leadership class, the instructor brought over a retired officer who had a strong opinion on the subject. He used his own example of playing golf with higher-ranking officers even though he disliked the game and some of the officers. At that point, the other students were directing rather unflattering eyes at him. His calculated pause lasted a few more seconds until he continued: "I did it for my men. When it rained hard in our base, the poorly built barracks invariably leaked water. I could bring in my golfing 'buddies' over, point to the leaks and get special attention on fixing them'.

There is saying "there are many shades of gray between black and white". That may be so, but some are darker than others.

Monday, December 04, 2006

The Battle of Borodino, and when you don't get to pick your battles

Ever heard the advice of "picking your battles"? Well, read on.

The Napoleonic wars lasted over two decades and changed the world in ways that still affect our lives. Most people remember the Battle of Waterloo as the final defeat in Napoleon's otherwise remarkable military career. Fewer will recall the Battle of Borodino, vividly depicted in Leo Tolstoy's War and Peace.

Move past the gory body count and you are bound to find a profound lesson in life.

Napoleon's strategy focused on decisive clashes with opposing armies, invariably inflicting heavy losses that forced an immediate surrender. After the word of these flash victories spread through the continent, his numerous enemies soon resigned to French rule.

On 1812, the French army marched inexorably towards Moscow and victory was certain as soon as the Russian army stopped its retreat and faced its destiny. The rest of Europe waited apprehensively for the fall of Alexander's empire. The Russian commander, General Kutuzov, a very experienced general that enjoyed the population support but not the Czar's respect, ordered a stand near the village of Borodino. In agreement with the collective expectation of all watching nations, the French army won the single-day battle.

But in the words of Tolstoy, even as the battle raged on, Napoleon's rule over Europe effectively ended that day. The news of the Russian stand would soon spread across Europe and reach France. The fierce resistance inflicted grievous losses on Napoleon's army; it was clear that equal resistance on future offensives would grind his ability to win through intimidation.

The lesson in life? Faced with an insurmountable challenge or in a difficult position, you may not win, but a strong show of interest and dedication is never wasted. If the problem is important enough, others will pick up from where you left and face that same challenge from a better vantage point. Losing well when there is no chance of victory is also a victory.

Of course it is all a matter of principle and how important and dear the subject may be to you. After all, in a single day, more than 45.000 Russians didn't walk away from Borodino.

Contrary to what you may be thinking, a final stand does not have to be an act of self-destruction. Kutuzov's army was well prepared and he eventually ordered a retreat, leaving Moscow unprotected against the Czar's will. He wanted to preserve his army's strength while the impossibly long supply lines and the rigorous Russian winter made short work of Napoleon's army. At the end of the campaign, the French positions throughout the continent deteriorated rapidly and 3 years later Napoleon saw himself stranded on an island in the middle of the ocean.

Kutuzov's reputation never matched the notoriety achieved by Arthur Wellesley, the Duke of Wellington, who led the Spanish resistance and the final victory against Napoleon. Nevertheless, Kutuzov's stand was the beacon of hope for the coalitions that faced Napoleon in the three-year span that separated Borodino from Waterloo.

I do realize the dissonant tone of this lesson. We live in a business environment where signs of discord are forcibly repressed through coaching, mentoring, and performance evaluations. Many people would wish to have a nickel every time they were told to pick their battles. That is not my advice. In my experience, on most of the times, you don't get to pick your battles; they are brought to your doorstep by someone with a disagreeable agenda.

Whether it is a Cowboy Coder explaining why unit testing is not required, or a creationist architect avoiding a formal requirements management process, the battle has been picked for you.

When all alternatives are weighted, all the negotiations failed, and you are faced with the need to make a stand, make it count. Be prepared and don't be stupid, as someone once told me: "The real heroes come back to fight another day".

Thursday, November 30, 2006

Ford Taurus, and innovation turned disaster

Since I took exception with the construction industry on my previous posting, I thought it would be fair to take a stab at my second favorite counterpoint to software quality problems: the auto industry.

To make the point, I resorted to the news of October 27th, where Ford Motor Company announced the end of the line for its (once) beloved Ford Taurus. After almost 7 million units sold, the automaker called it quits on one of the least known cases of success turned failure in modern industry.

The Taurus history begins at the end of 1985, with a rounded profile that blasted the boxy gaz-guzzling competition. After six years in production and facing mounting competition from increasingly better Japanese cars, the new cosmetic modifications to the 1992 model fueled a 5-year streak as the best selling car in USA.



And then tragedy. On the trail of that enormous success, Ford introduced the 2nd generation of the car, on 1996. In the less endearing terms used by Ford's engineering teams, it became known as "the car that could not be built". Its impossible combination of curved oval shapes colluded with the free falling quality control standards in the US auto industry to deliver the most severe blow to the company fortunes in decades.

The complicated assembly process forced the prices to go up and the body panels to go sideways (in opposite directions). On 1997, the car lost the top honors as the nations' best seller and forced Ford into the death spiral of pushing the cars at a loss. The idea was to keep the assembly plants running and bet on the sales of large trucks and SUVs. Meanwhile, its Japanese competitors worked overtime to meet the demand for their own best sellers. The sales chart hides the loss of favor amongst US consumers, and incorporates fleet-rental sales that accounted for the bulk of the Taurus sales towards the end of its life.

What happened during that 11 year span? Essentially Ford executives missed a market transition to where customers had overtaken product control from the companies. They decided to innovate from a shell, transplanting its oval emblem to everything in its best-selling model, from door handles to dashboard shapes. Deciding for the customers had worked in the past and lifted the company from a financial tail spin on 1986. Ten years later, the same recipe (go rounder than the competition) resumed the tail spin.

You can read more in this insightful book by Mary Walton: "Car: A Drama of the American Workplace".

Oh yes, what does it have to do with technology companies?

I think we all know. Of course, 20 years is an eternity for a technology product, but specialists in the industry all agree with the combination of increased complexity and lost quality as the source of Ford's impending demise. Insult added to injury, the Taurus '96s design was widely perceived as ugly by the press and the market. That ill-received design was somewhat rectified 4 years later, but its competitors countered it with even better all-new models packing engineering years ahead of what the cash-strapped Ford engineering teams could muster.

Tuesday, November 28, 2006

Road building gone awry

For those living in the RTP area, the gigantic failure of a recent resurfacing job in the I-40 interstate is already old news.

In the past I have cited the construction and auto industries as examples of more mature industries. In my view, these industries have better established practices and are more likely to achieve consistent results. Nevertheless, this example shows that they are not immune to problems when those practices are abandoned in favor of untested "innovative" techniques.

In this particular episode, the Department of Transportation decided to go with a concrete overlay procedure that had never been tested with such big traffic volume (100.000 cars a day). The DOT engineers lifted the procedure from a small road in the Raleigh area, made a few assumptions, and sold the idea to the state officials.

At a cost of U$18.6 million dollars, the reconstruction effort will involve the removal of 50.000 tons of concrete and subsequent resurfacing with regular asphalt. For the innovative employees, the consequences are expected to be just as dire, in the words of the Secretary of Transportation:
"My investigation revealed that a number of state DOT employees were at fault from 1999 through the completion of the project in 2004," he said. "Many of these employees have since retired or otherwise left the department. Others who still remain with the department will be subject to suspension, demotion or firing. "
Interesting and alien aspects to software engineers; ranging from a failure that cannot be explained away in terms of unrealistic expectations (the asphalt should last 30 years and is crumbling after only 2) to documented links between the failures and the people responsible for it.

In the presence of real liabilities, it seems that innovation should be restricted to the labs until proven. At least if one is building cars or roads.

Tuesday, November 21, 2006

The Hive Mind: gravity, magnetic fields and electricity

"I postulate the theory of the "Hive Mind", where individual or collective thought carries over great distances..."
A few weeks ago I explored the parallel between physics and corporate behavior, applying the concept of hysteresis to explain the mechanics underpinning the adoption of new technologies and processes within a large corporation. This week I wanted to ponder this interesting article mentioned on CNN, about a successful experiment involving people communicating through phones and e-mail.

The experiment showed that some people could actually predict the identity of a phone caller or email sender with a greater accuracy than simple statistics would dictate. The experiment lacked a more controlled environment to be considered scientific proof, but I somehow share the scientist's belief that there is an interconnectedness of all minds within a social grouping. If anything, there is a higher probability that we spend more time thinking about the same subjects as people in our social grouping than thinking about subjects that are not interesting to that group.

The shared interest on the same topics is part of the glue that holds a social grouping together, including the ones at the workplace. That shared interest is possibly a stronger bond in our diverse corporate environment because we are brought together around a shared goal and often lack the other demographic background that unites a social grouping outside of the work environment.

Without getting into the telepathic effects explored in the experiment reported by CNN, it comes as no surprise that sometimes we find ourselves thinking about writing an email to someone and are suddenly interrupted by that same person calling on the phone or IM. I am willing to go beyond the inference that such phenomenon is the result of diligent statistics at work; offering a hypothesis that explains the (unverified) results observed by Rupert Sheldrake, the British scientist behind the experiment.

I postulate the theory of the "Hive Mind", where individual or collective thought carries over great distances and establishes a communication mechanism with other people beyond the currently known media; an effect that will be completely explained by (then) common physics years from now. Furthermore, I postulate that such magnetic field is also a component in the organizational realignment example introduced in my previous analogy between magnetism and corporations.

Our brains generate an elaborate wave pattern while working through any technical challenge; it is somewhat expected that the average human brain activates the same regions of the brain while solving similar problems. Now, those brain waves are formed by massive (for our body masses anyway) electrical currents traveling over thin conduits, which is bound to produce a coaxial magnetic field around these conduits. Magnetism, electricity and gravity are all related forces according to the Unified Theory for Electricity, Magnetism, and Gravity, which offers a more complete model for the physics underlying the "Hive Mind" hypothesis.

Magnetic and gravimetric fields travel unimaginable distances through a mechanism that has eluded the most brilliant scientific minds of all times and will demonstrably reach people across the globe. I could even entertain the idea of other kinds of unknown fields beyond these two but would do so at the peril of a weaker hypothesis. The effect on the people reached by these fields would largely depend on the magnitude of the field and the susceptibility of the receptor to the field. Newtonian physics show that the effect of the field diminishes significantly with distance (to the tune of distance3) but do not account for the repeating effects of intermediary brains resonating and amplifying the waves along the way.

Another interesting thought about the Hive Mind comes from the effects of mass in the laws of gravity. What is "mass" in the hive mind? The strength of the transmitter, or the intensity of the thoughts around a certain subject, are certainly a factor; but the resonating effects of the retransmitters (other people's brains) in the hive mind are bound to offset and surpass the strength of the original thought. In other words, the "mass" parameter in the intensity and reach of a particular line of thought is dictated by the number of people attuned to that line of thought.

The tribulations of our daily lives and the strength of our individual thoughts introduce a lot of background noise to the resonant waves of the Hive Mind; creating a somewhat healthy obstacle to a Borg-like conscience breeding out of the work place. The magnetic properties of corporate direction covered in the posting about hysteresis also explain some of the impediments to a freer flow of waves in the hive mind.

At any rate, the hypothesis of the Hive Mind is worth a thought, or many.

And if you were thinking about the exact same thing but never cared to write it down, do not be spooked; maybe I picked the idea from you ;-)

Angry monkeys and cargo-cults

A good friend of mine shared his experiences at the annual "No Fluff Just Stuff" conference. This event features presentations on the themes of Java and Open Source and is open for external participation. It spans several US states and in Raleigh is hosted at the SAS institute.

He was ecstatic about the keynote presentation by Dave Thomas, co-author of the Agile Manifesto and the best-selling "Pragmatic Programmer".

In his talk, Dave turned to the story of "cargo cults", the aboriginal religions grown in the South Pacific during WWII. The significance to software development was obvious even before Dave started to address it more directly on the topic of "cargo cult programming".

Dave also turned to the famous anecdote of "Angry Monkeys", where scientists supposedly started an experiment with five monkeys in an enclosed environment. One of the central elements in the environment was a banana-tree, which the monkeys immediately attempted to climb. Whenever a monkey attempted to climb the tree, the scientists hosed all of them with water until the offending monkey climbed down.

Over time, whenever a monkey attempted to climb up the tree again, even before the scientists reached for the water-hose, the other monkeys would readily beat the upwardly primate. After a few days, none of the monkeys attempted the banana-gathering stunt anymore.

Then the second part of the experiment starts: the scientists replaced, one at at time, the water-savvy monkeys with monkeys that had not been water-hosed before. The scientists replaced a watter-savvy monkey as soon as the new monkey ceased his attempts for banana sprints through the reassuring beating of the other monkeys.

In the end, once all the original monkeys were replaced, none of the new monkeys would attempt to reach for the bananas anymore, even though they had never been interrupted by a stream of water while trying to do so. They only knew that they should beat any monkey who attempted, no questions asked.

Now, I don't know about you, but how many of the original people that worked on your project are still around?

Dave's points in his speech were about how software developers will learn to accept self-evident truths and not attempt something new or beneficial because something has not worked before. The conditions in our industry change so quickly and technologies may have reached the absent maturity that made them fail in the past.

Thursday, November 16, 2006

Up on the wall: when your career decisions are not yours

Last week I attended a two-day class titled "Problem Solving-Decision Making-Critical Thinking". The course was divided in 5 modules, one of them about decision making. One of the exercises was about choosing a new project lead amongst 4 candidates. We were split into 3 different teams of 5 people to make that decision.

Each team had a large whiteboard to list the decision objectives, alternatives, benefits and risks. The three teams eventually arrived at the same conclusion by applying a formal decision process starting from the performance evaluations and availability data provided for each candidate. As we ruled out one candidate after another and assigned weights to their strengths and weaknesses, something dawned on all participants as revealed later during a lunch break conversation:

"It could have been any of us"

The thought of having your name up on the wall and your skills and traits (including personal ones) dissected by an objective decision process can be frightening. For one, you are not there to explain past incidents or how some perceptions may be inaccurate. Many of the evaluators misread the availability of one of the candidates and ruled him out in the first minutes of the session. Additionally, the majority of the participants assumed that the strongest technical person would not be a good influencer for the rest of team. For this last assumption, everyone went along with the second assumption that the chosen candidate could always consult with the runner up.

With more than 30 minutes, these makeshift "decision boards" probably could have requested more information and arrived at different conclusions. One of the participants argued that the people making the decision would probably be the managers of the candidates. Others dismissed that assumption in favor of another hypothesis, where the first-line managers would present their assessment to the mid-level managers actually responsible for the final call.

Regardless of how someone's name reaches the board, fair decision makers will slap numbers in front of pros and cons, tally up the scores and pick a name. If all the right data and evaluations are considered, the right person will get the job. The others? Well, they may never known that they have been up on the wall.

Sunday, November 12, 2006

DNA Hardware and the final frontier

I did not mean it literally when I wrote about Software DNA, but DNA hardware seems to be at hand: 'Alive and clicking, the computer made from life itself".

The notion of an organic computer reminded me of a Time Magazine article titled "What will replace silicon". It also points us eerily past the reality of an intriguing article featured the Fortune Magazine, titled "Quantum leap", where headbands enable the interfacing between the human brain and powerful and pervasive computers.

For some reason, the possibility of organic computers that can be injected and integrated into the human body has darkened my thoughts about the future of mankind. During WWII, the battles between Axis and Allies were fought over world domination as much as they were fought over the unwarranted genetic superiority of a race.

Decades later, Jared Diamond, in his excellent "Guns, Germs, and Steel", further challenged the myth of genetic superiority in favor of geographic and societal factors.

What darkened my perception of the future was the thought of what could happen when genetic superiority becomes tangible and measurable. Ethics, science, law, even religion, as we know them will be challenged in ways that will dwarf the controversy over the cloning of human beings or stem cell research.

Far from the romantic belief in the superiority of men over machine, hidden in the inscrutable complexity of our brains and the scientifically unproven existence of an omniscient God-like entity; the possibility of an organic computer completely sidesteps the discussion. Humanity does not have to be defeated, just subverted. It is a subtle argument, but once we can no longer separate between what we wear and what we are, then Jared Diamond's book will have ceased to be a treaty about the triumph of human potential and have become a book of history about the time when all men could evolve equally given the same resources and conditions.

Some could argue that the innocent usage of the technology should preserve our humanity, but then again, humanity is not known for moderation. No matter how much I try, I cannot avoid imagining the 'what if' scenarios explored in John Mnemonic (1995) and Gattaca (1997) .

As a fellow blogger once said, "I don't have to like the future to know it is coming". And it is coming fast.

Monday, November 06, 2006

Software DNA

On the trail of Creationism, evolutionism and intelligent design, I wanted to introduce the notion of software DNA, which can be used to measure the quality of a project. There are several somewhat expected conclusions in the following posting but it is interesting to see Mother Nature at work in what is otherwise an aseptic activity.

We can all accept the notion of a "DNA" signature for a product, a sophisticated mesh of everyone's contributions to the final outcome. The interesting part of the analogy starts from how these contributions are distributed over time. Product development is a somewhat pipelined series of activities, with each stage requiring the intense participation from different sets of people. As the assembly moves on the pipeline belt, those sets of people donate their "DNA" to what is going to be the final product, compressing several generations of evolution in a matter of a few months.

In biology, a diverse pool of genes is a crucial factor for a successful species. The diverse DNA combination increases the odds that one will survive the harshness of the environment, while nature callously weeds out the less adaptable variants. Back to the software analogy, that diversity is equally important, originated from contributions from people from different backgrounds, with different expertise and perspectives on how the product should be shaped. Before I continue, think of it the next time you feel your participation is too small to make a difference.

From a customer perspective, that evolutionary diversity is beneficial as it results in better products. Even the powerful "species" coming out of the Microsoft pipeline can have their survival threatened by random mutations on foxes, owls, and penguins.

Although companies will try to influence the environment to improve the odds toward their offspring, environmental control is become increasingly difficult with long-tail markets emerging everyday. When environmental control becomes difficult to manage, the only hope for a company is to revert back to its own resources, carve the best product out of its gene pool and hope that the environment will be kind to it.

I already explored the importance of diversity in the early paragraphs; now let's take a look at mutations. Skunkworks, research, and an emboldened community of innovators are a fantastic start because they represent the random mutations in the gene pool. Some mutations are more beneficial than others and the environment can be harsh with seemingly superior species (can you say Betamax?)

But there is one gene we want to preserve in large and healthy doses in the final species: the customer genes. They are (or should be) injected on the first cells of a product and preserved in the final species at all costs. Skipping the business modeling phase for a project is an almost fatal handicap to any species. The lack of those genes makes the species unattractive and as soon as the parents exhaust their capacity to incubate it, it becomes extinct.

The lack of formal requirement analysis will also push those genes through the cracks in the pipeline. Here is a less obvious point. The new software is already in contact with the environment while it is being created. Once the customer genes are injected into the initial species, they are visible in the pipeline and some may be unpleasant to the DNA contributors. Without proper enforcement, an ill-controlled environment may become hostile to certain desired characteristics.

I do not see bodily rejections of the customer DNA in the software creation process as a sign of corruption or shortsightedness, but rather as a natural occurrence. While we do want the final product to preserve the customer genes, we need to acknowledge that they make the creation process harder. Without good safeguards for those genes, they become easy prey to a mismatched set of localized liabilities and greater goals.

As examples, even with the foreseeable disastrous results, assuming what a customer wants instead of actually asking the customer takes much less time and effort. In a large organization with experienced architects, those kinds of assumptions tend to fall on the creationist side of my previous posting titled "Creationism, evolutionism and intelligent design". If you ever take time to read, you will agree that software creationism is extremely hard to pull off.

Conversely, assuming that certain conditions will not be present during real deployment can save development time by avoiding more costly test injection techniques. The rugged reality of customer environments can be an exacting master to people making those kinds of assumptions. And if it was hard to debug it in the lab, try and imagine debugging it in the wild.

In the end, one can try to rely on common sense and good will, but nature is a mighty adversary. Creation of software is still an engineering practice, even when seen through biological lenses.

Monday, October 30, 2006

Creationism, evolutionism and intelligent design

This article is an exploration into the software development industry practices and an attempt at understanding the dominant role played by human nature into what customers end up installing on their machines.

I may not get this right on my first write up and will skip altogether the discussions around the scientific and religious arguments over the origin of life. My intention is to jump straight into the realm of discussions around the origin of software products. Note that my eventual opinions on the software analogy have no bearing on my beliefs and opinions on the mainstream theories about the origin of life.

Creationism

In terms of software development, this form of software design is supported by a super intelligent, omniscient engineering body. The engineering body is formed by architects, designers and market managers that have an outstanding knowledge of the work flows that will assimilate the resulting product. Such capable engineering body is thus able to craft the software features to perfectly match and improve those work flows. The team can anticipate every single adverse affect of a product feature on the users' daily routine and produce a superior creation that is easy to use, free of defects and capable of solving all stated and unstated customer problems.

The perceived perfection of the resulting product does not warrant modifications or corrections to fulfill the original customer requirements. If anything, changes are only required in response to significant new requirements; requirements so significant that would possibly entail the creation of a different product.

I will not outright dismiss the possibility of software creationism; in fact I contend that engineering bodies should have more specialists in their field and not generalists trained primarily in the art of software development. Ideally, these specialists should emerge from the ranks of the user community and land a leadership role in the creation of the products for that same community. This perspective is somewhat of a counterpoint to the "other" creationist theory in that the creator of the perfect product is forged by its experiences with imperfect ones.

I am afraid that the software industry tends to be incredibly short on those kinds of engineering bodies, given the abyss-like disparity between the skill set of the users and the skill set of a software engineer. In lieu of transforming a customer into a software engineer, one could always contend that we could listen to them carefully. In some more mature fields, that is a perfectly good solution because the customers know their business work flows inside-out and simply expect the software products to automate some of those work flows. Conversely, emerging areas (Web 2.0 anyone?) are not a good breeding ground for creationist software design in the sense that customers often do not understand the effects of the technology on their business models and, in many cases, may not even have a use for the technology. It is not uncommon for customers to establish pilot projects to study the technology out of a herd-like fear of being left behind by the next big wave.

Evolutionism

In this theory, software evolves by chance and probabilities. A multitude of companies throughout the world creates initial releases out of some limited understanding of the marketplace or simply to explore new concepts and ideas. The environment, represented by the market, causes transformations on the subsequent releases and the natural selection of consumer choices ensures the survival of the fittest.

Note that the quality of being fit for the environment is as impermanent as the environment itself, which translates to a very dynamic succession of killer applications vying for their very survival. As a particular branch of the environment matures, the fittest application eventually prevails and becomes an incumbent that reaches some level of symbiose with that same environment. In more precise terms, the users get used to the solution and the solution matures to a point where new releases are virtually unnecessary.

Decades-old systems such as SABRE are examples of systems reaching those levels of symbiosis with the environment, addressing their goals in a pure and efficient manner. Their survival is assured for as long as people have to reserve flight tickets, lock horns with software problems or have to correct software bugs.

Some evidences of the software evolutionism theory are recognized in form of a myriad of startups biting the dust everyday whereas an infinitesimal number of other ventures thrive against reason. I can't really draw a parallel to acquisitions of these overachieving companies as I have yet to see evidence in nature of smaller organisms being fully incorporated into larger ones :-)

Intelligent Design

In this case, the perceived natural evolution of a software product through natural selection is explained in terms of an external intelligent being driving the evolution to fit the environment characteristics. In other words, the evolution of a software product or its prevalence on the market is not caused by chance, but by intelligent interference from the engineering body behind it.

The engineering body considers the relationship between the product and the environment at several points in time and adapts the product to the changing conditions of that environment. The omniscience of this intelligent being is represented not by a superhuman awareness of reality, but by a relentless effort to understand the customer needs. Not a single effort is made to modify or design the product until its effects are fully understood under reasonably real scenarios.

It is in that context that intelligent software design is the better theory, acknowledging the gap between the expectations from the user base (the environment) and the technical bias built into products (the creation). If simply left to chance, product evolution results in tremendous waste realized in the form of throngs of projects fallen before the chosen (acquired?) one.

This is where formal software development processes such as RUP induce positive change by structuring the way the knowledge from customer operations is collected, recorded and reflected into the final product. The tight engineering linkage from software artifacts from inception to transition ensures a more deterministic intervention in the product evolution that stands a better chance of achieving customer goals.

Tuesday, October 24, 2006

Using magnets to understand the corporate culture

Magnet Grabbing Businessman
I have always been fascinated with the how human nature mirrors several aspects of Mother Nature. A natural extension, some of those patterns also repeat themselves in organizational systems created by humans.

A good example comes from magnets and organizational culture. A magnet made from an iron core has millions of microscopic metal particles polarized in the same direction. In the absence of that cohesive field orientation across all particles, you just have a regular piece of metal.

When you apply an orthogonal magnetic field, those metal particles tend to reorient themselves along the axis of the external field. When the external field is removed, the metal particles tend to return to their original alignment, but not entirely. This effect is known as hysteresis. In simple terms, magnets have a memory.

And now to the analogy to organizational structure. In a large company, employees are the little metal particles united in a common body. With the proper external force, the particles will align in the same direction; remove that force and the metal particles tend to go back to their old orientation. Apply a strong enough field for a long enough time and these particles tend to stay permanently aligned in the same direction, thus creating a magnet.

Another interesting part of the magnetization phenomena is that the reorientation happens in individual clusters rather than in a uniform way throughout the metal body. In other words, some portions of the metal reorient themselves along the new field before others do. At some critical point, there is an avalanche effect where all particles fall in line under the influence of the external field and its neighboring particles. There is a nice animated illustration for the effect here.

Sounds familiar?

I just liked the thought and its implications to the challenge faced by executives to reorient a company in a new direction, keeping the external field in the desired direction long enough for the employees to start maintaining that direction on their own.

Can you do your part in a situation where change depends primarily on external factors? Sure, as long as you are aligning yourself with the external field, you help exert field pressure on your peers. Typically, that is the role of early adopters, evangelists and technology champions; helping the top-brass move the company towards the intended direction.

Welcome to The RTP Scrolls

"The RTP Scrolls" is a blog dedicated to my observations about corporate culture and product development. The Internet is crowded with many good resources on both topics but I always wanted to write about the influence of the first over the second.

I believe human nature plays a large role in the way corporate culture is shaped. I also believe that human nature and corporate culture tend to affect the judgment of product development teams in ways that they cannot understand, or at least acknowledge at a conscious level.

The intent of "The RTP Scrolls" is to draw analogies from other fields of the industry (automotive and construction are my favorites) with the way software is created. I will also have some fun drawing the analogies from unexpected sources, borrowing examples from Mother Nature, mathematics, physics, biology, politics and on occasion, metaphysics.

PS: RTP stands for Research Triangle Park, a public/private, planned research park, created in North Carolina in 1959 by leaders from business, academia and industry. You can read more about RTP in http://www.rtp.org.

I hope you will enjoy the content as much as I have fun writing it.

LinkedIn