Peter Drucker once commented that “trying to predict the future is like trying to drive down a country road at night with no lights while looking out the back window.” Though we agree with Drucker that forecasting is hard, managers are constantly asked to predict the future — be it to project future product sales, anticipate company profits, or plan for investment returns. Good forecasts hold the key to good plans. Simply complaining about the difficulty does not help.

Nonetheless, few forecasters receive any formal training, or even expert apprenticeship. Too many companies treat the forecasting process like a carnival game of guessing someone’s weight. And given the frequency of sandbagged (deliberately underestimated) sales forecasts and managed earnings, we even wonder how often the scale is rigged. This lack of attention to the quality of forecasting is a shame, because an effective vehicle for looking ahead can make all the difference in the success of a long-term investment or strategic decision.

Competence in forecasting does not mean being able to predict the future with certainty. It means accepting the role that uncertainty plays in the world, engaging in a continuous improvement process of building your firm’s forecasting capability, and paving the way for corporate success. A good forecast leads, through either direct recommendations or informal conversation, to robust actions — actions that will be worth taking, no matter how the realities of the future unfold. In many cases, good forecasting involves recognizing, and sometimes shouting from the rooftops about, the inherent uncertainty of the estimates, and the fact that things can go very bad very quickly. Such shouts should not invoke the paranoia of Chicken Little’s falling sky; instead, they should promote the development of contingency plans to both manage risks and rapidly take advantage of unexpected opportunities.

Fortunately, better forecasting can be accomplished almost as simply as improving Drucker’s driving challenge. Turn on the headlights, focus on the road ahead, know the limits of both the car and the driver, and, if the road is particularly challenging, get a map — or even ask others for directions. By using the language of probability, a well-designed forecast helps managers understand future uncertainty so they can make better plans that inform ongoing decision making. We will explore the many approaches that forecasters can take to make their recommendations robust, even as they embrace the uncertainty of the real world.

The Flaw of Averages

In forecasting the future, most companies focus on single-point estimates: They propose a number for the market size or the company’s unit sales in the coming year, typically based on an average of expected data. Though companies generally manage against a specific target like revenue or profit, and also share that information with outside analysts, we often forget that a point forecast is almost certainly wrong; an exact realization of a specific number is nearly impossible.

This problem is described at length by Sam Savage, an academic and consultant based at Stanford University, in The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty (Wiley, 2009). He notes how focusing on an average without understanding the impact of the range can lead to flawed estimates. Better decisions result from taking the time to anticipate the likelihood of overshooting or undershooting the point, and then considering what to do today, given the range of possibilities in the future.

Savage highlights the simple example of a manager estimating the demand for 100,000 units of a product — based on a range of possible market conditions — and then extrapolating that average to produce a profit estimate. But the plausible demand could be as much as 50 percent above or below the average, with potentially dangerous consequences. If demand runs 50 percent above the average, the plant will miss some sales because it will be unable to increase capacity that much in the time period. Conversely, if demand runs 50 percent below the forecast average demand, the profit per unit will be dramatically lower, since the plant has to spread its fixed cost over fewer units. As a result, the profits at an average demand level will be much different from an average of the profits across the range of possibilities. Rather than a simple average, a better forecast would present a wide range of scenarios coupled with a set of potential actions to influence the demand and profitability. Such a forecast would encourage management to heed early signals of consumer interest to accelerate marketing and/or cut fixed costs if sales fall short, or to ramp up production quickly if sales appear to be at the high end of the forecast.

Reflecting risk in forecasts is a simple concept and one that may seem easy to put into practice, but managers commonly ignore the uncertainties and simply collapse their forecasts into averages instead. We often see this in predictions of project completion timelines. Consider a project with 10 parallel tasks. Each task should take between three and nine months, with an average completion time of six months for all of them. If the 10 tasks are independent and the durations are distributed according to a triangular distribution, chances are less than one in 1,000 that the project will be completed in six months, and the duration will be close to eight months. But using the six-month figure instead offers an almost irresistible temptation; after all, that’s the average input.

Despite the potential that point estimates carry for misleading decision makers, many firms default to them in forecasts. For example, Airbus and Boeing present passenger traffic and freight traffic annual growth rates over a 20-year horizon as point estimates in their respective biannual “Global Market Forecast” and “Current Market Outlook” reports. Although a close reading of the reports suggests that the forecasters considered ranges when generating the forecasts — and even conducted sensitivity analyses to understand the implications of different assumptions — such scenarios are not reported. A forecast showing the range and not just the average would be more valuable in making plans, and would help the industry avoid overconfidence.

In short, forecasting should not be treated as a game of chance, in which we win by getting closest to the eventual outcome. Occasionally being “right” with a particular prediction creates no real benefit and can in fact lead to a false sense of security. No one can produce correct point forecasts time and time again. Instead, it’s better to use the range of possible outcomes as a learning tool: a way to explore scenarios and to prepare for an inherently uncertain future.

Drivers of Uncertainty

The most useful forecasts do not merely document the range of uncertainties; they explain why the future may turn in different directions. They do this by “decomposing” the future into its component parts — the driving forces that determine the behavior of the system. Just asking “Why might this happen?” and “What would happen as a result?” helps to uncover possible outcomes that were previously unknown. Recasting the driving forces as metrics, in turn, leads to better forecasts.

For example, the general business cycle is a driving force that determines much of the demand in the appliance industry. Key economic metrics, such as housing starts, affect the sales of new units, but a consumer’s decision to replace or repair a broken dishwasher also depends on other factors related to the business cycle, such as levels of unemployment and consumer confidence. With metrics estimating these factors in hand, companies in that industry — including the Whirlpool Corporation in the U.S. and its leading European competitor, AB Electrolux — use sophisticated macroeconomic models to predict overall industry sales and, ultimately, their share of the sales.

Here, too, the effective use of metrics requires an embrace of uncertainty. Simply focusing on the output of the model (the projected sales figures) rather than the input (such as unemployment and consumer confidence) can actually do more harm than good. Whirlpool’s planners use their industry forecast models to focus executive attention, not replace it. The planners present the model for the upcoming year or quarter, describing the logic that has led them to choose these particular levels of demand and the reason the outcomes are meaningful. Executives can set plans that disagree with the forecasters’ predictions, but everyone has to agree on which input variables reflect an overly optimistic or pessimistic future. Even more important, managers can begin influencing some of the driving forces: For example, they can work with retail partners to encourage remodeling-driven demand to offset a drop in housing starts.

Black Boxes and Intuition

As the Whirlpool example demonstrates, mathematical models can help focus discussions and serve as a foundation for effective decision making. Thanks to the increasing power of personal computers and the Internet, we have a host of advanced mathematical tools and readily available data at our disposal for developing sophisticated models.

Unfortunately, such models can quickly prove to be a “black box,” whose core relationships and key assumptions cannot be understood by even a sophisticated user. Black-box models obfuscate the underlying drivers and accordingly can lead to poor decision making. Without a clear understanding of the drivers of the model, executives will not be attuned to the changes in the environment that influence the actual results. Executives who blindly trust a black-box model rather than looking for leading indicators inevitably find themselves captive to the “too little, too late” syndrome.

A lack of understanding of the black boxes tempts many managers to dismiss the planners’ models and simply “go with the gut” in predicting possible challenges and opportunities. But that approach poses equally daunting problems. Back in the early 1970s, Nobel laureate Daniel Kahneman and his longtime collaborator Amos Tversky began a research stream employing cognitive psychology techniques to examine individual decision making under uncertainty. Their work helped popularize the field of behavioral economics and finance. (See “Daniel Kahneman: The Thought Leader Interview,” by Michael Schrage, s+b, Winter 2003.) Work in this field has demonstrated that real-life decision makers don’t behave like the purely rational person assumed in classic decision theory and in most mathematical models.

As illustrated by a variety of optical illusions, our brains seek out patterns. The ability to fill in the blanks in an obscured scene helped early man see predators and game in the savannas and forests. Though critical in evolutionary survival, this skill can also lead us to see patterns where they do not exist. For example, when asked to create a random sequence of heads and tails as if they were flipping a fair coin 100 times, students inevitably produce a pattern that is easily discernible. The counterintuitive reality is that a random sequence of 100 coin flips has a 97 percent chance of including one or more runs of at least five heads or five tails in a row. Virtually no one assumes that will happen in an invented “random” sequence. (Any gambler’s perceived “lucky streak” offers a similar example of the typical human being’s pattern-making compulsion.)

Our tendency to see patterns even in random data contributes to a key problem in forecasting: overconfidence. Intuition leads people to consistently put too much confidence in their ability to predict the future. As professors, we demonstrate this bias for our MBA students with another simple class exercise. We challenge the students to predict, with a 90 percent confidence level, a range of values for a set of key indicators such as the S&P 500, the box office revenues for a new movie, or the local temperature on a certain day. If the exercise is done correctly, only one out of 10 outcomes will fall outside the predicted range. Inevitably, however, the forecasts fail to capture the actual outcome much more frequently than most of the students expect. Fortunately, the bias toward overconfidence diminishes over time as students learn to control their self-assurance.

History Matters

Although Peter Drucker fretted about looking out the rear window of the car, in reality too many forecasters fail to examine history adequately. Consider the subprime mortgage crisis. In 1998, AIG began selling credit default swaps to insure counterparties against the risk of losing principal and interest on residential mortgage-backed securities. AIG’s customers eventually included some of the largest banking institutions in the world, such as Goldman Sachs, Société Générale, and Deutsche Bank.

At the end of the fourth quarter of 1998, the delinquency rate for U.S. subprime adjustable-rate mortgages stood at just over 13 percent. By the end of the fourth quarter of 2008, this rate had almost doubled, to an astonishing 24 percent. This in turn led to the US$180 billion bailout of AIG. Although a 24 percent default rate seemed unprecedented to most bankers, a look back beyond their own lifetimes would have indicated the possibility. In 1934, at the height of the Great Depression, approximately 50 percent of all urban house mortgages were in default.

That is why looking back at past forecasts and their realizations can prove so valuable; it can help prevent overconfidence and suggest places where unexpected factors may emerge. Recently, researchers Victor Jose, Bob Nau, and Bob Winkler at Duke University proposed new rules to score and reward good forecasts. An effective “scoring rule” provides incentives to discourage the forecaster from sandbagging, a proverbial problem in corporate life. For example, Gap Inc. measures the performance of store managers on the difference between actual sales and forecast sales, as well as on overall sales. By assessing forecasting accuracy, the rules penalize sales above the forecast number as well as sales shortfalls. Unfortunately, Gap is an exception. To date, few firms have picked up on the research into incentive mechanisms and scoring rules to improve forecasts, despite the proven success in fields such as meteorology.

It may seem like an obvious thing to do, but most companies do not revisit their forecasts and track the actual results. A recent survey by decision analysis consultant Douglas Hubbard found that only one out of 35 companies with experienced modelers had ever attempted to check actual outcomes against original forecasts — and that company could not present any evidence to back up the claim. Airbus and Boeing spend resources in generating their “Global Market Forecast” and “Current Market Outlook” reports, but they do not report on the accuracy of their previous forecasts. On the other hand, Eli Lilly has developed a systematic process of tracking every drug forecast to understand its predictive accuracy.

Wisdom of Crowds

Increasingly, conventional wisdom also challenges the logic of expert forecasters even if they have been trained to rein in their overconfidence through continuous feedback of actual results. Journalist James Surowiecki presented the case in his bestseller, The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations (Doubleday, 2004). Furthermore, research into forecasting in a wide range of fields by Wharton professor J. Scott Armstrong showed no important advantage for expertise. In fact, research by James Shanteau, distinguished professor of psychology at Kansas State University, has shown that expert judgments often demonstrate logically inconsistent results. For example, medical pathologists presented with the same evidence twice would reach a different conclusion 50 percent of the time.

The old game of estimating the number of jelly beans in a jar illustrates the innate wisdom of the crowd. In a class of 50 to 60 students, the average of the individual guesses will typically be better than all but one or two of the individual guesses. Of course, that result raises the question of why you shouldn’t use the best single guesser as your expert forecaster. The problem is that we have no good way to identify that person in advance — and worse yet, that “expert” may not be the best individual for the next jar because the first result likely reflected a bit of random luck and not a truly superior methodology.

For this reason, teams of forecasters often generate better results (and decisions) than individuals, but the teams need to include a sufficient degree of diversity of information and perspectives. A naive forecaster often frames the question a different way and thinks more deeply about the fundamental driver of the forecast than an expert who has developed an intuitive, but often overconfident, sense of what the future holds.

Group dynamics can produce a different sort of challenge in bringing together a team; people vary in their styles and assertiveness. The most vocal or most senior person — rather than the person with the keenest sense of possibilities — might dominate the discussion and overly influence the consensus. This has been the case in a host of classroom simulations based on wildfires, plane crashes, and boat wrecks. They all place teams into a simulated high-pressure situation where collective insight should help. Typically, a dominant personality steps forth and drives the process toward his or her predetermined view, making little or no use of the wisdom of the crowd. In The Drunkard’s Walk: How Randomness Rules Our Lives (Pantheon, 2009), physicist and writer Leonard Mlodinow describes a number of research studies that show how most people put too much confidence in the most senior or highest-paid person. Does that sound like your executive team?

Culture and Capability

To become proficient at forecasting, a company must develop capabilities for both achieving insight and converting that insight into effective decision making. The firm need not seek out the star forecaster, but instead should invest in cultivating an open atmosphere of dialogue about uncertainty and scrutiny — one that brings to the fore a more complete picture of the expert knowledge that already resides in many of its existing employees.

The resulting culture will be one in which managers recognize and deal with uncertainty more easily; they won’t feel they have to resort to the extreme of either throwing up their hands in despair or pretending that they have all the answers.

In the end, overcoming the problems and traps in forecasting probably requires the use of all of these approaches together, within a supportive culture. An example of how difficult this is can be found in the U.S. National Aeronautics and Space Administration (NASA), which probably contains as analytically rigorous a set of people as can be found in a single organization.

The disintegration of space shuttle Columbia in 2003 on reentry during its 28th mission demonstrates how culture can overrule capability. After problems during the shuttle’s launch, NASA engineers developed extensive models for a wide range of scenarios, including the possibility that foam pieces had struck the wing, the event ultimately deemed responsible for the accident. But rather than focus on contingency plans for dealing with the known issue but unknown impact, NASA officials placed too much faith in their mathematical models, which suggested that the wing had not sustained a dangerous degree of damage. The results were catastrophic.

Less than a month after the Columbia disaster, this pervasive cultural problem at NASA was described in an article in the New York Times that quoted Carnegie Mellon University professor Paul Fischbeck. (Fischbeck, an expert on decision making and public policy, had also been the coauthor of a 1990 NASA study on the 1986 Challenger explosion caused by an O-ring failure at cold temperatures.) “They had a model that predicted how much damage would be done,” he said, “but they discounted it, so they didn’t look beyond it. They didn’t seriously consider any of the outcomes beyond minor tile damage.” In other words, even NASA’s brilliant rocket scientists couldn’t outsmart their own inherent biases. They needed processes and practices to force them to do so.

And so, probably, does your company. Too many managers dismiss the inherent uncertainty in the world and therefore fail to consider improbable outcomes or invest sufficient effort in contingency plans. The world is full of unknowns, even rare and difficult-to-predict “black swan” events, to use the term coined by trader, professor, and best-selling writer Nassim Nicholas Taleb. Overreliant on either their intuition or their mathematical models, companies can become complacent about the future.

Consider, for example, the 2002 dock strike on the West Coast of the U.S., which disrupted normal shipping in ports from San Diego to the border with Canada for a couple of weeks. A survey conducted by the Institute for Supply Management shortly afterward found that 41 percent of the respondents had experienced supply chain problems because of the strike — but only 25 percent were developing contingency plans to deal with future dock strikes.

We can train our intuition to offer a better guide in decision making. To do so, we must be aware of our biases and remember that all models start with assumptions. Engaging a diverse set of parties, including relatively naive ones, forces us to articulate and challenge those assumptions by seeking empirical data. No model is objective, reflecting some universal truth. Instead, business models represent highly subjective views of an uncertain world. Rather than seeking the ultimate model or expert, managers should adopt the axiom cited by General Dwight D. Eisenhower regarding the successful but highly uncertain D-day invasion in World War II. He asserted that “plans are nothing; planning is everything.” A good forecast informs decisions today, but equally important, forces us to consider and plan for other possibilities.

I really like this notion. It fits nicely with the tenets of collective intelligence and social business intelligence approaches to decision making. In order to be able to adapt, a business has to be able to have an accurate and up-to-date read on what’s really going on, which is exactly the insight such applications deliver.

Thing is, forecasting is still vital. General Patton‘s famous quote comes to mind: “A good plan, violently executed now is better than a perfect plan next week.” So you start executing to a plan. At the same time, you enable everyone involved to continuously share information about how things are tracking. So long as the collective wisdom of the people in the trenches lines up with the master plan, life’s (probably) good. But when the two start to diverge, it may be time to adapt.

The current leader in the market is Samuel Driessen, an Information Architect at Océ, a 22,000-employee provider of digital document management technology and services. In recognition of his performance, he has been awarded an all-access pass to the upcoming E2.0 Conference in Boston. Many thanks to Steve Wylie of TechWeb for sponsoring this prize.

The actual results for the forecasts in the Adoption Index Prediction Market won’t be known until early June. So what does it mean that Samuel is currently in first place? In essence, he’s a trend spotter. If you want to know what people think will happen with E2.0, talk to Samuel. In June, we’ll find out if he’s equally as good at calling actual outcomes.

What does Samuel think the future holds? “To me, I’m not as focused on the Web 2.0 applications as I am on the underlying concepts. These concepts are here to stay. They will fundamentally change the way we interact in this world, how businesses will run internally, and how they’ll interface with customers. Openness, transparency, and network effects are here to stay.”

Samuel is dedicated to instilling these concepts at Océ. He’s proud of the value he’s created for Océ by implementing E2.0 tools such as wikis, blogging, microblogging, and social bookmarking. “It’s really interesting to see how people are finding and helping each other as a result of these technologies.”

We particularly like Samuel’s comparison of structured information processes (e.g., ERP systems) and unstructured information processes (e.g., microblogging). Samuel thinks prediction markets have the potential to bridge the gap between these two types of processes. The “Adoption Index market gave me a great chance to see what the power of a prediction market is.”

It will be a couple more months before we know whether Samuel will keep his first place position. In the meantime, we appreciated the opportunity to hear from such a thoughtful member of the E2.0 community.

Some implementations of prediction markets compute probabilities of outcomes. To use Dr. Kambil’s example, one could ask, “Will the DJIA end above 10,000 before the end of the year?” Market participants who are bullish about the Dow buy shares. Those who are bearish sell. As trades come in, the market mechanism adjusts the probability of the event actually taking place.

While this is pretty interesting, getting to a collective forecast that gives you a probability distribution of possible outcomes, rather than just a point forecast, of the DJIA itself is awkward. To do it, you’d have to launch different markets around buckets of outcomes — as in, DJIA in 8,000-9,000, DJIA in 9,000-10,000, DJIA in 10,000-11,000, and so on — and then reason about the probabilities of each.

Crowdcast builds probability distributions automatically. This has two important implications. First, it enables a simple user interface and finer grained expression of beliefs. Rather than asking people to choose predefined buckets, they can select a precise range, as wide or narrow as they wish. And second, it supports some great applications for business. For instance, you can get alerts when the likelihood of hitting your target ship date falls below 50%.

Dr. Kambil also discusses the information dissemination “feature” of prediction markets. Prediction markets, like the public stock market for instance, not only collect information, but they also distribute it. While this is great for some applications — surely you’d want to know the price (and other indicators) of Apple stock before you bought some — it’s a problem for others.

In the enterprise, it is often the case that the more valuable and important a metric, the more secret it is. Revenues or earnings per share are but two examples. We’ve struggled with this reality for some time. Initially, our take was that we should simply concentrate our efforts on problems characterized by “public” metrics. Then we implemented access control lists, which enabled our customers to publish questions about sensitive metrics to a subset of the participants.

While this worked for some applications, the overall approach just didn’t sit well. There is knowledge in the enterprise about revenues and plenty of other sensitive metrics. And we’re all about harnessing knowledge. Won’t let the cat out of the bag yet, but we’ve cracked it! Details very soon.

This feature was unveiled last week during the keynote at SAP’s GRC 2010 conference in Orlando. The keynote was presented by Narina Sippy, SVP and GM of SAP’s Governance, Risk, and Compliance solutions. The theme of her address was arming yourself for risk in your business. She told a great cautionary tale about Toyota’s recent debacle, interviewed John Hummer of Hummer Winblad, and then introduced Ranga Bodla and our own Mat Fogarty to demo the products. They presented a compelling story about a pharma company managing its risks with GRC, while keeping a finger on the pulse of the organization with Crowdcast. The presentation couldn’t have gone over better.

We took advantage of our bully pulpit at the conference to announce Crowdcast’s formal relationship with SAP. Indeed, this development is hugely validating for our space in general and for Crowdcast in particular. I believe that our charter of supporting decision makers is a big part of the story.

In response to these ideas, Enterprise 2.0 pioneer and MIT Principal Research Analyst, Andrew McAfee, took to his blog to highlight some parts that really stuck with him. Andy’s synthesized version of the article really spoke to us and the ideas that we find valuable at Crowdcast. Andy highlighted a couple of values that we think are essential to smart business: breaking down hierarchies to unclog communication and fostering collaboration through peer effects, which allow people to get recognized by their peers for what they do rather than by their organizational rank.

Companies are beginning to understand the importance of communication within the ranks of their organization, not only to improve workflow, but also to improve access to employee intelligence. From there, managers can start to really monitor the pulse of their company. Ultimately, harvesting wisdom and gleaning well rounded insight is a competitive advantage — a very timely and relevant conversation to have as companies look ahead into 2010.