Crowdcast and the Efficient Forecasting Frontier

Crowdcast is all about bringing the power of collective intelligence to bear on real business metrics.  Most of the time we’re practitioners, working with clients to apply our technology and knowledge to their challenges in project risk management and sales metrics.  Every once in a while, though, we get a great opportunity to step back and challenge the state of the art.  Our current project, joint with an all-star cast of academics and funded by IARPA is such an opportunity. 

The US Intelligence Agencies have thousands of highly-trained analysts with access to a vast store of privileged information.  However, they have great difficulty efficiently synthesizing and surfacing this info actionable information.  Often the “a broken watch is right twice a day” phenomenon encourages analysts to stick with the status quo rather than flag new conditions.  These kinds of forecasting behaviors have real costs in dollars and in lives.

Despite its importance in modern life, forecasting remains (ironically) unpredictable. Who is a good forecaster?  How do you make people better forecasters?  Are there processes or technologies that can improve the ability of governments, companies, and other institutions to perceive and act on trends and threats?  Nobody really knows.  The goal of the Good Judgment Project is to answer these questions.

We will systematically compare the effectiveness of different training methods and forecasting tools in accurately forecasting future events.  We also will investigate how different combinations of training and forecasting work together.  Finally, we will explore how to more effectively communicate forecasts in ways that avoid overwhelming audiences with technical detail or oversimplifying difficult decisions.

Over the course of each year, forecasters will have an opportunity to respond to 100 questions, each requiring a separate prediction, such as “How many countries in the Euro zone will default on bonds in 2011?” or “Will Southern Sudan become an independent country in 2011?”  As the project evolves, we’ll iterate the mechanisms and the training, arriving at an idealized and verified system.

The Good Judgment research team is based in the University of Pennsylvania and the University of California Berkeley. The project is led by psychologists Philip Tetlock, author of the award-winning Expert Political Judgment, Barbara Mellers, an expert on judgment and decision-making, and Don Moore, an expert on overconfidence. Other team members are experts in psychology, economics, statistics, interface design, futures, and computer science.

If you’re interested in participating, we’d love to have you.  Head on over to our registration site to sign up.  We’re looking forward to sharing the results with you, and leveraging the valuable insights of The Good Judgment Project with our clients.