Drinking the Kool-Aid

We completely redesigned our knowledge aggregation mechanism — or “new mech” as it has endearingly became known here at Crowdcast — several months back.  The new mech is interesting for a number of reasons.  Here are two: it’s a novel and robust way to collect information (we’ve already filed two patents on it), and integrating it into theCrowdcast platform benefited a lot from managing the project using Crowdcast.  We will describe the new mech and our experience with using Crowdcast for managing complex projects in this and the next couple of posts.

The new mech was designed by our very own Leslie Fine and two UC Berkeley PhDs and implemented using MATLAB.  When it came time to hook it into the Crowdcast platform, we decided to use the MATLAB runtime rather than port the algorithms over to Ruby.  Among other benefits, this path would make incorporating mech changes much more efficient.  On the flip side, project logistics became quite hairy.  In addition to our dev guys here in San Francisco, we’d need to cozy up to the folks at MathWorks (authors of MATLAB) in Massachusetts and get much more tightly coupled to our math guys in Berkeley and a telecommuting systems integration consultant.

We finally got it working, but only after a painful false start.  Initially we managed the project the way we’d done it at other companies — Gantt charts and meetings.  That failed miserably.  Milestones slipped left and right.  People were stressed and unhappy.  Then we started to manage the project using Crowdcast.  The change to overall performance was astounding.  Will have more to say about this later.  First, here’s where Crowdcast came in.

Rev two of the project entailed creating a series of crowdcasts around our milestones.  Crowdcasts are accurate, unbiased predictions that incorporate insights of a diverse group of people.  Crowdcasts aggregate two components: “bets” on the outcome and a comments that explain participants’ rationales.

One of the crowdcasts was around the beta release date of the new mech.  Huned created the crowdcast and invited everyone involved to participate.  He specified a target date, which is not made available to participants, so that he could monitor performance over time.  In particular, the system would alert him if thecrowdcast moved too far outside of his target.

About a month back when the project was in full swing I had a chat with our integration consultant.  He’d just started load testing the system, but it kept falling over due to bugs in the MATLAB runtime (in fairness to MathWorks, we were pushing the envelope on the capabilities of some of MATLAB’s features).  I believed that I had rare and relevant information to share — the beta release was at risk because of MATLAB bugs — so I rushed to my desk to place a bet.

At the time, the crowd — all participants who’d shared their insights on this crowdcast before me — believed that beta would happen on January 15 +/- 7 days.  Given my information, I thought it’d be at least a week later, so I selected a range of January 20-31 and bet $5000 of my virtual money.  I also left a comment explaining what my point of view.  Relative to crowd beliefs, I had about a 25% chance of being correct, so the system offered me a payoff of about $19,200.  If someone were to place the same bet just after me, the odds would now be higher, and so the payoff offered lower.  Neat thing about this is that the system rewards those who not only have accurate information, but are early to reveal it.

Behind the scenes, Crowdcast infers my perception of the probabilities of various outcomes by looking at my bet range, wager amount, and total portfolio value. It then updates the crowd curve using this information. All this heavy lifting enables us to do some really amazing things. For instance, we can “slice and dice” crowdcasts, to analyze sentiment by department or region. In the next post, I’ll talk about how these capabilities helped us identify potential problems and course-correct to navigate around them.