Prediction markets have promised traders more accurate, real-time forecasts on crucial economic data releases.
On one of the most closely watched and economically consequential US economic events – the monthly jobs numbers – they haven’t yet lived up to the hype.
Last month, when the Labor Department reported that the economy added 178,000 jobs, the final aggregate estimate from bettors on Kalshi Inc. was off by over 90,000 jobs, after close to $1 million in wagers were placed.
That was slightly closer than the median forecast of 79 economists polled by Bloomberg. But over the last three years of monthly estimates, both Kalshi and the experts have generally missed the actual number by a similar, and relatively significant amount, according to a Bloomberg analysis of the data.
Professional forecasters did have a slight edge, but not at a level that is statistically significant. Across all 33 months, both economists and Kalshi missed the actual number by over 60,000 jobs, on average.
The jobs forecast has been among the most difficult economic data points for experts to predict, providing a particularly important testing ground with ramifications for monetary policy and the direction of the broader economy. The Labor Department acknowledges that the numbers it releases are preliminary, with a wide band for error, and are ultimately revised more than three times.
Some economists who track the data say that the results so far, and the reputation of prediction markets as a new form of online gambling for unsophisticated amateurs, have not convinced them to pay all that much attention to the odds.
“Kalshi’s just making money on a novelty bet, like The Price is Right,” said Brett Ryan, a senior US economist at Deutsche Bank, who is among the 10 most accurate payrolls forecasters over the past two years in Bloomberg’s panel.
The neck-and-neck race suggests to some skeptics that many prediction market traders are merely copying what they see from the economists, rather than offering any new information. Going into this week’s report, Kalshi’s estimate on Wednesday of 71,000 jobs is just 6,000 higher than what economists are expecting.
Jack Such, a spokesperson for Kalshi, said that the company’s traders are offering a crowdsourced wisdom that builds on the aggregate expertise of the economists.
“The fact that the eventual economists’ forecasts and Kalshi data are similar does not imply Kalshi is an inferior data source, it just means the existing forecasting mechanisms are much stronger compared to other subjects on Kalshi,” Such said.
The dream for prediction markets is that they incentivize traders to reveal insights from ordinary people who are spread across the country and may have access to evidence that the economists miss, like local signs of job losses or gains.
Kalshi’s chief executive officer, Tarek Mansour, has said his company harnesses the wisdom of the crowd to replace “debate and subjectivity with markets and accuracy.”
Kalshi’s odds have been materially different than the economists in several months, suggesting that the economists and the crowds have, at least sometimes, come to their conclusions independently, rather than just following each other.
Some top economists say they are watching the results in case the signal becomes more valuable.
“If we see that it gets it more right than wrong, I would be open to it” said Oscar Muñoz, the chief US macro strategist at TD Securities and another top-10 forecaster. “Maybe in the future we’ll use them, but not now.”
Ryan, at Deutsche Bank, said that in addition to seeming like something of a stab in the dark, the Kalshi forecasts miss the texture and detailed data below the headline number, which offer the most important information on the state of the economy.
“I’m actually not trying to forecast non-farm payrolls,” he said. “What I’m trying to focus on is, what does the entire report tell us about the labor market?”
Kalshi traders express their views by placing yes or no bets on a variety of contracts tied to whether the final number will be higher or lower than a specific number. On Wednesday, for example, a trader could pay 71 cents for a contract that would pay out $1 if the main payroll number comes in over 30,000, among several other similar contracts. Kalshi then creates a topline forecast from a blended result of all the bets.
Kalshi’s main rival, Polymarket, also lets customers trade on the jobs numbers. But it does not publish aggregate odds like Kalshi that make it possible to compare with economist forecasts.
A number of academic studies have offered support for the believers in the wisdom of the crowds. One recent study, conducted by three economists, found that Kalshi’s odds have been “roughly consistent” with professionals in predicting US interest rate decisions, inflation and unemployment. On one particular data set, headline CPI, it offered a statistically significant improvement.
The authors of that paper said that prediction markets are particularly valuable because they offer a continuous, real-time version of how the odds are moving, unlike economists, who only update their forecasts periodically. On the nonfarms payroll data, for instance, many economists’ final projections are submitted days in advance, while Kalshi’s odds are constantly updated as bets come in until the last minute.
To the degree that prediction markets have been accurate, there have been growing concerns that it might be because people with insider information are moving the markets. A US soldier was arrested last month after allegedly placing early bets on the ouster of Venezuelan dictator Nicolás Maduro right before taking part in the military mission to capture him.
Bets on geopolitical events and macroeconomic forecasts have been a relatively small part of Kalshi’s overall business. Sports betting has been the single largest category on the exchange, driving overall trading volume to $14.8 billion last month, more than five times what it was last September, according to user-compiled data on Dune Analytics.
The jobs report has generated much lighter trading, and it has not been growing — with an average of about $435,000 in monthly trading over the last three years. That is less than some basketball games see in a single quarter.
A recent academic paper found that only a small number of traders — as few as 3% — are responsible for whatever accuracy prediction markets have achieved, while many traders are speculating around the edges, and losing money.
The bets on the payroll data show that traders are making a much wider array of guesses than the economists, who tend to cluster together more closely. Stephanie Roth, one of the top ranked forecasters, and the chief economist at Wolfe Research, said that she and her peers have developed similar models and keep an eye on what each other are thinking.
“It’s like this little bubble where it does seem like people tend to gravitate towards similar numbers, and a lot of times because the models are similar,” Roth said.
One of the selling points of prediction markets is that they bring in people who are outside the expert bubbles. But Roth said that she has yet to see anything valuable coming out of that process.
“A lot of times people will ping me, with ‘What’s your forecast? Here’s mine. Here’s my inputs,’ and I’ve never heard anybody say ‘One of my inputs is a probability market.’”
