Category: forecasting

Under the Hood of Buying & Selling Predictions

How do those futures markets like Betfair and Intrade work?

The managers of a prediction market decide upon a finite number of prediction contracts. A contract is essentially a description of some hypothetical future event. For example, a contract might be “Fadebook trades at least $50 per share in 2012.” Another important aspect of the contract specification is the anticipation and preemptive resolution of ambiguity. If Fadebook does a 2:1 split, does the event become “…at least $25 per share?” What if the asking price for Fadebook shares reaches $50, but the bid price does not? Contracts also specify an expiration date, a time by which the event must occur. In the case of the Fadebook contract, the obvious expiration date would be January 1st of 2013. For other contracts, the expiration will be arbitrary to allow for a final decision on the event’s occurrence or nonoccurrence.

Virtual Currency
A prediction is a single user’s opinion on the likelihood of the contract’s event occurring. The prediction market encourages users to form opinions about contract events, and encourages users to wager virtual currency on a contract. Users purchase virtual currency (henceforth “credits”) with real money, or perhaps are granted an amount of virtual currency in a freemium offering.

Contract Size
Each contract also specifies a size, which is both the minimum number of credits a user can wager on a contract, and the marginal increase in the size of a wager. If the prediction market choose a size of “5 credits” for the Fadebook contract, then an individual user can wager only 5, 10, 15… credits on the contract. Contract sizes are necessary in order better match both sides of the wager. Somewhat confusingly, experienced traders refer to each increment of the contract size as “a contract.” If I wager 15 credits on the Fadebook contract with a size of 5 credits, then I am said to be wagering “3 contracts.”

Direction
When making a wager on a contract, the user specifies the number of credits she will risk, in contract size units, and the direction of the wager. If a user believes the event is certain to occur and the user is eventually proven correct, she will profit from buying a contract whenever its likelihood is less than 100%. If a user believes an event is certain not to occur and turns out to be right, the user will profit from selling the contract when its likelihood is greater than 0%. The buyer of a contract is said to be long and the seller of a contract we call short.

Payoff
When the contract’s event occurs, the long side earns the contract size in credits for each contract, less the likelihood level where she initially make the wager. So if Fadebook’s stock hits $50 in October of 2012 while I am long 3 contracts bought at 25% likelihood, then the prediction market would immediately close the contract and credit my account with 5 credits (size), times 3 contracts, less 25% of the total, or 11.25 credits. The opposite also occurs for the short side. In this case, the 3 contracts the short side sold are closed out worthless at 0%, and the short’s account is reduced by 5 credits size, times 3 contracts, times 25% of the total. Again this is 11.25 credits but deducted from the short side of the wager.

If the contract expires as the year 2012 comes to a close, and I shorted 2 contracts at 60% back in February of 2012, then I will earn a profit of 5 credits size, times 2 contracts, times 60% or 6 credits. This is exactly what the long side of those 2 contracts would lose. The long side may be 1 user, or 2 different users each long 1 contract.

By definition, the long and short sides of a contract will always balance. A user is not able to be long a contract unless another user is short. This would be similar to a real futures contract traded in Chicago or New York, but where the long side is committed to “buying” the event for 100% if it occurs.

Orders
The current likelihood (henceforth “price”) of a contract is determined by looking at the contract’s order book on the prediction market. Order books are how buyers and sellers of contracts are matched, and indicate a prediction contract’s current price or likelihood. An order book is an ordered list of buying and selling prices. If there is no market in a contract and a user wishes to make a wager, her estimated likelihood becomes the best buying or selling price for the contract. From now on we drop the price or likelihood’s percent sign for brevity.

Say the Fadebook contract has just been listed and publicized on the prediction market. If I believe the event is very likely to occur, then I might offer 99 to anyone willing to sell at 99. If I am correct, then I will eventually earn 1 credits per contract for my trouble. I will have paid 99 for something that will earns me 100. Though perhaps I want to leave myself more room to profit, and instead offer 25 to anyone willing to sell at 25. This would mean a profit of 75 per contract when it expires. The prices at which every user is willing to buy forms one side of the order book, the buying prices or bids.

A similar process happens on the selling side. I want to be short a contract when I do not believe the event will occur. So I would sell to anyone for more than 0 likelihood or price. If I end up correct in my prediction and the event does not occur when I went short at 90 likelihood, then I earn 90% of the contract size for each contract. The collected prices at which all users are willing to sell forms the other side of the order book, the selling prices or asks.

Market Orders
When a wager occurs at a price matching one of the buy or sell orders currently available in the market, the order is instantly made complete by matching a long and a short side of a wager. If I want to buy the Fadebook contract at 30 or 35 and there is already a user with a selling price or ask of 32, my wager will be immediately matched. In this sense, my order never actually appears in the prediction market order book for the contract, since the wagers are instantly matched.

Often a user will want to buy or sell at whatever likelihood or price the market is currently offering. In this market order, the user takes whatever happens to be the best price available at the time. Market orders are risky because the user may not know the exact price at which she commits to the wager. Market orders are also said to leduce liquidity in the market, and may be considered less healthy for the prediction market than regular limit orders.

Trading Out Early
The fundamental advantage to prediction markets over traditional oddsmaking is the option for a user to exit out of a profitable or loser wager early, before the contract event’s expiration. A user who is currently long or short a contract is free to trade the position to another party at any point, even if the user has only held the contract for a few minutes. This is advantageous for a user who wants to cut their losses early on a losing contract, or wants to take their profit early on a contract that becomes profitable.

If I went long 5 of the Fadebook contracts at 35 a few months ago, but that contract now has an order book centered around 50 bid/ask, then I can sell the 5 contracts for a 15 profit per contract right now. I do not need to hold my Fadebook contracts until 2013. This is not going short the contract, but selling to a willing buyer in order to net out my position with a profit.

Liquidity & Accurate Predictions
The more users are actively participating in wagering on a contract, the more accurate the collective estimate of likelihood. The most recent trade price or likelihood of a very busy and popular contract is an excellent estimate of the real likelihood of the contract event occurrence. If a single user believes a contract has a 35% likelihood, but a thousand other users are trading that contract around 75% likelihood, chances are the first user is wrong!

Prizes & Incentives
Prediction markets are more powerful when users are incentivized with real compensation. Therefore even if the prediction market’s virtual currency simplifies the regulatory aspects of the project, credits need to be closely tied to real money or prizes, so users assume actual risk when making wagers. Also any prizes need to incentivize users to make careful wagers and not necessarily “swing for the fences” on each wager. In other words, each bit of virtual currency must contribute to winning prizes.

Users who risk nothing of value when making wagers will not turn out to form a particularly accurate prediction market.

Case-Shiller April Forecasts

Another finger in the air, in the beginning of the month lull.

My forecasts for the March, 2011 Case-Shiller index levels were quite rushed. They were released quickly so I could publicly compare the forecasts with the CFE futures contracts about to expire. However, since the statistical models use active market data, there is no mathematical reason to wait on our forecasts until the end of the month. The April, 2011 index levels will be released on June 28th, but here are my forecasts given what the real estate markets were doing a few months ago:

City Confidence Forecast Predicted HPI
Minneapolis, MN +1 -10.52% 94.46
Phoenix, AZ +1 -2.85% 97.42
Las Vegas, NV +3 -1.56% 95.67
Atlanta, GA +2 -1.45% 96.93
Boston, MA 0 -1.32% 145.42
Los Angeles, CA -2 -1.22% 165.73
Seattle, WA +3 -0.46% 132.35
New York, NY -1 -0.21% 163.15
San Francisco, CA -3 -0.20% 129.56
Chicago, IL +2 -0.06% 110.50
San Diego, CA -3 +0.18% 154.16
Detroit, MI 0 +0.41% 67.34
Charlotte, NC 0 +0.50% 107.50
Miami, FL 0 +1.01% 138.66
Dallas, TX +1 +1.62% 114.72
Cleveland, OH +1 +2.12% 98.85
Denver, CO 0 +2.27% 123.29
Tampa, FL +1 +2.28% 129.98
Portland, OR +1 +4.71% 138.92
(The confidence score ranges from negative three for our weakest signals, up to positive three for strength. Unfortunately I am still sorting out a bug in our Washington, DC model.)

Housing Finger in the Air

The March, 2011 Case-Shiller numbers will be released this Tuesday, but the CME’s May futures contracts expire tomorrow. Some of the real estate transactions that will be summarized in Tuesday’s numbers are up to five months old, where our data is at most one week old. This is why Altos Research calls its statistics “real time,” since it is an order of magnitude more current than the benchmark in real estate data.

Below is a table of our forecasts for six of the Case-Shiller futures contracts. Check back in a few days, when I will compare with the actual March, 2011 numbers.

Metro Area Feb-2011 CS HPI Forecast Signal
Boston, MA 149.86 -2.33% 111bps below the future’s spot bid price
Chicago, IL 113.26 -1.28% in the spread
Denver, CO 121.26 -3.31% 64bps below the future’s spot bid price
Las Vegas, NV 98.28 -3.26% 96bps below the future’s spot bid price
Los Angeles, CA 168.25 -8.64% 763bps below the future’s spot bid price
San Diego, CA 155.05 +1.66% 209bps above the futures spot ask price
(all spot prices as of 10:30am PST on 26-May-2011)

Sour Grapes: Seven Reasons Why “That” Twitter Prediction Model is Cooked

The financial press has been buzzing about the results of an academic paper published by researchers from Indiana University-Bloomington and Derwent Capital, a hedge fund in the United Kingdom.

The model described in the paper is seriously faulted for a number of reasons:

1. Picking the Right Data
They chose a very short bear trending period, from February to the end of 2008. This results in a very small data set, “a time series of 64 days” as described in a buried footnote. You could have made almost 20% return over the same period by just shorting the “DIA” Dow Jones ETF, without any interesting prediction model!

There is also ambiguity about the holding period of trades. Does their model predict the Dow Jones on the subsequent trading day? In this case, 64 points seems too small a sample set for almost a year of training data. Or do they hold for a “random period of 20 days”, in which case their training data windows overlap and may mean double-counting. We can infer from the mean absolute errors reported in Table III that the holding period is a single trading day.

2. Massaging the Data They Did Pick
They exclude “exceptional” sub-periods from the sample, around the Thanksgiving holiday and the U.S. presidential election. This has no economic justification, since any predictive information from tweets should persist over these outlier periods.

3. What is Accuracy, Really?
The press claims the model is “87.6%” accurate, but this is only in predicting the direction of the stock index and not the magnitude. Trading correct directional signals that predict small magnitude moves can actually be a losing strategy due to transaction costs and the bid/ask spread.

They compare with “3.4%” likelihood by pure chance. This assumes there is no memory in the stock market, that market participants ignore the past when making decisions. This also contradicts their sliding window approach to formatting the training data, used throughout the paper.

The lowest mean absolute error in predictions is 1.83%, given their optimal combination of independent variables. The standard deviation of one day returns in the DIA ETF was 2.51% over the same period, which means their model is not all that much better than chance.

The authors also do not report any risk adjusted measure of return. Any informational advantage from a statistical model is worthless if the resulting trades are extremely volatile. The authors should have referenced the finance and microeconomics literature, and reported Sharpe or Sortino ratios.

4. Backtests & Out-of-sample Testing
Instead of conducting an out-of-sample backtest or simulation, the best practice when validating an un-traded model, they pick the perfect “test period because it was characterized by stabilization of DJIA values after considerable volatility in previous months and the absence of any unusual or significant socio-cultural events”.

5. Index Values, Not Prices
They use closing values of the Dow Jones Industrial Average, which are not tradable prices. You cannot necessarily buy or sell at these prices since this is a mathematical index, not a potential real trade. Tracking errors between a tradable security and the index will not necessarily cancel out because of market inefficiencies, transaction costs, or the bid/ask spread. This is especially the case during the 2008 bear trend. They should have used historic bid/ask prices of a Dow Jones tracking fund or ETF.

6. Causes & Effects
Granger Causality makes an assumption that the effects being observed are so-called covariance stationary. Covariance stationary processes have constant variance (jitter) and mean (average value) across time, which is almost precisely wrong for market prices. The authors do not indicate if they correct for this assumption through careful window or panel construction.

7. Neural Parameters
The authors do not present arguments for their particular choice of “predefined” training parameters. This is especially dangerous with such a short history of training data, and a modeling technique like neural networks, which is prone to high variance (over-fitting).

Getting Bought

I am happy to announce that I just signed the paperwork to transfer my machine learning software to Altos Research, where my position is now Director of Quantitative Analytics. Altos and I have been working together on a contract basis since last November, when I started forecasting with the Altos data. The software itself (“Miri”) is my professional obsession — a programming library for data mining, modeling and statistics. Miri was the core of the FVM product we released in February (http://www.housingwire.com/2011/02/07/altos-unveils-forward-looking-valuation-model).

Altos Research LLC is a real estate and analytics company founded back in 2005. We are about 15 people in Mountain View, CA who collect and analyze live real estate prices and property information from the web. Altos is not just “revenue positive,” but actually profitable. We are proud to have never taken outside funding.

Altos will continue to develop Miri, but I will also focus on technical sales, business development and my own trading portfolio. We have a serious opportunity to change the way the financial industry’s dinosaurs do modeling. I am still your friendly neighborhood data guy, just now mostly thinking about real estate.

My personal blog is at “http://blog.someben.com/”, where you can read my ramblings. And I talk shop on the Altos blog itself at “http://blog.altosresearch.com/”.