Curve-fitting in general is the process of finding the (mathematical) description which best matches a given set of data.    When its not applied to trading strategies, it can be a very useful way of drawing conclusions from experimental data.

 When applied to trading strategies, curve-fitting can produce over-optimized, over-optimistic results.   In any set of price data, there is some “magic”  combination of indicators and parameters that catches most every move and shows outstanding results.    Unfortunately, that magic formula is the result of chance and is different for every data set.   That means that future results probably won’t come close to the numbers generated with the full benefit of hindsight.

Extra Insight: 

There’s a fine line here.   On the one hand, we want to use backtesting to see how trading strategies performed in the past with an eye to picking the best one to trade.    On the other hand, we don’t want to trade a fantasy strategy that has little chance of working in the future.

I’m using the term curve-fitting as the negative connotation of over-optimization and data-mining as the positive connotation of selecting the best of many strategies via backtesting. 

Here are three things I do to help avoid the pitfalls of curve-fitting:

• Out-of sample testing, e.g. test and compare results across multiple time periods.
• Select parameters which fall in the middle of a range of good parameters.   Avoid the outlier settings that produce much better results than their neighbors.
• Forward-test new trading strategies in live trading with small amounts before committing to full size trades.

See Technical Traders Guide to Computer Analysis of the Futures Marketsfor more against curve-fitting.

(Backtesting Blog is an Amazon Associate.)

Last updated 11/11/08.

Tags: backtesting, curve-fitting, data, forward-testing, indicator, mining, probability, sizing, strategy, trading

Data-mining is the process of selecting the best of many strategies via backtesting.   Each strategy is tested across the same stocks and time periods.  Results are compared, and the best strategy from (proper) backtesting is likely to be the best strategy to trade live.   

Live trading performance numbers will vary as the market will not behave exactly like the historical price data used in the backtest, as well as other factors.  

Extra Insight:

Care must be taken to design the process to pick a robust strategy that will work under real-life market conditions going forward.   Limiting the number of degrees of freedom, e.g. the number of parameters, is one helpful tactic.    Another is to use advanced mathematics to gauge the statistical significance of the results.

David Aronson’s Evidence-Based Technical Analysis is an excellent reference for taking a scientific approach to data mining.  It also contains a chapter on estimating the statistical significance of trading strategies selected by data-mining.    Aronson says that data mining can identify the best strategy but, because results will have an upward ”data-mining bias”, they should not be used to estimate the performance of that strategy.   Thus only relative comparisons between strategies are possible.

I backtested a baseline strategy and use it as a reference for comparison.

(Backtesting Blog is an Amazon Associate.)

Last updated 11/11/08.

Tags: backtesting, curve-fitting, data, forward-testing

In the backtesting reports, the Data Vendor field identifies the source of the Historical Price Data used for the particular run.

Extra Insight:

Historical Price Data comes with the major backtesting engines such as TradeStation.   Data can also be purchased separately from companies that focus on data, such as CSI Data.  The stock exchanges themselves make their data available as well and this is the ultimate source of the information.

Tags: backtesting, data

 Dirty Data is historical price data with errors.   Backtesting results may not be very reliable if the database looks like its been handled by this muddy Green Bay Packer!

A process to identify and remove errors from the historical price data is a necessary first step before backtesting.

(Backtesting Blog is an Amazon Associate.)

Tags: backtesting, data

Forward-testing, means trading a strategy live with very small size to see how well the strategy (and the trader!) perform in real life.   

Forward-testing is typically done after backtesting to make sure the trading strategy is not the over-optimized result of curve-fitting or data mining.  It also gives a chance to try out the mechanics of entering, tracking and exiting trades.

Extra Insight:

I’ve heard varying advice on the size of trades for forward-testing ranging from smallest possible size - think 1 share - up to just enough to engage the trader’s emotions.

For more insight into this topic, Design, Testing, and Optimization of Trading Systemscomes highly recommended.

(Backtesting Blog is an Amazon Associate.)

Updated: 11/12/08.

Tags: backtesting, curve-fitting, data, exit, forward-testing, sizing, strategy, trading

Backtesting runs on Historical Price Data.  It is the record of actual transactions for a stock, index, commodity, or other market. 

The atomic unit of data is the bar.  Each bar represents one unit of time and has an open price, high price, low price, close price and volume.    For end-of-day data, each bar represents one day.

Extra Insight:

Data comes with the backtesting engine and is also sold separately by data vendors who make it their core business.  Data originates with the stock exchanges who keep records of each transaction.

A weekly chart can be constructed from daily data.   However, its impossible to go to smaller granularity than the available data, e.g. daily data cannot be used to create a 15-minute chart.

Updated: 11/12/08.

Tags: backtesting, data

Price and Other Indicators

An Indicator is an abstraction of historical price data which is used to gain insight into the stock or market behavior.   Examples of popular technical indicators are Moving Averages, Bollinger Bands, MACD lines and histogram, RSI, Stochastic Oscillator.

Stockcharts.com has a good comprehensive definition of technical indicators.

Extra Insight:

Indicators describe past market action in a way that can be calculated by computer.   We can objectively define trading signals in terms of the indicators.  For example, we could define a buy signal as the stock price increasing above a moving average.

Its also possible to use indicators subjectively, but that greatly diminishes their value (IMHO).

Backtesting checks market action subsequent to an indicator’s signal.  Backtesting a large number of stocks over a large time period gives insight into how the indicator performed in the past.   Backtesting can only be done with objective rules for evaluating an indicator.

Even an indicator that tested well in the past may not perform well in the future — there are no guarantees.

An objectively defined indicator can help a trader make crisp decisions and trade by a system of rules rather than be ruled by emotion.

Updated: 11/12/08.

Tags: backtesting, data, indicator, macdh, rsi, stochastic, system, technical, trading

By Instrumentation I mean adding my own special instructions into the code for the trading strategies to produce useful information for analysis. 

The commericially available backtesting engines provide limited data and what they do provide is not in a convenient format to process for thousands of runs.

Tags: backtesting, data, instrumentation, trading

Here’s the scoop straight from the SEC on limit orders. 

Extra Insight:

Limit orders are one way to reduce slippage because you specify the exact price you are willing to accept.

Backtesting with end-of-day data models limit orders reasonably well for liquid markets.    The backtesting engine will execute the whole order if the stock traded at limit the price during the day.  This is what normally happens in live trading.    Sometimes in live trading though, the full limit order doesn’t execute, but only a smaller number of shares are traded at the limit price.   This can happen if the limit order is larger than the market can bear at the time.

A small private trader is unlikely to get a partial fill in a large liquid market.   By sticking to high volume stocks, its possible to understand more about the differences between market orders and limit orders via backtesting.

There are Limit-on-Close (LOC) and Limit-on-Open (LOO) orders which corresponds to Market-on-Close and Market-on-Open orders, respectively.   I don’t use them in backtesting though.

Updated: 11/12/08.

Tags: backtesting, data, limit order, market-on-close, market-on-open, trading

A Market on Close (MOC) order is entered before the market closes and the transaction takes place at the day’s closing price.  

The US stock exchanges process these orders.  Check with your broker for exact instructions on how to enter them.

Extra Insight:

Due to using historical end-of-day data, a ”this bar at close” order in backtesting behaves similar to a Market-on-Close order because it takes today’s closing price.   A key difference is that the backtest actually “sees” the closing price before placing the order.   I wish I could do that in real life!

I use the MOC or “this bar at close” order in backtesting only for timed exits because the decision to exit in this case doesn’t depend on the closing price, only the number of days in the trade.

For large orders in thinly traded markets, a live market order might move the live market, resulting in a different closing price than would have occurred without the order – an effect that’s not modeled with historical end-of-day data.

Read a professional’s report here that live MOC orders often execute at the published closing prices.

A small private trader is unlikely to move a large liquid market.   By sticking to high volume stocks, its not only possible to backtest market orders, its also possible to understand more about the differences between market orders and limit orders via backtesting.

Updated: 11/12/08.

Tags: backtesting, data, exit, market order, market-on-close, trading