Curve-fitting in general is the process of finding the (mathematical) description which best matches a given set of data.    When its not applied to trading strategies, it can be a very useful way of drawing conclusions from experimental data.

 When applied to trading strategies, curve-fitting can produce over-optimized, over-optimistic results.   In any set of price data, there is some “magic”  combination of indicators and parameters that catches most every move and shows outstanding results.    Unfortunately, that magic formula is the result of chance and is different for every data set.   That means that future results probably won’t come close to the numbers generated with the full benefit of hindsight.

Extra Insight: 

There’s a fine line here.   On the one hand, we want to use backtesting to see how trading strategies performed in the past with an eye to picking the best one to trade.    On the other hand, we don’t want to trade a fantasy strategy that has little chance of working in the future.

I’m using the term curve-fitting as the negative connotation of over-optimization and data-mining as the positive connotation of selecting the best of many strategies via backtesting. 

Here are three things I do to help avoid the pitfalls of curve-fitting:

• Out-of sample testing, e.g. test and compare results across multiple time periods.
• Select parameters which fall in the middle of a range of good parameters.   Avoid the outlier settings that produce much better results than their neighbors.
• Forward-test new trading strategies in live trading with small amounts before committing to full size trades.

See Technical Traders Guide to Computer Analysis of the Futures Marketsfor more against curve-fitting.

(Backtesting Blog is an Amazon Associate.)

Last updated 11/11/08.

Tags: backtesting, curve-fitting, data, forward-testing, indicator, mining, probability, sizing, strategy, trading

Data-mining is the process of selecting the best of many strategies via backtesting.   Each strategy is tested across the same stocks and time periods.  Results are compared, and the best strategy from (proper) backtesting is likely to be the best strategy to trade live.   

Live trading performance numbers will vary as the market will not behave exactly like the historical price data used in the backtest, as well as other factors.  

Extra Insight:

Care must be taken to design the process to pick a robust strategy that will work under real-life market conditions going forward.   Limiting the number of degrees of freedom, e.g. the number of parameters, is one helpful tactic.    Another is to use advanced mathematics to gauge the statistical significance of the results.

David Aronson’s Evidence-Based Technical Analysis is an excellent reference for taking a scientific approach to data mining.  It also contains a chapter on estimating the statistical significance of trading strategies selected by data-mining.    Aronson says that data mining can identify the best strategy but, because results will have an upward ”data-mining bias”, they should not be used to estimate the performance of that strategy.   Thus only relative comparisons between strategies are possible.

I backtested a baseline strategy and use it as a reference for comparison.

(Backtesting Blog is an Amazon Associate.)

Last updated 11/11/08.

Tags: backtesting, curve-fitting, data, forward-testing

Forward-testing, means trading a strategy live with very small size to see how well the strategy (and the trader!) perform in real life.   

Forward-testing is typically done after backtesting to make sure the trading strategy is not the over-optimized result of curve-fitting or data mining.  It also gives a chance to try out the mechanics of entering, tracking and exiting trades.

Extra Insight:

I’ve heard varying advice on the size of trades for forward-testing ranging from smallest possible size - think 1 share – up to just enough to engage the trader’s emotions.

For more insight into this topic, Design, Testing, and Optimization of Trading Systemscomes highly recommended.

(Backtesting Blog is an Amazon Associate.)

Updated: 11/12/08.

Tags: backtesting, curve-fitting, data, exit, forward-testing, sizing, strategy, trading

 Test Period and Time Period both refer to the range of historical price data used for a particular backtesting run.  For example, 1999-2001, or May 1, 2004 – Apr 31, 2007.

Extra Insight:

To make apples-apples comparisons between runs, the test period needs to be exactly the same.

Most backtesting tools foster a lax sense of test periods.  They bury the date contols, encouraging you to just use what comes up.   By default, the tests go how-ever-far-back to today which means that the same test run today as yesterday will produce slightly different results.  The error from this shifting window builds up over time, especially for extensive market testing that takes months to get through all the different runs.   

I actually did a big round of testing 18 months ago and if I did not take steps to control my time period, I would now be running on only 25% of the original data!   Fortunately, I did take the trouble to stake out specific test periods and essentially quarentine the data so that I can make valid comparisons from run to run.

Another reason for setting specific test periods is to avoid curve fitting and over-optimization by doing out-of-sample testing.   That means running on two different time periods to make sure that good results in the first period are okay in the second time period and hence have a chance at persisting into the future.

Choice of test period strongly influences results as the market behavior differs.  One way is to try to isolate time periods of rising, falling and sideways markets.   Another method chooses time periods that include all three behaviors.

I started out with a ten-year window which includes all behaviors: May 1994 – Apr 2004.   I coupled that with a three-year anti-curve-fitting window of May 2004 – Apr 2007.    Lately, I’ve added a third window of May 2007 – May 2008.

Updated 11/13/08.

Tags: backtesting, curve-fitting, data, test period