Curve-fitting in general is the process of finding the (mathematical) description which best matches a given set of data.    When its not applied to trading strategies, it can be a very useful way of drawing conclusions from experimental data.

 When applied to trading strategies, curve-fitting can produce over-optimized, over-optimistic results.   In any set of price data, there is some “magic”  combination of indicators and parameters that catches most every move and shows outstanding results.    Unfortunately, that magic formula is the result of chance and is different for every data set.   That means that future results probably won’t come close to the numbers generated with the full benefit of hindsight.

Extra Insight: 

There’s a fine line here.   On the one hand, we want to use backtesting to see how trading strategies performed in the past with an eye to picking the best one to trade.    On the other hand, we don’t want to trade a fantasy strategy that has little chance of working in the future.

I’m using the term curve-fitting as the negative connotation of over-optimization and data-mining as the positive connotation of selecting the best of many strategies via backtesting. 

Here are three things I do to help avoid the pitfalls of curve-fitting:

• Out-of sample testing, e.g. test and compare results across multiple time periods.
• Select parameters which fall in the middle of a range of good parameters.   Avoid the outlier settings that produce much better results than their neighbors.
• Forward-test new trading strategies in live trading with small amounts before committing to full size trades.

See Technical Traders Guide to Computer Analysis of the Futures Marketsfor more against curve-fitting.

(Backtesting Blog is an Amazon Associate.)

Last updated 11/11/08.

Tags: backtesting, curve-fitting, data, forward-testing, indicator, mining, probability, sizing, strategy, trading

Monte Carlo Simulation is a method of stress-testing a trading strategy.   The general idea is to use random data to construct a larger sample space built according to the same results distribution as the original sample.   This more clearly shows the effects of chance on potential outcomes and gives a broader set of data to make decisions.

Monte Carlo methods may be applied at different places in the trading strategy development progress.

One way to apply Monte Carlo methods to backtesting results is to randomly re-sample trades.  Start with the distribution of results for a backtest.  Rather than go  trade to see what happens next, we can run simulated trades.  Tens of thousands of simulated trades.  The result of each simulated trade is generated randomly according to the actual distribution found in the backtesting run.  Then plot the results distribution of all the Monte Carlo simulations to see the broad range of possible outcomes for the trading strategy.

Monte Carlo simulation may also be used to assess the statistical significance of backtesting results.  The process is advocated in Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signalsand described in detail in this paper by Dr. Timothy Masters.  

Extra Insight:

Rather than try to digest the raw results of 100,000+ trades, set boundaries on potential outcomes and use the Monte Carlo method to assess the likelihood of a trading strategy producing those results.   For example, if we define a catastrophic loss as 50% of account value, we can keep track of the number times that happens in 10,000 runs of 1,000 trades each, for example.    That’s one estimate of the probability that the trading strategy will “blow up” in the future.

Of course, the market in the future may not follow the same probability distribution as our initial sample!   Also, we backtest stocks one at the time but a portfolio holds multiple stocks which may move together so the method described above doesn’t exactly model real life.    It is a useful approximation, however.  

For a more comprehensive definition see Wikipedia for Monte Carlo Method and Monte Carlo applied to finance.    For motivation in very accessible terms see Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets

(Backtesting Blog is an Amazon Associate.)

Updated 11/12/08.

Tags: backtesting, data, loss, monte carlo simulation, probability, statistics, strategy, technical, trading

The Probability section of my backtesting report breaks down into two mathematical constructs: expectancy and standard deviation.  The intent is to give insight into the relative performance of a trading strategy over a large number of trades, and to help traders select profitable strategies (positive expectancy).  

Extra Insight: 

The most desirable case is a high expectancy with a low standard deviation.

Like the toss of a coin, a strategy may vary considerably from the reported results for a small sample of trades.

For a full definition of probability theory, get a textbook on the topic.

(Backtesting Blog is an Amazon Associate.)

Updated on 11/12/08.

Tags: expectancy, probability, standard deviation