Backtesting is the process of checking a trading strategy by applying the strategy to historical price data to generate hypothetical trades, then analyzing the results to assess the performance of the trading strategy.

The backtesting engine is the most glorified piece of the operation however it would not be successful without clean data, objective trading strateiges, statiscally sound analysis, and someone to do perform all the tests.

Although most traders concur that backtesting is useful, many traders don’t do it because of the time, expense, and expertise required.    

Backtesting provides a model of trading.   Significant differences with live trading exist such as: commissions, slippage or skid, after hours trading, dirty data, software bugs, human error.  Backtesting results don’t guarantee future performance.    

Backtesting is a screening tool that is really good at eliminating under-performing trading strategies.   If a trading strategy didn’t perform well in the past, it’s unlikely to make money in the future.     

If a set of rules does perform well in backtesting - that’s great but not the end of the game.  Traders ultimately must decide if a given strategy is suitable for their own use.

Last updated 11/11/08.

Tags: backtesting, data, skid, slippage, trading

TradeStation Backtesting Engine At Work

The Backtesting Engine is the core software doing the backtest. 

It takes as inputs the historical price data and trading strategies.

The backtesting engine applies the trading strategies to the historical price data to get a series of hypothetical trades and records the results.

The outputs of the backtesting engine are typically performance statistics.    I have added instrumentation to gather additional information about each trade for later analysis.

Many backtesting engines are available commerically.    Well-known platforms include TradeStation, Worden Blocks BackScanner, WeathLab, Amibroker.    Many brokers such as TD Ameritrade offer backtesting engines for their customers.

In spite of the wide availability of software, many traders don’t backtest because of the huge amount of work it takes.    Reading a Backtesting Report is much easier. 

Last updated 11/11/08.

Tags: Amibroker, backtesting, data, instrumentation, TD Ameritrade, TradeStation, trading, WealthLab, Worden Blocks Backscanner

Click here to download the baseline issue of BackTesting Report free without registration.  

The Baseline is the backtesting results from a very simple trading strategy.  We use this as a basis for comparision.   We do this to weed out the trading strategies that look good because they happened by chance to be in sync with the market over the test period.   Instead we want to find the strategies that add value other than just riding the market.

Trading strategies must test out better than the baseline to be considered for live trading.

For example, an entry strategy with a 55% win rate might sound good by itself.   But if the baseline had a 60% win rate over the same stocks and time period, that 55% strategy is actually a loser!

Extra Insight:

We test a simple strategy that enters the market at every opportunity and blindly exits at the end of specified hold periods.   We choose the hold periods to match popular trading styles.  This gives a win rate baseline.   It also shows the market’s directional bias for the test period.  We calculate the expectancy although this is not a real trading strategy.

This sample strategy has some dependence on the start date.    We can measure it to see the impact.   We can also reduce the start date impact by doing a random re-sampling using Monte Carlo simulation to get a more robust baseline.

The difference between “buy and hold” of a benchmark index and the baseline strategy is that the baseline takes into account the transaction costs of commission and slippage.  The baseline also spends a fraction of the time out of the market between trades — overnight in the case of End-of-Day data.

Win rate is my comparision metric for entry strategies.  

Expectancy is my metric for comparison for exit strategies.

Last updated 02/04/09.

Tags: baseline, data, expectancy, monte carlo, trading, win rate

 Curve-fitting in general is the process of finding the (mathematical) description which best matches a given set of data.    When its not applied to trading strategies, it can be a very useful way of drawing conclusions from experimental data.

 When applied to trading strategies, curve-fitting can produce over-optimized, over-optimistic results.   In any set of price data, there is some “magic”  combination of indicators and parameters that catches most every move and shows outstanding results.    Unfortunately, that magic formula is the result of chance and is different for every data set.   That means that future results probably won’t come close to the numbers generated with the full benefit of hindsight.

Extra Insight: 

There’s a fine line here.   On the one hand, we want to use backtesting to see how trading strategies performed in the past with an eye to picking the best one to trade.    On the other hand, we don’t want to trade a fantasy strategy that has little chance of working in the future.

I’m using the term curve-fitting as the negative connotation of over-optimization and data-mining as the positive connotation of selecting the best of many strategies via backtesting. 

Here are three things I do to help avoid the pitfalls of curve-fitting:

• Out-of sample testing, e.g. test and compare results across multiple time periods.
• Select parameters which fall in the middle of a range of good parameters.   Avoid the outlier settings that produce much better results than their neighbors.
• Forward-test new trading strategies in live trading with small amounts before committing to full size trades.

See Technical Traders Guide to Computer Analysis of the Futures Marketsfor more against curve-fitting.

(Backtesting Blog is an Amazon Associate.)

Last updated 11/11/08.

Tags: backtesting, curve-fitting, data, forward-testing, indicator, mining, probability, sizing, strategy, trading

The Dollar Trailing Stop is one way to limit losses and protect profits. A stop loss order is set a given dollar amount away from the current stock price per share. As the price moves in the trade’s favor, the stop rachets along with, never giving ground once its protected by the stop. For example, after buying long, a trader may set a trailing stop $1 below the current price. As the price moves up, the trader moves up the stop but never moves it down when the price goes down. Eventually the price does retrace the $1, the stop is hit, and the trade exits.

Extra Insight:

In backtesting, the same dollar stop value is applied to all stocks. This is not ideal because each stock has a different daily price range.  For example, setting the stop $1 away from the price of a $10 stock makes a fairly wide stop but the same $1 stop on a $100 stock is very tight.

As with all trailing stops, the dollar trail never exits at the extreme of a movement. Hence it always gives back some of the profits.

Click here for BackTesting Reports on Trailing Stops

(Backtesting Blog is an Amazon Associate.)

Last updated 11/11/08.

Tags: backtesting, exit, long, loss, stop, trading

A Drawdown is a dip in account value from its highest point.

Wikipedia has a very rigorous definition here.

Extra Insight:

An open drawdown is calculated by taking the current market value of both open and closed positions.

Two tricky things about drawdowns:  knowing how much you can handle, and estimating how much you might be in for.  That’s the key to pick a trading system (strategy and sizing) that doesn’t risk more than you can afford to lose.

During backtesting, keeping track of adverse excursions, or how far winning trades go in the wrong direction can give some insight into potential drawdowns of a trading strategy.  

To be absolutely sure to avoid devastating drawdowns its necessary to limit the amount of money in play.

Each person must define what level is devastating for themselves however 50% loss is often called “blowing up”.

Last updated 11/11/08.

Tags: backtesting, drawdown, loss, risk, sizing, system, trading

  An Entry Strategy is the set of rules specifying the conditions to enter a trade. 

For a long trade, entering means buying a stock.   For a short trade, entering means selling the stock.

Extra Insight:

Having a strategy for entry allows a trader to plan with a cool head rather than getting caught up in the heat of the moment.   Backtesting the entry strategy gives a trader insight and confidence in the plan.

The main goal of an entry strategy is getting into profitable trades.  on the flip side, it is useful to stay out of losing trades, making it a Do-Not-Enter Strategy as well.

(Backtesting Blog is an Amazon Associate.)

Updated 11/12/08.

Tags: backtesting, entry, goals, long, short, strategy, trading

 The Exit Strategy is a well-defined plan specifying the conditions to get out of a trade.  

 For a long trade, exiting means selling the stock. For a short trade, exiting means buying a stock.

Extra Insight:

Having a strategy for exit allows a trader to plan with a cool head rather than getting caught up in the heat of the moment.   Backtesting the exit strategy gives a trader insight and confidence in the plan.

Most traders have two purposes for exiting:  taking profits and cutting losses. 

Sometimes both ends are served by one exit order, such as a trailing stop.    Other times, they are two distinct orders, such as a fixed stop loss and a target limit order.

A third goal of an exit strategy may be the efficient use of capital.    In that case, the exit strategy may have rules to exit a trade that isn’t going anywhere in order to redeploy the resources elsewhere.

Click here for BackTesting Reports on Exit Strategies

(Backtesting Blog is an Amazon Associate.)

Updated 11/12/08.

Tags: backtesting, exit, goals, loss, stop, strategy, trading

Expectancy measures a trading strategy’s profit potential.   It considers both the reliability or win rate as well as the amount gained by each win.    That way, it can compare trading strategies that often win small gains with strategies that rarely win but win big when they do. 

Expectancy = (win_rate * avg_win) - (loss_rate * avg_loss)

Van Tharp defines expectancy in terms of risk here, as the average of the R-multiples returned by trading or backtesting the system.

Extra Insight:

Over a large number of trades, the expectancy is the expected gain of the trading strategy.  Higher expectancy is generally better.  Always avoid trading strategies with negative expectancy.

Scaling the expectancy by risk is indeed useful, especially when it comes time to compare different systems.  I use the R-multiples as suggested by Van Tharp for ease of calculation.

Expectancy is also known as the Kelly Criterion for the Bell Labs researcher who proved the equation as an upper bound on the amount to risk.     A common language way to say it is to risk an amount proportional to the expected gain.   So if the expectancy is 45%, Kelly advocated risking 45% of the account value.   This may be mathematically optimal over a large number of trades but it can have a very vicious drawdown!   Imagine trading a high expectancy system, say 80% and the first trade is a loss.  For a $100k account, that would leave only $20k in the account and a long road to make a 4x gain to break even.

Expectancy is not the be-all and end-all of a trading system.   The standard deviation or variance of the results is important.  The win rate is too.  Both give insight into how psychologically difficult it is to stick to the trading strategy.

Updated: 11/12/08.

Tags: backtesting, expectancy, Kelly criterion, long, R-Multiple, risk, strategy, system, trading, Van Tharp

Forward-testing, means trading a strategy live with very small size to see how well the strategy (and the trader!) perform in real life.   

Forward-testing is typically done after backtesting to make sure the trading strategy is not the over-optimized result of curve-fitting or data mining.  It also gives a chance to try out the mechanics of entering, tracking and exiting trades.

Extra Insight:

I’ve heard varying advice on the size of trades for forward-testing ranging from smallest possible size - think 1 share - up to just enough to engage the trader’s emotions.

For more insight into this topic, Design, Testing, and Optimization of Trading Systemscomes highly recommended.

(Backtesting Blog is an Amazon Associate.)

Updated: 11/12/08.

Tags: backtesting, curve-fitting, data, exit, forward-testing, sizing, strategy, trading