Forward-testing, means trading a strategy live with very small size to see how well the strategy (and the trader!) perform in real life.   

Forward-testing is typically done after backtesting to make sure the trading strategy is not the over-optimized result of curve-fitting or data mining.  It also gives a chance to try out the mechanics of entering, tracking and exiting trades.

Extra Insight:

I’ve heard varying advice on the size of trades for forward-testing ranging from smallest possible size - think 1 share – up to just enough to engage the trader’s emotions.

For more insight into this topic, Design, Testing, and Optimization of Trading Systemscomes highly recommended.

(Backtesting Blog is an Amazon Associate.)

Updated: 11/12/08.

Tags: backtesting, curve-fitting, data, exit, forward-testing, sizing, strategy, trading

Monte Carlo Simulation is a method of stress-testing a trading strategy.   The general idea is to use random data to construct a larger sample space built according to the same results distribution as the original sample.   This more clearly shows the effects of chance on potential outcomes and gives a broader set of data to make decisions.

Monte Carlo methods may be applied at different places in the trading strategy development progress.

One way to apply Monte Carlo methods to backtesting results is to randomly re-sample trades.  Start with the distribution of results for a backtest.  Rather than go  trade to see what happens next, we can run simulated trades.  Tens of thousands of simulated trades.  The result of each simulated trade is generated randomly according to the actual distribution found in the backtesting run.  Then plot the results distribution of all the Monte Carlo simulations to see the broad range of possible outcomes for the trading strategy.

Monte Carlo simulation may also be used to assess the statistical significance of backtesting results.  The process is advocated in Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signalsand described in detail in this paper by Dr. Timothy Masters.  

Extra Insight:

Rather than try to digest the raw results of 100,000+ trades, set boundaries on potential outcomes and use the Monte Carlo method to assess the likelihood of a trading strategy producing those results.   For example, if we define a catastrophic loss as 50% of account value, we can keep track of the number times that happens in 10,000 runs of 1,000 trades each, for example.    That’s one estimate of the probability that the trading strategy will “blow up” in the future.

Of course, the market in the future may not follow the same probability distribution as our initial sample!   Also, we backtest stocks one at the time but a portfolio holds multiple stocks which may move together so the method described above doesn’t exactly model real life.    It is a useful approximation, however.  

For a more comprehensive definition see Wikipedia for Monte Carlo Method and Monte Carlo applied to finance.    For motivation in very accessible terms see Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets

(Backtesting Blog is an Amazon Associate.)

Updated 11/12/08.

Tags: backtesting, data, loss, monte carlo simulation, probability, statistics, strategy, technical, trading

My trading strategies follow this Naming Convention:

[Direction]_Entry_TestPeriod_[Dataset]_Exit

where:

• Direction is either L for buying long or S for selling short.   Direction is optional and if missing defaults to L.
• Entry indicates the entry strategy used.
• TestPeriod is the abbreviated years of the test data.   The data runs from May to April.  So 0407 means May 1, 2004 to April 30, 2007.
• Dataset indicates the data vendor.   It is optional and defaults to CSI Data if not used.
• Exit indicates the exit strategy used.

If one of the above field’s parameters are varied during the test, the exact settings for the run are shown next to it.   If settings are not given, then the commonly used settings apply.

For example, L_All_9404_CSI_Timed_200day

• Trades Long (enters by buying stock)
• Enter always
• Spans the time period  May 1, 1994 – April 30, 2004
• Runs on CSI Data
• Exits on a specific time setting of 200 days

Another example, MACDH_0407_ATR3

• Trades Long (enters by buying stock)
• Enter when MACDH ticks up, settings 12, 26, 9
• Spans the time period  May 1, 2004 – April 30, 2007
• Runs on CSI Data
• Exits on a trailing ATR stop of 3

Updated 11/12/08.

Tags: ATR, backtesting, data, exit, indicator, long, macdh, stop, strategy, trading

Number of Trades or # Trades is simply the number of trades taken by the backtesting engine for a particular trading strategy during a particular time period.

Extra Insight:

I have two reasons for prominently recording the number of trades:

1. The opportunity to trade determines the opportunity to profit and also determines the requirements for capital.
2. Comparing the number of trades between backtesting runs is a good consistency check.

Updated 11/12/08.

Tags: backtesting, strategy, trading

Out-of-sample testing is a way to guard against curve-fitting.   Its a good practice because we don’t know how the market will go in the future. When we ultimately trade our strategy it will be on live data as it evolves, not on the historical price data used for backtesting.

Here’s how out-of-sample testing works:  First a backtest is performed on a given test period.    Then the same backtest is run on a new test period — a different sample of data, hence the name.     If the parameters or settings were over-optimized in the first backtest, its unlikely that they will perform well in the second time period.   

For example, its possible to tweak the parameters on just the right indicators to make over 1000% gains in backtesting.    But when we run those same settings in another period, it might actually lose money.   If it is custom fit to one set of data, it won’t work as well in a different set of data.  Much better to find that out with an additional backtesting run rather than live trading!

Extra Insight:

With two different time periods, the results are almost always going to be at least a little different.   

The most challenging situation is if the original sample is a bull market and the out-of-sample is a bearish period (or vice versa).

My backtesting reports are broken into distinctly different samples for exactly this reason.

To be completely effective, the out-of-sample data should only be used once.   Each backtest should have its own out-of-sample data because if it is used frequently, the out-of-sample data too easily becomes in-sample data.  Using Monte Carlo method is better in this respect.    See Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signalsfor more information.

(Backtesting Blog is an Amazon Associate.)

Updated 11/12/08.

Tags: backtesting, data, indicator, mining, out-of-sample testing, strategy, trading

 A Position Trader seeks intermediate-term opportunites.   Like a swing trader, the position trader is ready with a quick exit strategy but more willing to stay with a trade for several weeks or more.   

Extra Insight:  

For backtesting, I use timed 20 day exits to approximate position trading.

(Backtesting Blog is an Amazon Associate.)

Updated 11/17/08.

Tags: backtesting, exit, position trader, strategy, trading

Productivity refers to how hard your money works under a particular trading strategy.   Both the number of trades and the average hold time impact how efficiently a strategy manages funds.

Extra Insight: 

The number of trades reflects the opportunity to trade.   In general, a higher number is better for a profitable strategy because it means more opportunity to make money.   However, if a strategy is not profitable or breaks even, more trades just means more commissions paid to the broker and more of the trader’s time consumed.  (Even with completely automated trading, a trader still has to deal with record-keeping and other administrative tasks that grow with the number of trades.)

Backtesting short-term strategies over any significant time period generates many more trades than the average trader can afford.    Realistically, traders need to choose which signals to take.   Eventually, I’d like to make the selection criteria part of the backtest so that it can be measured and compared as well.    Right now, my backtesting engine, TradeStation, only tests one stock at a time and cannot simulate a whole portfolio.

All else being equal, the trading strategy with the smallest average hold time is best.   Not all things are equal in practice though.   People may seek out longer or shorter hold times based on their temperment, tax situation, global market views, margin, and a host of other factors.   The average hold time gives insight into which strategies will fit these other requirements.  I also use it to classify different strategies and compare those with similar hold times.

Updated 11/12/08.

Tags: backtesting, hold time, number of trades, productivity, strategy, TradeStation, trading

A trading strategy might have several different types of exits, among them the Profit-Taking Exit.  As the name suggests, the idea is to bag some profits.  Cha-ching!

Extra Insight:

The profit-taking exit won’t necessarily mean selling at the top.  That’s difficult, maybe impossible, to do consistently and its often called a fool’s errand to try.

Some examples of profit-taking exits are price hitting the upper channel boundary or a pre-defined target percentage gain.

Another profit-taking exit is a trailing stop.   A stop (loss) order is put in place below the current price (or above it for a short).   As the stock price moves up, the stop price moves up too.   Different methods of trailing a stop: ATR (average true range), percentage, and fixed dollar to name a few.

I’m just listing a few possibilities here, not suggesting which one to use.

Click here for BackTesting Reports on Exit Strategies

(Backtesting Blog is an Amazon Associate.)

Updated 11/12/08.

Tags: ATR, backtesting, exit, loss, percentage stop, stop, strategy, trading, trailing stop

The Results Distribution is a graphical way to show the performance of a trading strategy.   The graph above shows the results of backtesting a trading strategy across several thousand stocks over a 3-year period.

Extra Insight:

Let’s take apart a simpler example:

Each trade is assigned to a bin depending on its profit/loss results.   Different people use different measure of results: dollar gain, percent gain, etc.  I use Van Tharp’s R-Multiple which is the gain divided by the amount risked.   I label the bins with the mid-point of the range.  

For example, a trade that gains twice what was risked goes into the “1.5″ bin along with all the other trades that returned between 1 and 2 times the risk amount (R-Multiple).    

The horizontal axis shows each bin and the vertical shows the number of trades in the bin.   The red line is the zero point which separates winning and losing trades.  In some graphs, profitable bins are green and losers red.  

In our example above, we see that 196 trades returned a profit that was less than the amount risked (less than 1 R-Mult) because the bar for the “0.5″ bin is 196 trades high.   Unfortunately for this strategy, even more trades lost money, which we can tell at a glance from this graph.

In general, better strategies have more action to the right of zero on the chart – more profitable trades either in quantity or quality or both!

The Puppetmaster’s article on Redistribution is an excellent illustration comparing the results distributions for two different trading strategies.

Updated 11/12/08.

Tags: backtesting, loss, Puppetmaster, quality, Results Distribution, risk, strategy, trading, Van Tharp

Risk management is crucial because all trading strategies lose sometimes.    By limiting risk, a trader has a chance to survive long enough to find a way to thrive.   The topic is way too big for my little glossary but its here because I want to encourage traders to think about risk.

Extra Insight: 
Funny Carrie Underwood parody reminder to manage risk:

Backtesting trading strategies provides insight into the risk of various trading strategies.   Understanding risk is the first step towards managing it.   A complete understanding includes the knowledge that there’s always the risk of something completely unknown and unforeseen happening.

Different people have different risk tolerance and even the same person will view various risk differently.  That means you have to decide for yourself what to risk.

Here are a few “rules of thumb”:

• Don’t risk more than you can afford to lose.
• Limit the risk on each position.  1% of your trading account risked per stock is a middle-of-the-road estimate.  By risked, I mean the amount lost if the stop is hit.  No stop?  Then limit the size of the whole position to a small fraction of your account.
• Limit the total exposure of your account.   What happens if all your stops get hit?   What happens if the market gaps past all your stops.   Make sure you can handle it.
• “Blowing up” means losing 50% of account value.   Don’t let that happen!

Here’s a link to the book advertised in the video: Mastering the Trade (McGraw-Hill Trader’s Edge).

(Backtesting Blog is an Amazon Associate.)

Updated 11/12/08.

Tags: backtesting, long, management, risk, sizing, stop, strategy, trading