A Market on Open (MOO) order is entered before the market opens and the transaction takes place at the day’s opening price.  

The US stock exchanges process these orders.  Check with your broker for exact instructions on how to enter them.

Extra Insight:

Due to using historical end-of-day data, a next day market order in backtesting behaves most like a Market-on-Open order because it takes the opening price.   

For large orders in thinly traded markets, a live market order might move the live market, resulting in a different opening price than would have occurred without the order – an effect that’s not modeled with historical end-of-day data.

Several traders report that live MOO orders don’t always execute at the published open prices.   Read a trader complain of worse prices here, and a professional find better prices here.

Updated 11/12/08.

Tags: backtesting, data, market order, market-on-open, trading

Here’s the scoop straight from the SEC on market orders. 

Extra Insight:

Backtesting with end-of-day data differs from live trading with respect to market orders.   No way can an order in backtesting move the market — it is assumed to execute at the historical price.  One way to account for this discrepency is to specify a slippage assumption that the backtesting engine applies to each trade.

Another approach is to reduce slippage by trading in liquid markets — I look for a volume of 500,000 shares to trade.

A small private trader is unlikely to move a large liquid market.   By sticking to high volume stocks, its not only possible to backtest market orders, its also possible to understand more about the differences between market orders and limit orders via backtesting.

Updated: 11/13/08.

Tags: backtesting, data, market order, slippage, trading

Monte Carlo Simulation is a method of stress-testing a trading strategy.   The general idea is to use random data to construct a larger sample space built according to the same results distribution as the original sample.   This more clearly shows the effects of chance on potential outcomes and gives a broader set of data to make decisions.

Monte Carlo methods may be applied at different places in the trading strategy development progress.

One way to apply Monte Carlo methods to backtesting results is to randomly re-sample trades.  Start with the distribution of results for a backtest.  Rather than go  trade to see what happens next, we can run simulated trades.  Tens of thousands of simulated trades.  The result of each simulated trade is generated randomly according to the actual distribution found in the backtesting run.  Then plot the results distribution of all the Monte Carlo simulations to see the broad range of possible outcomes for the trading strategy.

Monte Carlo simulation may also be used to assess the statistical significance of backtesting results.  The process is advocated in Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signalsand described in detail in this paper by Dr. Timothy Masters.  

Extra Insight:

Rather than try to digest the raw results of 100,000+ trades, set boundaries on potential outcomes and use the Monte Carlo method to assess the likelihood of a trading strategy producing those results.   For example, if we define a catastrophic loss as 50% of account value, we can keep track of the number times that happens in 10,000 runs of 1,000 trades each, for example.    That’s one estimate of the probability that the trading strategy will “blow up” in the future.

Of course, the market in the future may not follow the same probability distribution as our initial sample!   Also, we backtest stocks one at the time but a portfolio holds multiple stocks which may move together so the method described above doesn’t exactly model real life.    It is a useful approximation, however.  

For a more comprehensive definition see Wikipedia for Monte Carlo Method and Monte Carlo applied to finance.    For motivation in very accessible terms see Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets

(Backtesting Blog is an Amazon Associate.)

Updated 11/12/08.

Tags: backtesting, data, loss, monte carlo simulation, probability, statistics, strategy, technical, trading

My trading strategies follow this Naming Convention:

[Direction]_Entry_TestPeriod_[Dataset]_Exit

where:

• Direction is either L for buying long or S for selling short.   Direction is optional and if missing defaults to L.
• Entry indicates the entry strategy used.
• TestPeriod is the abbreviated years of the test data.   The data runs from May to April.  So 0407 means May 1, 2004 to April 30, 2007.
• Dataset indicates the data vendor.   It is optional and defaults to CSI Data if not used.
• Exit indicates the exit strategy used.

If one of the above field’s parameters are varied during the test, the exact settings for the run are shown next to it.   If settings are not given, then the commonly used settings apply.

For example, L_All_9404_CSI_Timed_200day

• Trades Long (enters by buying stock)
• Enter always
• Spans the time period  May 1, 1994 – April 30, 2004
• Runs on CSI Data
• Exits on a specific time setting of 200 days

Another example, MACDH_0407_ATR3

• Trades Long (enters by buying stock)
• Enter when MACDH ticks up, settings 12, 26, 9
• Spans the time period  May 1, 2004 – April 30, 2007
• Runs on CSI Data
• Exits on a trailing ATR stop of 3

Updated 11/12/08.

Tags: ATR, backtesting, data, exit, indicator, long, macdh, stop, strategy, trading

Out-of-sample testing is a way to guard against curve-fitting.   Its a good practice because we don’t know how the market will go in the future. When we ultimately trade our strategy it will be on live data as it evolves, not on the historical price data used for backtesting.

Here’s how out-of-sample testing works:  First a backtest is performed on a given test period.    Then the same backtest is run on a new test period — a different sample of data, hence the name.     If the parameters or settings were over-optimized in the first backtest, its unlikely that they will perform well in the second time period.   

For example, its possible to tweak the parameters on just the right indicators to make over 1000% gains in backtesting.    But when we run those same settings in another period, it might actually lose money.   If it is custom fit to one set of data, it won’t work as well in a different set of data.  Much better to find that out with an additional backtesting run rather than live trading!

Extra Insight:

With two different time periods, the results are almost always going to be at least a little different.   

The most challenging situation is if the original sample is a bull market and the out-of-sample is a bearish period (or vice versa).

My backtesting reports are broken into distinctly different samples for exactly this reason.

To be completely effective, the out-of-sample data should only be used once.   Each backtest should have its own out-of-sample data because if it is used frequently, the out-of-sample data too easily becomes in-sample data.  Using Monte Carlo method is better in this respect.    See Evidence-Based Technical Analysis: Applying the Scientific Method and Statistical Inference to Trading Signalsfor more information.

(Backtesting Blog is an Amazon Associate.)

Updated 11/12/08.

Tags: backtesting, data, indicator, mining, out-of-sample testing, strategy, trading

A Stop Order typically comes in two flavors:  a stop loss which turns into a market order when price goes below it, and stop limit which turns into a limit order for the stop price. 

A Stop order may be either a buy or a sell.     It may be used to enter or exit a position. 

Check with your broker for the exact commands to use for your own trading. 

Extra Insight:

The stop order is typically thought of for exiting a trade, however, it can also initiate a trade.   For example, a trader buying on new highs may set a buy stop slightly above the current high.   If the price hits the new high, the stop is triggered and the buy order executes.    In this case, the cool guys say the stop was “lifted”.

Many traders use a stop order to cut losses.   Some traders also “trail” the stop by moving up the stop trigger as the stock price goes up to protect partial profits.

In backtesting, I do not use the stop limit order, just the regular stop order.    The historical price data cannot show the potential effect of our stop might on a live market — that is a known inaccuracy.   Even so, backtesting can show the benefits and trade-offs of using stop orders.

Click here for BackTesting Reports on Stop Losses

(Backtesting Blog is an Amazon Associate.)

Updated 11/13/08.

Tags: backtesting, data, entry, exit, loss, protect profits, stop, trading

 Test Period and Time Period both refer to the range of historical price data used for a particular backtesting run.  For example, 1999-2001, or May 1, 2004 – Apr 31, 2007.

Extra Insight:

To make apples-apples comparisons between runs, the test period needs to be exactly the same.

Most backtesting tools foster a lax sense of test periods.  They bury the date contols, encouraging you to just use what comes up.   By default, the tests go how-ever-far-back to today which means that the same test run today as yesterday will produce slightly different results.  The error from this shifting window builds up over time, especially for extensive market testing that takes months to get through all the different runs.   

I actually did a big round of testing 18 months ago and if I did not take steps to control my time period, I would now be running on only 25% of the original data!   Fortunately, I did take the trouble to stake out specific test periods and essentially quarentine the data so that I can make valid comparisons from run to run.

Another reason for setting specific test periods is to avoid curve fitting and over-optimization by doing out-of-sample testing.   That means running on two different time periods to make sure that good results in the first period are okay in the second time period and hence have a chance at persisting into the future.

Choice of test period strongly influences results as the market behavior differs.  One way is to try to isolate time periods of rising, falling and sideways markets.   Another method chooses time periods that include all three behaviors.

I started out with a ten-year window which includes all behaviors: May 1994 – Apr 2004.   I coupled that with a three-year anti-curve-fitting window of May 2004 – Apr 2007.    Lately, I’ve added a third window of May 2007 – May 2008.

Updated 11/13/08.

Tags: backtesting, curve-fitting, data, test period

I start this blog while immersed in the early phases of my fourth major US stock market backtesting effort. 

The purpose of the blog is to record my key decisions and tactics for backtesting.   I intend it to be a resource for traders and active investors . I hope that others will learn from my efforts and we all learn from each others’ comments and discussion.

My goals for backtesting are:

1. Design trading strategies for my own use.   Specifically, 

• US Stock Market
• both buying long and short selling 
• investigate both trend following and band trading  
• swing trading: end-of-day (EOD) or daily charts and trades that last several weeks  

2. Provide information that other traders can use to develop their own systems.    That includes the areas above, and in addition, I want to illustrate for new traders such key concepts as:

• stop loss orders 
• market orders vs limit orders vs stop entry orders
• trailing stops
• price targets
• indicators like moving average, RSI, MACDH, Stochastic

3. Do this with a scope and scale that goes beyond the resources typically available to private traders, including:

• delisted stocks
• over 15 years of historical data
• clean database
• multiple time periods to avoid curve fitting
• crude and robust strategies only…limited fussing with parameters
• statistically sound methodologies
• monte carlo simulations to generalize beyond the given data

 Let’s dive in!

(Backtesting Blog is an Amazon Associate.)

Tags: backtesting, data, goals, limit order, macdh, market order, rsi, stochastic, stop loss, swing trading