TradeStation Backtesting Engine At Work

The Backtesting Engine is the core software doing the backtest. 

It takes as inputs the historical price data and trading strategies.

The backtesting engine applies the trading strategies to the historical price data to get a series of hypothetical trades and records the results.

The outputs of the backtesting engine are typically performance statistics.    I have added instrumentation to gather additional information about each trade for later analysis.

Many backtesting engines are available commerically.    Well-known platforms include TradeStation, Worden Blocks BackScanner, WeathLab, Amibroker.    Many brokers such as TD Ameritrade offer backtesting engines for their customers.

In spite of the wide availability of software, many traders don’t backtest because of the huge amount of work it takes.    Reading a Backtesting Report is much easier. 

Last updated 11/11/08.

Tags: Amibroker, backtesting, data, instrumentation, TD Ameritrade, TradeStation, trading, WealthLab, Worden Blocks Backscanner

Wilshire 5000 Weekly Chart 4/2004 - 5/2008

Most mutual funds quote their performance versus a benchmark — an index which most closely represents the holdings of the fund.

Extra Insight:

Grading a manager solely on performance versus the benchmark is known as relative returns.    This allows managers to quote that they beat the market when the merely lost less than their benchmark index.  

Absolute returns are calculated without reference to the benchmark.   For private individuals and arguably anyone else whose money is on the line, its really absolute returns that matter.

For my backtesting, the Wilshire 5000 is the most relevant index because it represents the widest selection of stocks as does my backtesting data set.   I don’t consider dividends in the backtesting and therefore don’t consider dividends from the index either.

In addition to comparing to holding a market index, its important to compare backtesting results against a very simple baseline strategy to get insight into the basic market behavior of the time period under test.

Last updated 11/11/08.

Tags: backtesting, baseline, data

The Chart Time Scale refers to the amount of time covered by each bar in a price chart. For example, here is a collection of three charts with different time scales for AAPL.

Weekly Chart

The bars on the weekly chart open with Monday’s opening price and close with Friday’s closing price.

Daily Chart

The bars on the daily chart open with each day’s opening price and close with that day’s closing price.   This is the most frequently shown chart time scale.

15 Minute Chart .Daily Chart

 On a 15-minute chart, each bar is 15 minutes wide. The first bar of the day opens with the market open at 9:30am and closes 15 minutes later at 9:45. The next bar opens at 9:45am and closes at 10:00am. This continues throughout the day with 26 bars showing the trading for the 6.5 hour regular session of each day.

Extra Insight:

Since I am using End-of-Day data for backtesting, the corresponding chart time scale is daily.

Different vendors calculate the weekly charts their own ways. TradeStation, shown above, always starts the week on Monday open and ends on Friday close. I believe this is the usual way of doing it. Telechart, however, closes the weekly bar as of the most recent close and uses the open from 5 days ago as the open of the weekly bar. So a weekly chart on a Wednesday in Telechart shows Thursday – Wednesday “weeks” for every bar on the chart all the way back.

Last updated 11/11/08.

Tags: backtesting, data, Telechart, TradeStation

Most backtesting engines assume a commission is applied to each trade.    The commission assumption is applied each time a stock is bought or sold.   

Extra Insight:

For my backtesting, I assumed $0.005 per share.  No typo, that’s one-half a cent per share.   This is offered by Interactive Brokers.

Last updated 11/11/08.

Tags: backtesting

Correlation measures how well two things move together.    For example, if gold miner stocks usually rise as the price of gold rises, we say they are correlated.    If bonds rise as stocks fall, we say they are negatively correlated. 

Using statistics, we can measure the degree of correlation.    The scale is a range between

• -1 (moving exactly opposite)
• 0 (completely independent) 
• 1 (moving in lock-step in the same directions)

Extra Insight:

Correlation doesn’t say anything about cause and effect.   Just because two things are highly correlated, it doesn’t mean that one causes the other, or that they will be as highly correlated in the future.  If two things are not correlated, that says one does not cause the other.

Its important to stock trading and backtesting to find two things that move together — that can be a trading opportunity!   A high correlation between trading signal and profit is exactly what we want. 

However, its possible that the correlation is due to chance which won’t make a good trading rule at all. 

To find a trading strategy which is likely to really work requires disciplined backtesting and statistical analysis.

Tags: backtesting, statistics

 Curve-fitting in general is the process of finding the (mathematical) description which best matches a given set of data.    When its not applied to trading strategies, it can be a very useful way of drawing conclusions from experimental data.

 When applied to trading strategies, curve-fitting can produce over-optimized, over-optimistic results.   In any set of price data, there is some “magic”  combination of indicators and parameters that catches most every move and shows outstanding results.    Unfortunately, that magic formula is the result of chance and is different for every data set.   That means that future results probably won’t come close to the numbers generated with the full benefit of hindsight.

Extra Insight: 

There’s a fine line here.   On the one hand, we want to use backtesting to see how trading strategies performed in the past with an eye to picking the best one to trade.    On the other hand, we don’t want to trade a fantasy strategy that has little chance of working in the future.

I’m using the term curve-fitting as the negative connotation of over-optimization and data-mining as the positive connotation of selecting the best of many strategies via backtesting. 

Here are three things I do to help avoid the pitfalls of curve-fitting:

• Out-of sample testing, e.g. test and compare results across multiple time periods.
• Select parameters which fall in the middle of a range of good parameters.   Avoid the outlier settings that produce much better results than their neighbors.
• Forward-test new trading strategies in live trading with small amounts before committing to full size trades.

See Technical Traders Guide to Computer Analysis of the Futures Marketsfor more against curve-fitting.

(Backtesting Blog is an Amazon Associate.)

Last updated 11/11/08.

Tags: backtesting, curve-fitting, data, forward-testing, indicator, mining, probability, sizing, strategy, trading

Data-mining is the process of selecting the best of many strategies via backtesting.   Each strategy is tested across the same stocks and time periods.  Results are compared, and the best strategy from (proper) backtesting is likely to be the best strategy to trade live.   

Live trading performance numbers will vary as the market will not behave exactly like the historical price data used in the backtest, as well as other factors.  

Extra Insight:

Care must be taken to design the process to pick a robust strategy that will work under real-life market conditions going forward.   Limiting the number of degrees of freedom, e.g. the number of parameters, is one helpful tactic.    Another is to use advanced mathematics to gauge the statistical significance of the results.

David Aronson’s Evidence-Based Technical Analysis is an excellent reference for taking a scientific approach to data mining.  It also contains a chapter on estimating the statistical significance of trading strategies selected by data-mining.    Aronson says that data mining can identify the best strategy but, because results will have an upward ”data-mining bias”, they should not be used to estimate the performance of that strategy.   Thus only relative comparisons between strategies are possible.

I backtested a baseline strategy and use it as a reference for comparison.

(Backtesting Blog is an Amazon Associate.)

Last updated 11/11/08.

Tags: backtesting, curve-fitting, data, forward-testing

In the backtesting reports, the Data Vendor field identifies the source of the Historical Price Data used for the particular run.

Extra Insight:

Historical Price Data comes with the major backtesting engines such as TradeStation.   Data can also be purchased separately from companies that focus on data, such as CSI Data.  The stock exchanges themselves make their data available as well and this is the ultimate source of the information.

Tags: backtesting, data

The Direction of trades in a backtesting run may be either long, short, or a mixture of the two.

Tags: backtesting, long, short

 Dirty Data is historical price data with errors.   Backtesting results may not be very reliable if the database looks like its been handled by this muddy Green Bay Packer!

A process to identify and remove errors from the historical price data is a necessary first step before backtesting.

(Backtesting Blog is an Amazon Associate.)

Tags: backtesting, data