In response to questions about choosing backtesting tools, I’ll share my experiences in the next few posts.     If you’re doing your own backtesting, its worth taking some time and effort to pick the right tool for you because you will need to use it extensively to get meaningful results.   And you risk your account equity when you trade on the results so it better not be buggy!

The first most important step when you are looking for a backtesting engine — or anything else for that matter — is to think about what you want.    Are you testing with just one commodity or across the whole stock market?   What is your budget?  How much programming flexibility do you want vs having it work very intuitively?

My criteria for this current project were:

1. Backtests correctly.    Don’t take this for granted.   Being able to verify results and spot-check each indicator test by hand is essential.
2. Open to writing custom technical analysis strategies.
3. Open to reading in outside data.    CSI data tested out as the cleanest data and that is an external data source to everything but CSI’s own tools.
4. Open to writing out calculated data beyond the scope of the canned reports.   For example, adjusting the position size based on the distance to the stop can have a huge effect on performance.   Its very helpful to be able to write out the stop placement and the size but you won’t find them in the typical vendor’s report.
5. Computer-readable output reports.   Necessary for compiling stats and drawing conclusions from a huge number of runs.  You can get around this requirement if you can write out all the data you need ala bullet #4.
6. Reproducible results.   Different runs of the same backtest should have the same results.   This means being able to lock down the version in use.
7. Can run automatically through a large list of stocks.   Sadly, most backtesting engines only run on one stock at a time.  Others allow you to add on software to run multiple tickers automatically.
8. Cost.    Personally, I’m willing to pay to get what I need so this was a lower priority.   Even so, it makes sense to choose the most affordable option that gets the job done.
9. Familiarity.    Since I already had experience with several tools, those had the inside track in my evaluation of backtesting tools.

Those were my key selection criteria.      Before reading further, take a minute to write down what you want from a backtesting tool.

Tags: backtesting, backtesting engines