Here’s another mind-numbing task when you are doing your own backtesting:  testing the tester.

This week I was forced to do something I don’t like at all.    I had to upgrade my backtesting software, TradeStation, to a new version in the middle of a set of runs.    But when my computer went whacko, upgrading TradeStation proved to be my only option to restore sanity.

The truth of the matter is that TradeStation hangs intermittently on 2 of my machines.   The other two are fine…so far.   I’ve posted the problem in the forums, taken lots of suggestions, but nothing seems to help for long.   Eventually, TradeStation gives a client access layer error, becomes unresponsive and needs to be shut down.    This leaves 2-3 TradeStation processes running which I manually kill.    Microsoft puts up a little warning box that says terminating a process unexpectedly may cause system instability, or some such.   Mostly its just fine, but do it enough times and Whacko!    I needed to re-install TradeStation and the only version available was one level up.  

In my previous career I had the priviledge of working with one of the best software engineering teams on the planet who’s customers are the best hardware engineering teams on the planet.     Many of those teams would avoid changing versions period.   But in the middle of the project, they really shied away.   When they did upgrade, as a mteer of course, many would run their own regression tests and compare the results between the old and new versions.

For this week’s upgrade, I did regression testing and am happy to report that the results of extensive backtests matched between new and old versions.   Too bad the intermittent failure still matches as well!

Even though it tested out okay this time, there’s always the potential for bugs to be introduced in a version change.   This goes not only for TradeStation but other backtesting engines and other software as well.    This is another important thing for you to think about if you are doing your own backtesting: you need to have a controlled process for introducing new software to make sure that unexpected errors aren’t introduced.

Tags: backtesting, regression, TradeStation