Rigorous testing methods

Its important to thoroughly and rigorously test and really put the system through its paces during the testing phase. This is the only way you can have enough confidence in the system to stick with it through those inevitable periods of drawdown. And thorough testing and examining each trade, helps you understand what makes the system profitable. As Nick Radge always says, its important to understand WHY your system works.

A great trader once told me: "Try to make your system fail".

I didn't go into this in much detail beforehand, so will do now.

When I said my system was initially back tested in the 1998-2003 six year period, and optimised on this period, the testing did not stop there.

I also run the test through different 2-year blocks and even single year tests, though single year tests do not give an accurate representation of the profitability of long term weekly systems such as this one, as the big winners often run for much more than 1 year.

The testing through 1 and 2-year blocks was an attempt to try and address start and end date biases. You don't want 6 years to look good because of one or two great years.

All my testing always includes delisted securities. I never test on the current All Ordinaries or ASX300 Indices. Quite pointless, it will give you great results, but not realistic results, which is what we are after. What I was after when I started testing was dynamic indices, with historical index constituents. Now that would be handy. We include delisted securities to overcome survivorship biases. For example, if we test using the current ASX100, which includes ZFX, we would've bought ZFX a few years ago at $2. But ZFX actually wasn't included in the ASX100 in 2005, so we wouldn't have picked it up during actual trading as its not in our universe. Stocks that are listed today can easily become delisted. In fact, my system performs better with delisted securities than without. This test was only done for my own personal interest. Shows that those dogs were once champions. The trailing stop/exit is what saves you.

The system was tested through a period I'd like to call THE WORST. Its the closest we've ever come to a bearmarket in the last 15 years on the ASX. From 01-07-2001 until 01-03-2003, is as bad as its been for sometime. XAO lost 18% during this time (21 months), its pretty much peak to trough. My system was put through 20,000 simulations. More than 82% were profitable. The average portfolio gained 8%. Not bad.

During 2001, there were 2 price shocks. First the dot.com crash and then 9/11. So a very volatile year. Market actually gained 6.98% though. For my system, more than 99% of the portfolios were profitable, with the average portfolio gaining 16%.

During 2002, the market lost about 10%. Solid downtrend. 99.97% of my 20,000 portfolios were profitable. The average portfolio gaining more than 12%.

I also test the system using worst case slippage. I always buy the high of the week and sell on the low of the week. Profitability declines by about 40%.

Another thing I like to do is remove the best 3 and worse 3 trades from the TradeSim trade database. Hardly any difference in the results. I don't want one or two champion stocks to be responsible for me making money. The largest winner contributes about 10% of the overall profit of the system over the 6 year test. I'm happy with that. You don't want 20%+ coming from just one stock.