System Development

Firstly, just a statement to clarify the intention of this blog.
Anything mentioned on this blog cannot be interpreted as financial advice.
Any securities mention therein is not a recommendation to buy or sell.
I am not licensed to give financial advice.

This blog is simply a journal and the documentation of my journey of systems development and mechanical trading.
When I do begin trading live in real-time, I will post which securities I buy and sell for the purpose of having an online track record by which my results can be verified.

Now lets get onto system development.

There are 5 stages of systems development.

1. Ideas.

Every great system begins as an idea.

2. Design.

The idea is then put into a form (code) which can be used for backtesting.

3. Testing.

Testing is done over a certain timeframe in the past. How far back to backtest is a commonly asked question for which there is no hard and fast answer. Firstly it depends on how long-term your system is. If you are trading intraday (ie. Daytrading), using 15-minute bars, then every trading day (assuming ASX stocks) contains 24 bars of data, each bar with details of the open price, close price, high, low, and volume. So perhaps 6-12 months of data is enough for backtesting. For weekly system, you are looking at about 5-10 years.

While its important to use enough data so your system can take on enough trades to be considered statistically significant, its important not to go too far back as market dynamics are more likely to have changed the further back you go.

Data must be spilt up into in-sample (which is the data on which the testing and optimisation is carried out on) and out-of-sample data.

Is it best to leave aside the most recent data as out-of-sample data for the walk-forward analysis.

4. Optimisation.

This is the stage where each parameter in the system rules are optimised. AmiBroker can do this very easily by creating those pretty 3D charts.

I use MetaStock coupled with TradeSim so the process is a little more laborious.

An example of optimisation is, say that your system uses the cross of the long-term moving average as an exit. At this stage you don't know which length moving average is best to use. So you try 150 periods, 200 periods and 250 periods, and in each test, you keep every other parameter the same.

When you find which parameter works best, that means you have optimised for that parameter.

One danger of optimisation is over-optimisation, also referred to as curve-fitting. This usually happens when you attempt to get perfect results by adding in too many parameters or conditions, to the system.

Walk-forward analysis will indicate whether indeed the system is robust or whether it has been curve-fitted to the in-sample data set.

5. Walk-forward analysis.

Up until this stage, we have no idea whether or not a system will be profitable going into the future. We just know that the system would have been profitable over the timeframe over which it has been tested and optimised.

The out-of-sample data, on which the walk-forward analysis will be performed, is brand new data that the system has not yet seen before. This should proceed immediately after where the in-sample period ends.

For example, you test and optimise on data from 1998-2002 (5 years) and then you leave 2003-2007 (almost 4 years) for the walk-forward analysis.

The results from the walk-forward analysis are very useful in determining the future profitability of the system. Because its exactly the sample as making the last 5 years as an in-sample data set, and using tomorrow and beyond (the next 5 years) as an out-of-sample data set. Tomorrow and beyond is exactly like the out-of-sample data, brand new data that the system has not seen before.

The only difference is that when you start trading tomorrow based only upon results of in-sample data tests, you are trading a system and risking real dollars with a system that has not been validated.

Walk-forward analysis is what validates a trading system.

Howard Bandy explains this nicely in his book "Quantitative trading systems" (2007).