Walk forward analysis

I have completed a walk forward analysis on my system, and tested it on the out-of-sample data from 01-01-2004 until 30-06-2007 (3.5 years).

This was previously untouched data that the system has never seen. It can be considered as "tomorrow's data".

The results were promising and have validated the robustness of the system.

Robert Pardo, in his excellent book, included in my recommended texts, mentions that one method of assessing out-of-sample testing results is by a ratio known as the walk forward efficiency (WFE).

WFE is simply the ratio between the objective function score on the out-of-sample testing and the objective function score on the in-sample testing. For a system to be considered validated, the WFE should be >50%.

An objective function is the outcome that we try to maximise when trading. My objective function is (profit/maximum peak-to-valley drawdown). This will differ for everybody. The WFE for my system was 0.89, or 89%.

Out-of-sample data testing should always be worse than in-sample data testing, because the system was optimised on the in-sample data. We did not stop testing and optimising until the results on the in-sample data were acceptable. But if the out-of-sample results are not acceptable, this is a sign that the system has been curve-fitted to the in-sample data, and we must start the testing process again.

Thus, 89% is a rather pleasing result, and is a sign of the robustness of the system, indicating that it has not been curve-fitted.

I'm in the middle of creating an Excel spreadsheet that will be used for trade monitoring and position sizing. Live trading should start in a few weeks.

Testing part 2

Above are some of the charts produced by a single portfolio simulation.

Note the equity curve, and monthly/yearly returns, are all on closed equity, which is generally what is important to the professional trader.

Just to explain what some of the terms mean in the previous post.

Position value limit means no position can make up more than 15% of the portfolio. This does not differentiate between stocks. So if I pyramided into one stock 3 times, this is regarded by this value as 3 positions, the same as 3 positions into 3 different stocks.


Portfolio heat means how much i were to lose if i was stopped out of all my positions. I have set this to 20% but in the trading timeframe the maximum has been 17%.


Entry and order type is market order, which means the actual entry will be a randomly generated figure between the high and the low of the price on the entry bar (weekly). Testing with these parameters gives me the flexibility to enter and exit stocks at any time during the week, not just Monday's open.


Trade processed by each simulation was 1265. This is how many trades were triggered by the system's entry. The number of executed trades is how many trades were taken, and this is governed by the amount of available trading capital.

Now, for a single simulation I can extracts all sorts of information from TradeSim, but below is the page that I'm most interested in.

Profit Summary

Profit Status: PROFITABLE
Starting Capital: $45,000.00
Finishing Capital: $281,900.17

Maximum Equity/(Date):
$236,900.17 (24/12/2003)
Minimum Equity/(Date):
$687.26 (9/04/1998)


Gross Trade Profit: $328,373.16 (729.72%)
Gross Trade Loss: -$91,472.99 (-203.27%)
Total Net Profit: $236,900.17 (526.44%)

Average Profit per Trade: $1,548.37

Profit Factor: 3.5898
Profit Index: 72.14%

Total Transaction Cost: $6,732.00
Total Slippage: $77,130.18

Total Trade Interest: $0.00

Daily Compound Interest Rate: 0.0841%
Annualized Compound Interest Rate: 35.9254%

Trade Statistics

Trades Processed: 1271
Trades Taken: 153
Partial Trades Taken: 0
Trades Rejected: 739


Winning Trades: 82 (53.59%)
Losing Trades: 71 (46.41%)
Breakeven Trades: 0 (0.00%)

Largest Winning Trade/(Date): $27,505.82 (7/12/2001)
Largest Losing Trade/(Date): -$4,233.32 (28/06/2002)

Average Winning Trade: $4,004.55

Average Losing Trade: -$1,288.35
Average Win/Average Loss: 3.1083

Trade Breakdown
Long Trades
Normal Exit: 121 (79.08%)
Protective Stop: 18 (11.76%)
Open Trade: 14 (9.15%)


Trade Duration Statistics
Winning Trades

Maximum: 1239 days

Average: 203 days

Minimum: 20 days

Losing Trades
Maximum: 245 days

Average: 63 days

Minimum: 0 days

Consecutive Trade Statistics
Maximum consecutive winning trades: 8
Maximum consecutive losing trades: 6
Average consecutive winning trades: 2.22
Average consecutive losing trades: 1.97


Trade Expectation Statistics
Normalized Expectation per dollar risked: $1.33
Maximum Reward/Risk ratio: 35.43
Minimum Reward/Risk ratio: -1.23
Average Positive Reward/Risk ratio: $3.00
Average Negative Reward/Risk ratio: -$0.59

Relative Drawdown
Maximum Dollar Drawdown/(Date): $13,533.74 (5/07/2002)
Maximum Percentage Drawdown/(Date): 5.9480% (5/07/2002)

Absolute (Peak-to-Valley) Dollar Drawdown
Maximum Dollar Drawdown: $24,202.46 (10.5600%)
Capital Peak/(Date): $229,178.79 (5/04/2002)
Capital Valley/(Date): $204,976.33 (14/03/2003)

Absolute (Peak-to-Valley) Percent Drawdown
Maximum Percentage Drawdown: 10.5600% ($24,202.46)
Capital Peak/(Date): $229,178.79 (5/04/2002)
Capital Valley/(Date): $204,976.33 (14/03/2003)

Now, to explain some key terms.

Profit factor: Ratio of gross trade profit to gross trade loss.
Profit Index: Total net profit to gross trade profit. 100% would mean no losses.


Slippage is the difference between the price you want and the price you actually pay. In our case, its the difference between the opening price of the stock compared to our actual entry price. Because our system in a trend following system, a fair bit of slippage is to be expected as we are entering traded in the direction of the trend.

Profit factor: Ratio of gross trade profit to gross trade loss.
Profit Index: Total net profit to gross trade profit. 100% would mean no losses.


Slippage is the difference between the price you want and the price you actually pay. In our case, its the difference between the opening price of the stock compared to our actual entry price. Because our system in a trend following system, a fair bit of slippage is to be expected as we are entering traded in the direction of the trend.


Amount of trades that are exited through the initial protective stop is 11%. This is healthy and means our stop isn't too tight. 10-15% is good for this figure. Lower the better.


Average trade duration. The longer you are in a trade generally the more profit you make. If winning trades are banked quicker this generally provides a smoother equity curve.


Relative drawdown is the loss in equity from a previous equity high.


Maximum string of winners and losses. Best if they are equal or favouring the winners (which is the case here). This system shouldn't have a problem here because it picks more winners (52%) than losers. This statistic would be of concern to those trading methods that pick 30-40% winners.

Testing part 1

These are the details for the backtesting of the latest version of my system.

The system periodicity is weekly.

The universe is the entire ASX, with liquidity filter of $1million turnover per week.

The timeframe is from 01-01-1998 until 31-12-2003.

Monte Carlo Report

Trade Database Filename

C:\TradeSimData\Eagle Egg ORIGINAL.trb

Simulation Summary

Simulation Date: 23/09/2007
Simulation Time: 8:37:23 AM
Simulation Duration: 286.52 seconds


Trade Parameters

Initial Capital: $45,000.00
Portfolio Limit: 100.00%
Maximum number of open positions: 15
Position Size Model: Fixed Percent Risk
Percentage of capital risked per trade: 1.50%
Position size limit: 15.00%
Portfolio Heat: 20.00%
Pyramid profits: Yes
Transaction cost (Trade Entry): $22.00
Transaction cost (Trade Exit): $22.00
Margin Requirement: 100.00%
Magnify Position Size(& Risk) according to Margin Req: No
Margin Requirement Daily Interest Rate (Long Trades): 0.0000%
Margin Requirement Yearly Interest Rate (Long Trades): 0.0000%
Margin Requirement Daily Interest Rate (Short Trades): 0.0000%
Margin Requirement Yearly Interest Rate (Short Trades): 0.0000%


Trade Preferences

Trading Instrument: Stocks
Break Even Trades: Process separately
Trade Position Type: Process all trades
Entry Order Type: Market Order
Exit Order Type: Market Order
Minimum Trade Size: $500.00
Accept Partial Trades: No
Volume Filter: Reject Trades if Position Size is greater than
10.00% of the maximum traded volume
Pyramid Trades: Yes
Favour Trade Pyramid: Yes
Start Pyramid at any level up to level: N/A
Maximum Pyramid Level Limited to: N/A
Maximum Pyramid Count Limited to: N/A


Simulation Stats

Number of trade simulations: 20000
Trades processed per simulation: 1246
Maximum Number of Trades Executed: 175
Average Number of Trades Executed: 149
Minimum Number of Trades Executed: 130
Standard Deviation: 6.87


Profit Stats

Maximum Profit: $404,529.75 (898.96%)
Average Profit: $243,636.83 (541.42%)
Minimum Profit: $108,794.38 (241.77%)
Standard Deviation: $34,823.52 (77.39%)
Probability of Profit: 100.00%
Probability of Loss: 100.00%


Percent Winning Trade Stats

Maximum percentage of winning trades: 62.58%
Average percentage of winning trades: 53.33%
Minimum percentage of winning trades: 41.62%
Standard Deviation: 2.64%


Percent Losing Trade Stats

Maximum percentage of losing trades: 58.38%
Average percentage of losing Trades: 46.67%
Minimum percentage of losing trades: 37.42%
Standard Deviation: 2.64%


Average Relative Dollar Drawdown Stats

Maximum of the Average Relative Dollar Drawdown: $2,503.85
Average of the Average Relative Dollar Drawdown: $1,119.33
Minimum of the Average Relative Dollar Drawdown: $584.95
Standard Deviation: $213.27


Average Relative Percent Drawdown Stats

Maximum of the Average Relative Percent Drawdown: 1.7230%
Average of the Average Relative Percent Drawdown: 0.9085%
Minimum of the Average Relative Percent Drawdown: 0.5263%
Standard Deviation: 0.1387%


Maximum Peak-to-Valley Dollar Drawdown Stats

Maximum Absolute Dollar Drawdown: $60,067.90
Average Absolute Dollar Drawdown: $16,956.65
Minimum Absolute Dollar Drawdown: $4,375.91
Standard Deviation: $6,799.79


Maximum Peak-to-Valley Percent Drawdown Stats

Maximum Absolute Percent Drawdown: 16.3946%
Average Absolute Percent Drawdown: 8.0761%
Minimum Absolute Percent Drawdown: 4.1432%
Standard Deviation: 1.5186%

Cutting those losses

If you can't take a small loss, sooner or later you will take the mother of all losses. The elements of good trading are: 1. Cutting losses, 2. Cutting losses, and 3. Cutting losses. If you can follow these three rules, you may have a chance - Ed Seykota.

The popular adage to success in the markets is: Cut your losses short, and let your winners run.

Though in theory it may seem simple, for most of those in the trading arena, it is rather difficult. And that's why we have that statistic that 95% (or was it 97%) of traders are net losers over the long term. So the question must be asked; Why?

William Gallacher, in his book "Winner take all" says its very easy to spot traders that lose, simply take a look at their open and closed trades. Most of the open trades are losses, some of them substantial, and the closed trades are always positive. Usually small in magnitude.

So from this it can be said that most traders do exactly opposite to what is required to be successful, they take quick, small profits, and never take losses.

Why do they trade this way? Because of their ego. Taking a loss is like admitting you were wrong. Some people, and I don't mean Average Joe's here, I mean professional money managers, think that a loss is not a loss until you sell.

That's why I think fundamental analysis is so dangerous. Because of the hours spent analysing the company and its prospects, you are more likely to hold onto a loser for longer than you should, rather than somebody who looked at a chart for a few seconds or bought the stock because it triggered an entry on the system.

I was having a conversation with my dad the other day and I asked him if he thought anybody has held PDN all the way, and he said probably not. Yet there are a few that I know personally that are still holding the script of dot.com stock, bought for a few dollars, and now if the company is not delisted then its worth a few cents at best.

Those that constantly lose either:

1. Don't have a well-tested robust trading system which allows them to cut their losses and let their winners run.

2. Have a trading system but do not have the discipline to stick to the rules or understand that every system inevitably will experience periods of drawdown.

So it seems it is human nature so trade EXACTLY opposite of what is required to win.

The only thing I can think of which is worse than letting your losses run is adding to losing positions. If you just think about what this will achieve, by the end of it, you will have many shares of a dog. If you pyramid as the share price goes up, there's a good chance what you'll end up with is many shares of an absolute champion.

The reason why profits seem to be taken quickly by those who lose is that they take the view that you better lock in a profit while its there. Problem with that thinking is that you will never have any big winners.

It's not hard to be in the 3-5% of those that win. As William Eckhardt said:
"Anyone with average intelligence can learn to trade. This is not rocket science".

I'll end this post with a few quotes from Michael Covel's "Trend Following".

Referring to $8.4million in WorldCom stock now only worth about $492,000: "Until you actually sell it, you haven't lost it" - Robert Leggett, Kentucky Retirement Systems.

"If you had Enron in your portfolio and didn't sell it at $90 or even at $10, don't feel embarrassed. On the surface it had always seemed to be a fairly good growth stock. We bought it all the way down" - Alfred Harrison, money manager at Alliance Capital Management Holding LP.

(The above 2 are professional money managers, these guys actually get paid to manage other people's money).

Enron stock was rated as "Can't miss" until it became clear that the company was in desperate trouble, at which point analysts lowered the rating to "sure thing". Only when Enron went completely under did a few bold analysts demote its stock to the lowest possible Wall Street analyst rating, "Hot Buy".

(Gotta love analysts).

Money Management

I have two basic rules about winning in trading as well as in life: 1. If you don't bet, you can't win. 2. If you lose all your chips, you can't bet - Larry Hite.

Money management is a very overlooked component of a trading system. This is unfortunate because, arguably, it is your money management that can have much more an impact on your results than entries or exits.

It's money management, often referred to as position sizing, which dictates how quickly you will compound your account when you are winning, and how slowly you will kill your account when you are losing.

Van Tharp, in his book Trade your way to financial freedom, tells an analogy of a test given to a number of Ph.D students (It was either 20 or 40). Each student was given $1000 to begin with and were told to play a game where they had a 60% chance of winning each time they played, and the size of each win and the size of each loss was the same. Obviously a positive expectancy system. There was no fee for playing and they could play as many times as they liked. None of the students made money except for two!

Why?
1. Either their position size was too large and/or
2. They followed the Martingale system.

Now, remember how I mentioned how trade frequency was very important in assessing the profitability of a system. A fair coin has a 50% chance of landing on heads and an equal chance of landing on tails when flipped. But if you flip the coin 10 times, there's more than a slim chance that you could get 8 or 9 heads consecutively. But flip it 100 times, and the results are more likely to be closer to 50%.

The Martingale system originates from gambling, and says that, you should double your betting size after you lose. Those that have even a basic understanding of probabilities know that the probability of a win is something that does not change irrespective of whether the previous bet was a win or a loss.

And note that once a trading system is in a period of drawdown, its very hard to even claw back to breakeven. If you lose 50%, you have to double your money, i.e., make 100% to just breakeven.

A common method of position sizing used by those that win are either the:
*Fixed percent risk, or
*Percent of total equity, models.

With these methods, when your account size (closed equity) increases, you position size increased, and when your account is in drawdown, your position size decreases. These methods, in essence, can be considered an anti-Martingale approaches. They work to decrease the risk of ruin.

Remember what Larry Hite said: If you lose all your chips, you can't bet.

To conclude, I just want to demonstrate the effect that money management can have on a system.

Systems 1 and 2 are exactly the same system. The only difference is in their money management (and their results!).

System 1 uses $10k equal dollar units.
System 2 uses 10% of total available capital.

Starting capital is $100k.
5000 Monte Carlo simulations were run.

System 1:
Average Number of Trades Executed: 346
Average Profit: $372,838.43 (372.84%)
Average Absolute Percent Drawdown: 9.4199%

System 2:
Average Number of Trades Executed: 152
Average Profit: $498,482.84 (498.48%)
Average Absolute Percent Drawdown: 8.2077%

Backtesting with TradeSim

A question I get asked often is that what is the point of backtesting when the future won't be the same as the past. A valid question.


There are 3 ways to know whether you are trading with a valid (profitable) strategy. A system that has a positive expectancy.

1. Backtest it.
2. Paper trade it.
3. Trade it in real time.


Do either of the 3 over a long enough time frame to get a statistically significant number of trades, with documentation of average win, average loss, and the win percentage, and expectancy can be easily calculated. As discussed previously, trade frequency is also important.


Okay, so let's go through the options. Paper trading is fine, though some say it is boring and doesn't test the emotions the same way that real trading does, but at least with this method, dollars are not risked. Still, it would take a long time to get a statistically significant number of trades. And very hard to be able to test the system over a variety of market conditions. Same problems with real time trading, but the main concern with this is, why would you trade a system and risk real dollars if you have no idea of the probabilities of whether your system will make a profit or not?

Backtesting, with correct validation techniques (i.e. walk forward analysis) can provide a very good indication is a system is likely to profitable when traded in real time.

The software I use for all my backtesting is TradeSim. It was initially designed to be a plugin for MetaStock (MS) but also be used with Amibroker and Bullcharts though the integration isnt as seamless as it with MS.

TradeSim is a trading simulator and backtester that is design to test a system exactly as if you were trading it. So trades will not be taken if there is insufficient capital, brokerage fees are factored in, there are different models of slippage to choose from, several position sizing strategies can be explored, and volume information can be also considered.

But the standout feature for me was monte carlo analysis capabilities. Because many trading systems do not have enough capital to be able to take every trade that is signalled, is it of paramount importance that the system be exhaustively tested through as many combinations and permutations of trades and portfolios that is possible.

David, the developer of TradeSim gives this example in his excellent User Manual. Take the heavily publicised MACD crossover system, where the entry is when the fast moving average crosses the slower one to the upside, and the exit when the fast moving average average crosses the slower one to the downside. If you only do a single portfolio simulation, it is possible that the results show that the system makes 22%pa. But if you run the simulation again, its a 2% gain. And then run it again its a 38% loss. Its only when you run it through 10,000 or 20,000+ simulations that you realise its a break-even system at best.

Another feature of TradeSim which is useful is the options for slippage. Slippage is the difference between the price you want and the price you actually get. Say for example, your system is set so that if you get a signal today, then you will buy tomorrow's open. If you set TradeSim to default order, then it will buy what the next day's opening price actually was. No slippage is taken into account here.

If you set TradeSim to market order, the program, through a random algorithm, with randomly choose a price between the day's hi price and lo price as the entry price. Slippage is the difference between the price you are getting and the price you wanted (the actual open). One of the other options for slippage is worst case, where when you buy, you pay the day's hi.

One way of testing the robustness of the method is to play with different slippage models. If a change from default order to market order, or market order to worst case, puts your very profitable system into the red, then you have some work to do.

With the system I'm currently developing, which is a weekly system, i was designing it with flexibility in mind. I may not be able to buy Monday's open all the time. So I set market orders during backtesting which gives me the flexibility to buy anytime during the week.

Trading with an Edge

To have some sort of probability to succeed in the financial markets, you have to have an edge.
An edge can also be referred to as a positive expectancy.

My aim and I'm sure this is the aim of all those that design and develop trading systems, is to develop a model which has a mathematical positive expectancy.

Having a system with a positive expectancy basically means that over the long run, over X number of trades and N number of years, the system will make money.

The formula for Expectancy is.
(Average win * Percent winners) - (Average loss * Percent losses).

This figure, if positive, tells you how much you will make, per trade.

Expectancy can also be calculated as a % per dollar risk.

Now "per trade" is the key part, and this is something that I only fully understood a few days ago.

I was testing two systems, both systems were exactly the same, and what I was trying to do was optimise the ATR multiplier.

When I was testing the system with a 2.5*ATR trailing exit, these were the relevant stats.

• Average win/Average loss = 5.4654
• Percent winners = 42%
• Average profit per trade = $1,285.02 (Expectancy)
• Annualised compound interest rate = 22.6859%

When testing with a 2*ATR trailing exit, these were the relevant stats.

• Average win/Average loss = 3.0486
• Percent winners = 49%
• Average profit per trade = $754.89 (Expectancy)
• Annualised compound interest rate = 31.5157%

Now, note how both systems have a positive expectancy. The first system has a greater expectancy than the second one YET the rate of return of the second system is much higher.

Why?

(Note that even though system 2 has more percent winners than system 1, this is taken into account when working out the average profit per trade)

The answer to the WHY lies in what i was highlighting before.

Trade frequency.

System 1 in fact made only 109 trades in the testing period (6 years).

System 2 made 247 trades.

In summary, whilst its crucial to trade a system with a positive expectancy, its also important to keep in mind that trade frequency is what determines how often you can exploit that edge.

Its useless having a system that picks 80% winners and has an average win/average loss of 12 if it only trades 8 times a year.

Van Tharp's Trade Your Way to Financial Freedom is one of the best texts I've read on the topic of expectancy.

System Development

Firstly, just a statement to clarify the intention of this blog.
Anything mentioned on this blog cannot be interpreted as financial advice.
Any securities mention therein is not a recommendation to buy or sell.
I am not licensed to give financial advice.

This blog is simply a journal and the documentation of my journey of systems development and mechanical trading.
When I do begin trading live in real-time, I will post which securities I buy and sell for the purpose of having an online track record by which my results can be verified.

Now lets get onto system development.

There are 5 stages of systems development.

1. Ideas.

Every great system begins as an idea.

2. Design.

The idea is then put into a form (code) which can be used for backtesting.

3. Testing.

Testing is done over a certain timeframe in the past. How far back to backtest is a commonly asked question for which there is no hard and fast answer. Firstly it depends on how long-term your system is. If you are trading intraday (ie. Daytrading), using 15-minute bars, then every trading day (assuming ASX stocks) contains 24 bars of data, each bar with details of the open price, close price, high, low, and volume. So perhaps 6-12 months of data is enough for backtesting. For weekly system, you are looking at about 5-10 years.

While its important to use enough data so your system can take on enough trades to be considered statistically significant, its important not to go too far back as market dynamics are more likely to have changed the further back you go.

Data must be spilt up into in-sample (which is the data on which the testing and optimisation is carried out on) and out-of-sample data.

Is it best to leave aside the most recent data as out-of-sample data for the walk-forward analysis.

4. Optimisation.

This is the stage where each parameter in the system rules are optimised. AmiBroker can do this very easily by creating those pretty 3D charts.

I use MetaStock coupled with TradeSim so the process is a little more laborious.

An example of optimisation is, say that your system uses the cross of the long-term moving average as an exit. At this stage you don't know which length moving average is best to use. So you try 150 periods, 200 periods and 250 periods, and in each test, you keep every other parameter the same.

When you find which parameter works best, that means you have optimised for that parameter.

One danger of optimisation is over-optimisation, also referred to as curve-fitting. This usually happens when you attempt to get perfect results by adding in too many parameters or conditions, to the system.

Walk-forward analysis will indicate whether indeed the system is robust or whether it has been curve-fitted to the in-sample data set.

5. Walk-forward analysis.

Up until this stage, we have no idea whether or not a system will be profitable going into the future. We just know that the system would have been profitable over the timeframe over which it has been tested and optimised.

The out-of-sample data, on which the walk-forward analysis will be performed, is brand new data that the system has not yet seen before. This should proceed immediately after where the in-sample period ends.

For example, you test and optimise on data from 1998-2002 (5 years) and then you leave 2003-2007 (almost 4 years) for the walk-forward analysis.

The results from the walk-forward analysis are very useful in determining the future profitability of the system. Because its exactly the sample as making the last 5 years as an in-sample data set, and using tomorrow and beyond (the next 5 years) as an out-of-sample data set. Tomorrow and beyond is exactly like the out-of-sample data, brand new data that the system has not seen before.

The only difference is that when you start trading tomorrow based only upon results of in-sample data tests, you are trading a system and risking real dollars with a system that has not been validated.

Walk-forward analysis is what validates a trading system.

Howard Bandy explains this nicely in his book "Quantitative trading systems" (2007).