Many traders develop trading systems using a process of testing and optimizing different sets of trading rules and parameters on the same data. Systems developed using this repetitive process tend to fall apart quickly when they are traded moving forward in real-time if the developer is not careful. The primary problem with this type of system development is what mathematicians call ‘too few degrees of freedom.’ In trading parlance, it is known as ‘curve fitting.’ Walk forward testing aims to solve this issue.
In testing, whenever multiple strategies or sets of parameters are tested on the same set of data, the ‘degrees of freedom’ of the results are reduced. This process of using the same historical data results in improved performance, but serves to reduce the chance that the strategy or strategies will be profitable in the future.
How can a trading system get developed while maintaining a reasonable ‘degree of freedom?’ The simplest way to improve results comes from using ‘out-of-sample testing’ where the data series is divided into two segments covering different time-periods. The system(s) are developed using the first or ‘in-sample’ time-period and then the system(s) is tested to see how it performs on the second or ‘out-of-sample’ period.
In effect, one test is performed on the ‘out-of-sample’ data no matter how many ideas are tried on the ‘in-sample’ data. The ‘degrees of freedom’ are preserved and the ‘out-of-sample’ performance provides a much better indication of real-time system performance. Of course, every time the testing process is repeated with a different system, ‘degrees of freedom’ are lost. If enough testing is done using the data, the ‘out-of-sample’ data becomes part of the ‘in-sample’ data and the advantage of this method of testing is lost.
Walk Forward Analysis carries the idea of ‘out-of-sample’ testing to the next level. Think of it as ‘out-of-sample’ testing on steroids.Walk Forward Analysis works like this. Let’s say that you have twelve years of data extending from 1995 to 2006 for the markets that you want to trade. Let’s also assume that your trading strategy needs a minimum of three years of data for testing and optimization.
To begin, start by developing and optimizing the system using only the first three years of data – in this example, 1995 – 1997. On these three years of data, try as many ideas as you like and optimize parameters in as many ways as you can think of. It is important not to look at any data after 1997! When you think you have found the ‘Holy Grail’ of trading systems, record the rules for the system with the optimum parameters. These rules and optimized parameters are to be used later for the final testing with new data starting with 1998.
Slide the three-year time window of data forward a little – say one month. Now, the data that you are working with runs from the 2nd month of 1995 to the 2nd month of 1998. Repeat the analysis, including optimization and record the rules and optimized parameters. In the final pass, these parameters will be used for the 2nd month of 1997.
Continue with ‘walking forward’ and optimizing the three-year data periods. Record the results for use in the first month following the three-year optimization period. When your data finally runs out in 2010, go back, and test the system for the entire period from 1998 to 2010. Switch the rules and parameters each month to use the ones that you found and recorded. In effect, you are performing a new out-of-sample test for each month. The system performance for these thirteen out-of-sample years (156 out-of-sample months) is a much better indication of how a system will perform in real time than the performance of any single time period used for optimization.
There is nothing magic about the assumed time-periods – three years for system development and one month for the walk-forward interval. Picking these two time parameters is a trade-off between optimization time and statistical validity of the results. In practice, I have found that using about 20% of the optimization period for the walk-forward window works fairly well.
If the results for the ‘out-of-sample’ months look good, continue the walk-forward process in real-time to find the parameters to use with real money. Another advantage to this method of system development and trading is that your system will better adapt to changes in market behavior over time. Markets do change with time – we have all seen systems that have made money for several years and then simply stopped working because the markets changed.
In the next section of this series, we will take a look at how walk forward testing can be done within TradersStudio using an example that mimics a real-world scenario.