Walk forward testing within TradersStudio is an extremely valuable tool, as evidenced by the first part of this series. The concept for walk-forward testing is similar to using ‘in-sample’ and ‘out-of-sample’ testing periods. Instead of optimizing on twenty years of data and using the last four years of data for testing, the optimization is done across ten years and the system is tested on the eleventh. Once this test is completed, move the whole time window forward one year and run the test run on the next year. Find the optimum set of parameters for each of the 10-year windows and use that set of parameters to trade for the next year. Move the time window forward one year and run the test on the next year until all of the years in the data series have been tested.
When the system performance is evaluated, all of the one-year windows are consolidated to compose the out-of-sample periods for each of the optimal windows. The out-of-sample performance is used to judge how good the system is. A major problem with doing this type of testing is that there is currently no software available that does walk-forward optimization.
In this example, we will use the Custom Report capabilities of TradersStudio® to develop a walk-forward simulation of a simple trend-following system. Walk-forward testing is a hot area of research and there are many issues to address.
An example would be what if the optimal parameter value changes as the testing moves to a different time window, how do we change the parameter used in the ‘out-of-sample’ walk-forward period? If a parameter value changes from ‘20’ to ‘22,’ it is not much of a problem but if there are two peaks with one at ‘20’ and the other at ‘50’ and the optimal value changes from ‘20’ to ‘50’ on a single walk-forward step, what value do we use?
This is a problem called ‘transitional’ or ‘boundary trades’ in walk-forward analysis. Let’s look at the following example.
In this example, the optimal parameter is different for the two periods shown in the screen shot. It changes from 25 to 30, which sets up the possibility that the trades are different between the two ‘out-of-sample’ windows. This sets up a ‘transitional trade’ that is handled by different rules. We can exit the open trade at the end of the ‘out-of-sample’ period and enter the open trade that bridges both the next optimization period and the next run period. In this example, the trade from October 15, 1996 (2nd table) to April 18, 1997 (bottom table) bridges the training and run periods, which is why it is a ‘transitional trade.’ It has three options.
If we wanted to “exit all trades” (in above figure), we would exit the trade for the old run period on October 15, 1996 and wait until the first new entry to buy on April 18, 1997.
In our testing setup, if we choose to “exit if direction changed” (in below figure) meaning to exit trades only when the direction changes, we would create a trade that was entered on July 18, 1996, old run period and exited on February 7, 1997, which is in the new run period.
Finally, the last option is to “exit and re-enter” meaning to ‘rollover’ (below figure). This method exits the trade from the run period and enters in the direction (the same or different) that the system is in on the first day of the new run period.
These are a few simple examples attempting to explain this important walk-forward concept. Now that you have a basic understanding how this works, the TradersStudio walk-forward optimizer will help you take care of it.
Walk-Forward Testing in TradersStudio
Walk-forward analysis is very powerful, but it has never been built into a product until it was included in TradersStudio. TradersStudio version 1.3.6 had a simple walk-forward analysis program that was implemented as a ‘global macro.’ Now, TradersStudio® Professional has a fully integrated walk-forward analysis program.
In the next section, we will run a walk-forward test using a channel breakout approach on one market and a basket of markets to illustrate how this technology works. To start, create a New Session from the TradersStudio Main Menu or from the opening screen.
In this walk-forward testing example, create a commodities session called ‘WalkForwardSimple’ and use the Channel Breakout system called ‘ChanBreakOut.’ Click on Next to continue.
Run the test on a contract lot size of ‘1’ with $75 deducted for both slippage and commission. Click on Next to continue.
Click on Next to continue to the next screen because no changes need to be made in the Time Frame screen.
In the Session Data screen, select Natural Gas from the futures directory as the market that is added to the session. Run the session using following parameters.
The results for the system test are as follows:
After the Session has run, we will need to do simple optimization.
Here are the results for the optimization of the system:
Next, do walk-forward optimization. The Walk-Forward feature is an icon located on the toolbar. When you click on the Walk-Forward icon, the Walk-Forward Optimize box appears on the screen.
This box is like a super-optimizer. It includes a length entry for each optimization window plus the length for the out-of-sample run window. It also includes variable search criteria for isolating the best parameter set.
In this Walk-Forward test, we are going to optimize from ‘5’ periods to ‘20’ periods in incremental steps of ‘5.’ Please note that walk-forward optimization only works on the overlapping date range, which is all we have since this is a single market. In this test, we are choosing to optimize using 1000 days or roughly 4 years of data. Then, we will trade the market for 200 days while dropping off the first 200 days of data and rolling the window forward. This is repeated until the last day of data if reached. The walk-forward optimizer can also be used to generate orders for tomorrow’s trading. This tool can be used for much more than backtesting!
A dropdown box lists the choices for the criteria for the optimization. In this example case, we will select the set of parameters that produced the Best Net Profit on the optimization window. NetProfit/Drawdown Ratio and Custom Optimization Factors can also be used as search criteria for optimization. Custom Optimization Factors require that you set the optimization factor in your trading system code. The following code example shows how this is done.
From this example, we have illustrated that it is possible to optimize the best system walking forward by using ‘Optimal f’ as the search method. ‘Optimal f’ is normally used as a money management methodology, but it inherently incorporates the trade distribution of the system and gauges system performance. A higher ‘Optimal f’ value means that the system is performing better. Using ‘Optimal f’ as an optimization search method illustrates that it is possible to create complex measures of performance and use them in a systematic walk-forward testing approach.
The Walk-Forward test shows many useful reports. These reports include Walk- Forward Periods, Raw Trades, Clean Trades, Clean Trades by Period and Active Orders.
The Walk-Forward Periods Report shows the start and end dates for the trading and the run out-of-sample periods. It also shows the Net Profit, Percent Winning Trades, and Drawdown for each time-period. The Criterion column contains the value for the best set of parameters, which is in the Optimal Parameters column.
The Raw Trades Report shows all trades for the trading period and the run period for each window.
The Clean Trades Report shows only the out-of-sample trades with the boundary points between resolved trades. Following the screen shot of the Clean Trade Report, we will investigate the dropdown options in the Clean Trade Report.
When navigating from window to window, the optimal parameters for each window can be different. In this case, it is possible that a system can switch market directions (long to short or vice versa) between optimization windows. Another common problem is when the trade direction does not change, but the trade entry and/or exit dates change with overlap. To account for these issues, the trades need to be consistently fixed up at the endpoints.
There is more than one way to handle these issues. First, TradersStudio can be set to always exit trades at the end of a run window and wait until the next new signal to re-enter. Another method is to exit if the market direction changes. Finally, trades can be exited at the end of a window and re-entered without a new signal in the direction of the new run window.
In our work through example, we will select ‘exit all trades.’ The set of rules that is used to exit can drastically affect system performance. For example, ‘exit all trades’ makes 9.5% to 11% more profit in this case than the other two options. You can check the different results by selecting different modes without re-running the Walk-Forward Optimizer. Click the Run button, next to the drop down to get the “Summary Report.”
The next window shows Clean Trades by Period, labeling each walk-forward period and all transitional trades. This window tells what parameters were used in each walk-forward session and how the boundary trades were handled.
The final window is the Active Order Report, which is just like the Active Order Report in a normal session run. These are the orders to be placed for tomorrow’s trades.
Now, let’s return to the Clean Trades Report window. You will notice a Run button at the top of the screen. This feature allows you to run the complete system as a normal session. Note that this feature exports the trade history TradersStudio has run as a system to recreate the results. On some occasions, rounding issues may cause slight differences in results.
Since, this feature uses our Export Trade-by-Trade technology, it produces a Text file that TradersStudio can use to recreate a full normal system run. With a little macro programming, this feature allows a Trade Plan to be applied to our walk-forward analysis.
Click on the Run button on the Clean Trade Report, it gives you a full set of reports like any other session and the included charts like ‘Start Trade Drawdown’ and ‘Underwater Equity Curve.’
If you navigate to the Session File menu for the walk-forward testing session, you will find the option to Export Walk-Forward Trades.
If you select the option to Export Walk-Forward Trades, you will get a Windows ‘Save As’ box to save the file under a selected filename. TradersStudio® Basic™ has a Trade Import function making it possible to hardcode the path of the special export trade file that was created and run this on the same session with the same markets.
This allows us to duplicate the session results. This method can be used to include the sessions in a trade plan so money management strategies can be applied to a walk-forward system.
In the next section of this series we will show how we can do the same analysis on a portfolio!