We have discussed Walk Forward analysis and its importance in the past. The first part of this tutorial series introduced the concept. The second part demonstrated how to use Walk Forward in TradersStudio. Finally, the third part described how to use Walk Forward on a portfolio. One of the biggest issues with system development is that many systems might test well using historical results, but they do not perform well in the future. There are several reasons for this. The most common one is that the system is not based on a valid premise. Other reasons include:
- Lack of robustness in the system due to improper parameters. A system is considered robust if it runs well in any market condition.
- Inconsistent rules and improper testing of the system using “out-of-sample” and “in-sample” data.
As described through our other tutorials on this site, Walk Forward Analysis does optimizations on a training set, then tests on a period after the set and keeps rolling it forward repeatedly. Thus we have multiple out-of-sample periods and we can look at these results combined. Originally discussed by Robert Pardo, this type of analysis can keep a trading system model a step ahead. The reason is that you are not over-fitting the training set but rather you are fitting a portion of it, and then testing it on the out-of-sample data to ensure that it performs well outside of the training set.
TradersStudio’s walk forward tester was designed by Murray Ruggiero with the help of Robert Pardo and the TradersStudio walk forward tester is what Pardo uses for his own research. Indeed, our walk forward analysis has a distinctive Pardo influence. For example, other testers run all of the optimizations and then slice and dice them up to do the tests. However, TradersStudio runs each window as it walks forward. The reason our software does this is for robustness. Consider a simple example: suppose we are optimizing a channel breakout system over the past 10 years. We have a 1000 bar (about four years) training window and a 250 bar (about one year) out-of-sample window. The way most walk forward optimizers work is they optimize over the entire full 10 years, and then they “slice” the optimized period into the windows. So suppose your first window ends December 15, 2009 and you made a buy trade on December 10, 2009. However, your system does not close out this trade until December 21, 2009. Having optimized over the entire timespan the only thing most walk forward optimizers can do is pretend like you have gone short on December 21 because you would have sold at that time.
This violates the basic premise and theory behind walk forward optimization however. The idea is that we actually walk forward, that is we optimize over 1000 bars, then test over 250, then optimize over the next 1000, and test over the next 250. In the above example, we want to see how, if we had started trading our system the day after December 15, 2009 it would have performed. Maybe we would not have gotten a sell signal on December 21. Maybe we would have gotten a buy signal on the 17th and continued from there. This is the type of walk forward testing that TradersStudio does. We optimize over the 1000 bar period, then we pretend like the day after that period is day 0 and begin trading from there. This is the whole reason that Robert Pardo uses TradersStudio for his research. The whole premise of walk forward is that we want to pretend like we were really trading in the past. That is, we want to pretend like we did our backtesting results and just started trading in the out-of-sample window. To optimize over the entire time and then slice into different time windows not only violates the pretense of walk forward but it also makes for incorrect and unreliable results!
To this effect, a system could look good using the method most walk forward methods use, but the exact same system could fail using the TradersStudio way. In fact, trying a system with different out-of-sample window sizes is the best stress test available – better than any other sampling, Monte Carlo, or other statistical methods. You don’t get this with the way most platforms slice up the time continuum because you don’t get to see the random edge effects on the robustness of your system. Our method increases the time necessary to do small out-of-sample window optimization but it gives much more realistic results and adheres to the true ideals behind walk forward analysis.
The other problem with other walk forward optimizers is that they do not generate active orders. Historically, even with TradersStudio you would need to re-run the whole walk forward system to generate active orders for the next day. As well, with a little coding in TradersStudio, you can use a walk forward system in a tradeplan. This was a big advance over other walk forward testing tools. The way everyone else used walk forward was to run it and then use the last set of parameters and run the system for the last windows. In addition, TradersStudio can do walk forward analysis on a portfolio. The only one other product that can do that without additional programs is Trading Blox, and it retails at $2995 – a price significantly higher than TradersStudio!
The problem with not being able to seamlessly and intuitively use walk forward analysis in real trading bothered me and I think it is one of the main reason this analysis is not used by more traders. The newest upgrade to TradersStudio Turbo has solved this issue. I’m proud to introduce Live Walk Forward™. Live Walk Forward allows you to develop your walk forward model and then save it so it can be used as a stand-alone system or as part of a tradeplan. In addition, this model will also re-optimize automatically when needed upon executing.
Using the new Live Walk Forward feature is very simple. We will start with the normal walk forward analysis. In keeping with the spirit of our examples earlier, we will optimize our hypothetical system using 1000 bars – about 4 years and trade for 1 year.
Using 1000 for the optimization and trading for 250 bars gives us a first trade of 11/18/1991. Above you can see all the training/run windows walking through the data set. We also see the optimal parameters using NetProfit/Drawdown as the search criteria. You can see that the parameters are very stable over time. They stay the same for 2-3 windows and only shifting slightly over 4-5 windows.
We will now select another walk forward mode. This is another unique feature of TradersStudio. We support three different modes. Exit all trades (the one we were using before) waits for a new signal in the new window. There is also exit if the direction changes, and exit and re-enter. These three modes work different and will give different results. Depending on the system, sometimes different modes are better. We will select “exit if direction changes”. This is often the best when the parameters are stable. After that we will select the run button again.
This will run the system and show the results based on our walk forward. This is the run that was built into the walk optimizer before. That is, this is simply showing how to run our original walk forward analysis within TradersStudio.
Looking at the trade by trade you see the first trade out of sample trade was 11/18/1991.
Now select the Live Walk Forward button and you will get a save as box which is loaded with a default name. We can change the name to whatever you want.
We are going to change the name to WFRT_IntermarketUS_UTYFinal and press OK. This will save our analysis as a “LIVEWALKFORWARD” session.
When you press ok you will get a box asking you if you want to run the session.
Let’s now press yes and run it.
You can see the run box for our Live Walk Forward session. Notice how there are no arguments. This is due to the fact that the parameters are coded in the walk forward session and every 250 bars, this will roll to the newest 250 bars forward and then use the last 1000 as training data to optimize the new window. It will then use that for the next 250 days as real-time trading.
From the results above you can see that this system did really well making over $185k even after losing the first 1000 days. Now, let’s see how a normal in-sample optimization would have performed if we remove the same 1000 starting days.
Our walk forward results produce over 75% of the results of a pure in-sample walk forward in this case. This shows how robust this system is. We did not even optimize the training and run window sizes. We just started our experiments with about four years in-sample and about one year out-of-sample.
Let’s look at the coolest feature of Live Walk Forward™ – active orders!
When we run the session going backwards in time you will see the correct results and active order history being generated. In addition, and more importantly, as we go forward, we will run the system with the new date as the walk forward period and then when we hit a new boundary we will run a new training and run window and update the Live Walk Forward Result tab with the new window and parameters it will also generate the new results including new active orders.
Next, besides being able to run a live walk forward session as a normal session, we can also use it as part of a tradeplan. We used my DynamicMarginPlanSimple script using parameters 40, 100, 40, 7, 19911119 and ran this session in a tradeplan.
What’s really cool about this is that you can see that the positions are sized. Also, the active orders are also sized correctly. You can now mix the session with other walk forward sessions or even other sessions not using walk forward.