Milktrader
Milktrader
Tuesday, November 17, 2009
Sunday, November 8, 2009
Evaluating the Fitness of a Fitness Function
The walk-forward process of system development is the final test of a system before real capital gets allocated. It validates the system on out-sample data, or data that hasn't been peeked at during development. It's not very complicated, really. You optimize your system on a range of data and then choose the best parameter set to trade with in the out-sample period. Then you observe the results and determine if the system warrants any capital investment. As with most things in system trading, you need to make a decision on how you approach this idea of picking the best parameter set. Who or what decides what is best? The arbiter is known as a fitness function, because it determines which parameter set is best suited for future trading. Not all fitness functions are created equal though. Let's look at three fitness functions and see how they offer different results.
I'm using data derived from a walk-forward of the White Bumblebee system and the DX future contract (US Dollar). The out-sample data is over 10 years in length, and the total number of walk-forward periods comes to 10, as I've optimized on 400 days and traded on 200 days. The different fitness functions include Net Profit, Net Profit/Drawdown, and Pessimistic Return on Margin (PROM). For Net Profit, the system returns the parameter set that had the highest Net Profit during the optimization period and then uses those settings for the trading window. Same logic for the other fitness functions.
I've broken down the analysis into two sections: Profit Metrics, and Drawdown Metrics. In each category, I'm interested in the total, range, max, min, mean and standard deviation. My backtesting software is TradersStudio and the statistical software that paints the graphs is R.
First let's look at the Profit Metrics.
In the end, all three produced about the same total profit over 10 years. Net Profit registered the largest profit during a single period, but also the highest range. All three fitness functions had losing periods, but it is interesting to note that the Net Profit/Drawdown function had the least worst period, or it lost small when it lost.
Next, let's look at the Drawdown Metrics.
Here again, the Net Profit/DD fitness function yields the most attractive results. It experienced the least drawdown in a period and also the least amount of total drawdown during the 10-year testing period.
Keep in mind that the sample size is fairly small here with only 10 walk-forward periods being analyzed. You could easily argue that these results are not statistically valid, but at least it points out that all fitness functions are not created equal. Finding the best one may require some time and brain cells on the system trader's part, but hopefully well-spent.
I'm using data derived from a walk-forward of the White Bumblebee system and the DX future contract (US Dollar). The out-sample data is over 10 years in length, and the total number of walk-forward periods comes to 10, as I've optimized on 400 days and traded on 200 days. The different fitness functions include Net Profit, Net Profit/Drawdown, and Pessimistic Return on Margin (PROM). For Net Profit, the system returns the parameter set that had the highest Net Profit during the optimization period and then uses those settings for the trading window. Same logic for the other fitness functions.
I've broken down the analysis into two sections: Profit Metrics, and Drawdown Metrics. In each category, I'm interested in the total, range, max, min, mean and standard deviation. My backtesting software is TradersStudio and the statistical software that paints the graphs is R.
First let's look at the Profit Metrics.
In the end, all three produced about the same total profit over 10 years. Net Profit registered the largest profit during a single period, but also the highest range. All three fitness functions had losing periods, but it is interesting to note that the Net Profit/Drawdown function had the least worst period, or it lost small when it lost.
Next, let's look at the Drawdown Metrics.
Here again, the Net Profit/DD fitness function yields the most attractive results. It experienced the least drawdown in a period and also the least amount of total drawdown during the 10-year testing period.
Keep in mind that the sample size is fairly small here with only 10 walk-forward periods being analyzed. You could easily argue that these results are not statistically valid, but at least it points out that all fitness functions are not created equal. Finding the best one may require some time and brain cells on the system trader's part, but hopefully well-spent.
Thursday, November 5, 2009
A Short List of Candidates
You can't trade everything. Well, at least you can't trade everything well. At some point, you should limit the universe of things you trade. And it's okay. You don't need to trade everything because quite frankly, nobody knows what you're talking about anyway at the weekend cocktail party, and they don't care. I implore you to be prudent in this way: commit to trading a specific world of underlyings, and quit chasing bright lights in the depths of the ocean, like everyone's favorite Nemo character, Dory the Blue Tang.
It can be anything such as large cap stocks, penny stocks, major currency pairs, or just one stock or currency pair. Or just the ES futures. Or you can make the case of trading what is cool and in vogue, such as GOOG into earnings, or gold stocks when everyone is buzzing about gold. I've chosen to trade the universe of liquid futures. And I've wittled down the list of qualifying candidates to just fifty-two.
My list is crafted carefully around major market groups, which have a tendency to inter-relate and correlate. The basic groups include:
1. Currencies (7)
2. Interest Rates (10)
3. Equities (9)
4. Metals (5)
5. Energy (4)
6. Commodities (17)
Currencies:
This group includes the Dollar, Euro, Pound, Yen, Aussie, Swiss Franc and Canadian Dollar. Pretty much the major currencies with the Aussie and Canadian related to commodities.
Interest Rates:
This groups consists of US Treasuries (2-year, 5-year, 10-year, 30-year), the Eurodollar and foreign treasuries that include the Canadian 10-year, the Gilt Long Bond (British debt), and the German debt complex of Eurobund (10-year), Euro BOBL (5-year) and Euro Shatz (2-year). Potential inter-market relationships between these markets and both the currency and equity markets abound.
Equities:
Okay, the standard US equity markets need to be included. I've chosen to leave out the DOW futures and focus on the S&P 500, Nasdaq 100 and the Russell 2000. For international exposure I've included the Nikkei (Japan), CAC 40 (France), Euro STOXX 50 (Europe), FTSE 100 (Britain), DAX 30(Germany) and the Heng Seng (Hong Kong).
Metals:
You can't leave out gold or silver. Their ratio is actually an important metric. I've also included other precious metals such as platinum and palladium (think catalytic converters and cell phones), and the base metal of copper.
Energy:
Of course we include crude oil and natural gas. Also included are refined products such as heating oil and gasoline.
Commodities:
This is the largest group, so let's divide it up into Grains, Livestock, Forest and Softs (I'm following the CME Group convention here).
Grains include corn, wheat, soybean, soybean oil, soybean meal, oats and rough rice.
Livestock includes live cattle, lean hogs, feeder cattle and everyone's favorite future - pork bellies.
Forest includes lumber.
Softs include cocoa, coffee, cotton, sugar and orange juice.
And that's it.
Just 52 underlyings, many of which have interesting inter-market relationships.
This is my final roster of players, and they will all participate in future backtesting regimes. Will I trade them all? No. Money management may limit the total number of underlyings to a small percentage of the universe, but those 5-7 futures will represent a good diversification of product, and will also be the best representative for a particular trading strategy.
Dory got confused once in Finding Nemo and said: 'I don't know where I am... I don't know what's going on. I think I lost somebody but I, I can't remember.' Don't be that fish.
It can be anything such as large cap stocks, penny stocks, major currency pairs, or just one stock or currency pair. Or just the ES futures. Or you can make the case of trading what is cool and in vogue, such as GOOG into earnings, or gold stocks when everyone is buzzing about gold. I've chosen to trade the universe of liquid futures. And I've wittled down the list of qualifying candidates to just fifty-two.
My list is crafted carefully around major market groups, which have a tendency to inter-relate and correlate. The basic groups include:
1. Currencies (7)
2. Interest Rates (10)
3. Equities (9)
4. Metals (5)
5. Energy (4)
6. Commodities (17)
Currencies:
This group includes the Dollar, Euro, Pound, Yen, Aussie, Swiss Franc and Canadian Dollar. Pretty much the major currencies with the Aussie and Canadian related to commodities.
Interest Rates:
This groups consists of US Treasuries (2-year, 5-year, 10-year, 30-year), the Eurodollar and foreign treasuries that include the Canadian 10-year, the Gilt Long Bond (British debt), and the German debt complex of Eurobund (10-year), Euro BOBL (5-year) and Euro Shatz (2-year). Potential inter-market relationships between these markets and both the currency and equity markets abound.
Equities:
Okay, the standard US equity markets need to be included. I've chosen to leave out the DOW futures and focus on the S&P 500, Nasdaq 100 and the Russell 2000. For international exposure I've included the Nikkei (Japan), CAC 40 (France), Euro STOXX 50 (Europe), FTSE 100 (Britain), DAX 30(Germany) and the Heng Seng (Hong Kong).
Metals:
You can't leave out gold or silver. Their ratio is actually an important metric. I've also included other precious metals such as platinum and palladium (think catalytic converters and cell phones), and the base metal of copper.
Energy:
Of course we include crude oil and natural gas. Also included are refined products such as heating oil and gasoline.
Commodities:
This is the largest group, so let's divide it up into Grains, Livestock, Forest and Softs (I'm following the CME Group convention here).
Grains include corn, wheat, soybean, soybean oil, soybean meal, oats and rough rice.
Livestock includes live cattle, lean hogs, feeder cattle and everyone's favorite future - pork bellies.
Forest includes lumber.
Softs include cocoa, coffee, cotton, sugar and orange juice.
And that's it.
Just 52 underlyings, many of which have interesting inter-market relationships.
This is my final roster of players, and they will all participate in future backtesting regimes. Will I trade them all? No. Money management may limit the total number of underlyings to a small percentage of the universe, but those 5-7 futures will represent a good diversification of product, and will also be the best representative for a particular trading strategy.
Dory got confused once in Finding Nemo and said: 'I don't know where I am... I don't know what's going on. I think I lost somebody but I, I can't remember.' Don't be that fish.
Subscribe to:
Posts (Atom)