Comments on Milk Trader: Evaluating the Fitness of a Fitness Function

I opted for the phrase 'fitness' function ...

2009-12-07T06:18:04.100-08:00

I opted for the phrase 'fitness' function because I just bought the Perfect Pushup and had 'fitness' on the brain.

Good point on selecting the fitness function that yields the best results, even on out-of-sample testing. I'm not advocating cherry-picking a fitness function based on results, but rather am highlighting the fact that there are a wide range of metrics one can use to select a discriminating function.

It's a free country, and fitness centers aboun...

2009-12-04T20:19:48.889-08:00

It's a free country, and fitness centers abound, let alone fitness functions (why the ugly name? Common mortals call them objective functions, or just metrics). But any function that takes min and max or a random quantity has usually very bad statistical properties (e.g., see quantile estimators in statistics). Different fitness functions will have different values on identical data (obvious). The issue is on which functions make economic sense and have good statistical properties in a backtest.

Last remark. Fitting strategies on a training set, and selecting the best based on out-of-sample test is just another form of data mining and will result in overfitting. Backtesting is not very complicated to perform, but is actually hard to do right.

Milk - you might have seen it but I did put the re...

2009-11-10T08:34:07.436-08:00

Milk - you might have seen it but I did put the results on the blog:
http://bit.ly/3SPEV4

This was for a simple e-ratio calc going through 50 steps and there is no optimisation logic there as they all need to be run.

AmiBroker was 25x faster!

I'm looking forward to the results. Please lis...

2009-11-09T14:03:51.047-08:00

I'm looking forward to the results. Please list the number of permutations that are being tested. Once you get into four parameters, the number of permutations can be astounding for an optimization run, so brute force is typically abandoned in favor of more complex search methods such as genetic algorithms, particle swarm and simulated anealing.

I am going to run a speed comparison test (hopeful...

2009-11-09T07:22:50.373-08:00

I am going to run a speed comparison test (hopefully tonight) and post the results on the blog. Obviously I will try to get the results as close to each other in each system/platform (with same data and system parameters) - completely agree that the test is only relevant if trades and calculations are the same (or close enough).

From reading the Amibroker user manuals I got the impression that Amibroker was built from the start to optimise speed - and I can believe that it could have a (big) impact on performance compared to a platform built with only functionality in mind. In my "real job" I have cut down running times of some batches by a factor 10 to 100 just by a re-write with performance as the main objective (ie bulk operations vs. individual loops, data caching, etc.). There are usually lots of places code can be optimised for performance if you put your mind to it.

The downside - for Amibroker - is that it might have its limitations in terms of functionality available (ie I get a felling TradersStudio with its dedicated Trade Plan functionality allows for more flexible money management/position sizing/portfolio allocation).

So it might be that Amibroker as a quick and fast prototyping tool combined with TradersStudio as a more "in-depth" full strategy testing is a great combo. This is definitely an impression I got from reading some other user comments (on elitetrader, etc.).

The walk forward didn't take long at all, abou...

2009-11-09T06:16:20.405-08:00

The walk forward didn't take long at all, about 20 minutes.

I'd be interested in knowing how Amibroker is so much faster than TradersStudio. Are the results the same from each program?

This sketch of fitness function comparison is by no mean conclusive, and I'd like to see how the comparison plays out with multiple markets.

Hi Milk, I see you've made great inroads in R...

2009-11-09T02:02:28.546-08:00

Hi Milk,

I see you've made great inroads in R - well done. Great charts and interesting analysis results. It seems to possibly validate what you inferred (ie that Profit/DD is the best measure). Good luck on using these brain cells to find the best fitness function! How long did the walk-forward tests take to run in TradersStudio by the way?

I was actually planning a post on the bliss function myself (following our discussion on the walk-forward post comments) to clarify my thoughts and expand on it (albeit more from a measuring aspect only rather than predictability also).

PS: I managed to get up early this morning to work on the e-ratio code in Amibroker - looks much faster than TraderStudio (ie. mins vs hours). Felt good to manage and squeeze a bit of work at the weekend..

-Jez