Matching and fitting models - a perfect solution?

Following my ill-informed rant about data fitting a couple of weeks ago, I had a short exchange on Facebook that pointed me in the direction of a couple of possible solutions: Akaike information criteria (AIC) and Goedel machines.

I'm still not satisfied, although I don't think I have fully got to the bottom of the significance and power of a Goedel machine - in my spare time I am still trying to swot up on what they can do. However, these ideas got me thinking and (while in the shower, of course) came up with the following proposal:

Mix algorithms, using each when it becomes better than the others. I propose three (basic versions of) algorithms: a genetic algorithm, Markov chain Monte Carlo (MCMC) and conjugate gradient fitting. Here's a three step method for a general fit. As a reminder, we are trying to optimize the parameters of a model to minimise least squared difference (or, even better, maximize a likelihood function):

- Start off with a genetic algorithm. This is clearly better than the others when nothing is known about the parameters. It could be thought of as a kind of multi-strand MCMC with arbitrary large step size coupled with choromosome splicing which can only serve to improve the answer.

- Once parameters are known to within a certain range with e.g. 95% confidence, switch to MCMC. This avoids non-optimal sampling of random parts of the parameter space while still maintaining the ability to cope with a rough high-gradient parameter space.

- By some kind of parameter space sampling determine when it seems like there are no more local minima (ok, that's a bit wooly but I think it could be done) and gradients are 'small', then switch to conjugate gradient. Unlike the other algorithms this will give a final answer in a finite time.

If you have more information you could skip any of the steps. A danger is, of course, that checking the switching criteria could take up more effort than it is worth!

This is really a thought experiment and I don't have the time to implement such an algorithm just at the moment. I think what it shows, though, is that there is a clear case to be made as to where each algorithm is optimal. Perhaps before attempting a fit you could run a quick check to see how big the gradients in the parameter space are, how wide a range your parameters might lie in and do a quick search for local minima. Ok, you there's an inherent uncertainty if you are simply sampling to do this, but I think some time spent checking what your parameter space looks like might help to decide which algorithm you really should be using.

This still leaves the question: if I have a completely new algorithm, is it better than alternative X? I guess it's a complicated question and to some extent subjective if you don't have a particular application in mind.

I don't claim that the above is completely novel, although it is to me. A brief search has uncovered the following references, I'd be happy to add others:

-http://amstat.tandfonline.com/doi/abs/10.1080/01621459.1997.10473613 - a Monte Carlo Newton-Raphson algorithm. pdf is behind a paywall, though, so I can't read it.

- http://www.sciencedirect.com/science/article/pii/0304407688900772 also behind a paywall: a discussion of local regression. Maybe a bit off-topic, but could be thought of as similar to step 3 of my suggestion above.

I suspect, though, that most users of algorithms have spent so much time convincing themselves that their method is so good it couldn't possibly be bettered by combining with some other.