A primitive Maybrain GUI

This is an update on the Maybrain project to plot brain data using Mayavi2. Some progress has been made in a couple of areas. Before that, though, here's a movie of Maybrain in action

The wiki

This is coming along. Pages now exist for installation and class definitions for the brain object. There are still some gaps, and lots more work to do, but it's taking shape. If there is something you really want to know, get in touch and we can fast-track progress on some part of the wiki, or send you some demo files.

The GUI

The GUI now works with basic functionality. That is to say, there are a lot of buttons that don't do anything but you  can load a brain, plot and edit colour, visibility and opacity.

Here's a brief quickstart guide to point you in the right direction (there's nothing on the wiki yet - it's all in the pipeline):

1. Start the gui by running the file maybrainGUI.py
2. Select the adjacency and spatial information files you want to load in the top two filenames of the first tab and put in the desired threshold. Some sample data is available on the project website.
3. Click the 'load' button (next to the spatial information filename) and wait until the 'plot' button is activated
4. Click the 'plot' button
5. Now go to the second tab and use the sliders, dials and check boxes to edit the properties. 

Here are a couple of screenshots:

Matching and fitting models - a perfect solution?

Following my ill-informed rant about data fitting a couple of weeks ago, I had a short exchange on Facebook that pointed me in the direction of a couple of possible solutions: Akaike information criteria (AIC) and Goedel machines.

I'm still not satisfied, although I don't think I have fully got to the bottom of the significance and power of a Goedel machine - in my spare time I am still trying to swot up on what they can do. However, these ideas got me thinking and (while in the shower, of course) came up with the following proposal:

Mix algorithms, using each when it becomes better than the others. I propose three (basic versions of) algorithms: a genetic algorithm, Markov chain Monte Carlo (MCMC) and conjugate gradient fitting. Here's a three step method for a general fit. As a reminder, we are trying to optimize the parameters of a model to minimise least squared difference (or, even better, maximize a likelihood function):

- Start off with a genetic algorithm. This is clearly better than the others when nothing is known about the parameters. It could be thought of as a kind of multi-strand MCMC with arbitrary large step size coupled with choromosome splicing which can only serve to improve the answer.

- Once parameters are known to within a certain range with e.g. 95% confidence, switch to MCMC. This avoids non-optimal sampling of random parts of the parameter space while still maintaining the ability to cope with a rough high-gradient parameter space.

- By some kind of parameter space sampling determine when it seems like there are no more local minima (ok, that's a bit wooly but I think it could be done) and gradients are 'small', then switch to conjugate gradient. Unlike the other algorithms this will give a final answer in a finite time.

If you have more information you could skip any of the steps. A danger is, of course, that checking the switching criteria could take up more effort than it is worth!

This is really a thought experiment and I don't have the time to implement such an algorithm just at the moment. I think what it shows, though, is that there is a clear case to be made as to where each algorithm is optimal. Perhaps before attempting a fit you could run a quick check to see how big the gradients in the parameter space are, how wide a range your parameters might lie in and do a quick search for local minima. Ok, you there's an inherent uncertainty if you are simply sampling to do this, but I think some time spent checking what your parameter space looks like might help to decide which algorithm you really should be using.

This still leaves the question: if I have a completely new algorithm, is it better than alternative X? I guess it's a complicated question and to some extent subjective if you don't have a particular application in mind.

I don't claim that the above is completely novel, although it is to me. A brief search has uncovered the following references, I'd be happy to add others:

-http://amstat.tandfonline.com/doi/abs/10.1080/01621459.1997.10473613 - a Monte Carlo Newton-Raphson algorithm. pdf is behind a paywall, though, so I can't read it.

- http://www.sciencedirect.com/science/article/pii/0304407688900772 also behind a paywall: a discussion of local regression. Maybe a bit off-topic, but could be thought of as similar to step 3 of my suggestion above.

I suspect, though, that most users of algorithms have spent so much time convincing themselves that their method is so good it couldn't possibly be bettered by combining with some other.

Openscience.org

To explore how to carry out science in an open way, the best way to proceed has to be to find out who is already doing it and what lessons can be learned from their approach.

Today's post is based on my thoughts after looking at openscience.org. It's a website that hosts open source software for scientists, with  the majority of programs within maths, computational chemistry and life sciences. It also hosts a blog written by Daniel Getzler which gives pointers to good uses of open source within scientific research.

The concept of open source software is one that has existed for a long time and provides probably the closest working model of what my ideal research environment might look like. In fact, many of those writing open source software are researchers, who have a vested interest in as many people using their software as possible and little reason to restrict use or charge for their code.

What the open source software model tells us is that open collaboration can work and produce some really quite stunning results. Linux springs to mind - an entire computer operating system for free, mostly built by people in their spare time and given away for the benefit of many thousands around the world. Delving deeper, the structures used for software development can also be instructive. For example, the numbering of software versions shows a clear development and gives a staging point for other pieces of software to reference and establish compatibility. Bug tracking gives a nice method for community feedback, which might be feature suggestions or code deficiencies.

Where I see the most obvious application of these kinds of methods to experimental science is in protocols and standard operating procedures. If you are not familiar with these concepts, just think of them as the recipe for an experiment. Now imagine coming up with a new procedure for converting lead into gold. Under the current system, when you publish there could be all sorts of questions as to whether the protocol was robust. I have heard anecdotal evidence that in some fields important steps are deliberately missed out in papers to create difficulty for anyone trying to replicate the experiment. Now imagine that the protocol was published as a version 1.0 on a website similar to gihub or googlecode, or maybe something like these crossed with Wikipedia. On the download page there is soon a comment from Dr Rumpelstiltskin that his lab has successfully reproduced the results. This is confirmation that it works - under the current system no such feedback is possible without another publication that no-one is interested in reading. Later Dr Al-Kumis adds an improvement that increases the yield by 10%. With your permission the protocol is moved on to version 2.0. And so on and so on.

Ultimately your protocol can be used by many people, but you can still maintain ownership of it if you wish, or hand it on to others (i.e. keeping or sharing adminitrator rights). I think the move away from traditional publishing also allows for the addition of comments like 'this only worked with my left hand' which might be necessary for the protocol to work, but cause embarrassment in a published paper. I see no reason why this cannot be combined with peer review, the thing that many researchers value about publishing above all else.

And if you have any suggestions for an improvement on this method for publishing protocols, please let me know.

An open science work flow

I'm convinced that research should be done in a more open manner. The reasons for this are discussed in numerous places by people who have taken much more time thinking it over than me (I'm also including networked science in this). What I have found less information on is how you go about it. Even the definition can be a bit vague. To me, open research means that everything you do in the project can be scrutinized and commented on by others who can also become involved in the project.

An open project requires a well defined structure so that potential collaborators can come in at any point. It also helps keep the project on track and stop it it either heading off in directions it was never intended to (ok, sometimes this is good but you need to know it's happening!), or get stuck in a rut with people spending all their time on an insignificant part.

As an aside, I was recently in a room of people discussing some data. The data was good, in fact very good. The problem was that they had not though in advance how they were going to use it! It seemed to me that the project had lost sight of being useful in practice and even a proof of principle problem had not really been defined. Just asking what the end product of the project was would have been quite useful, and should have been set out much earlier on.

Below is the work flow of a typical research project. Ok, I know it doesn't work out like that in a lot of cases, but if you think about it either it should have been like that, or it was and you just couldn't see it at the time.

An open project needs collaborative tools for all of these stages. To me what is exciting is that you can then taken any stage in isolation and work on just that. What does that mean? Imagine a scenario where you have a great idea for a project but just don't have the time to do it. You post it to a message board for open questions and someone else picks up on it, expands the problem and go and solves it. You come back at the analysis stage when the results are put on a blog (linked to your original question) and realise it has results that you can feed back into your other current research.

Even more importantly, it divorces the necessity for the person collecting the data to be the one who analyses it. Very few people have the capability to do both of these well. As long as the aims of the project are well expressed and understood, you can have experts working at every stage to really make the most of the project.

Where do we go from here? Well, you need the tools for each stage and that is something I would like to discuss in future posts.

The perfect match: model and algortihm

Today is a Friday, so a good day to blog about this kind of topic. It's something I have been wondering about for a while and occasionally made some effort to make progress on or find an answer to, but never with much success.

Here is the problem, and one I have come across in numerous contexts: I have a model, let's call it $M:R\rightarrow S$ and an algorithm, let's call it $G:R\rightarrow R$. I want to optimise the values of M such that the model fits some data. Different ways to express this would be, for example, as minimising squared difference or maximising a likelihood function.

This problem can be solved in various ways. The principal three I've seen used are some kind of steepest descent (what I call deterministic), a statistical method such as Markov chain Monte Carlo or simulated annealing (statistical) and genetic algorithms.

My question is whether, given a model, it is possible to predict which algorithm will solve the problem the best. For each of the three types of algorithm above, I have heard exponents claiming that theirs is the best, to the point that you would think it foolish to even attempt to use any other algorithm. I can easily think of problems that are particularly suited to each one, but the question is where does the boundary lie between the effectiveness of one algorithm and another? If I invent an entirely new algorithm, what properties do I need to check to see if it works well?

The answer is almost certainly that it is very difficult to tell, but can you be sure? If so, how sure can you be, and is there any good reason not to use algorithm G on problem M?

So, my applied mathematician friends, has this problem already been solved? I think my issues with trying to found out lie in the fact that algorithms come from different fields using different terminologies. What language would you use to express this problem? What sort of tools would you use to solve the problem? If no such method could exist, can you prove it?

Here is what I envisage an answer might look like: the algorithm I use has a set of scores associated with various properties, such as calculations per step, difficulty experienced by a local minimum, speed at solving a standard problem. Given my model, I can sample the domain of parameters a finite number of times to get certain properties (or even analytically find a value) e.g. does it have local minima, how smooth is the resulting space, are there large local gradients. By multiplying or combining in some way corresponding properties of algorithm and model you assess the suitability of the algorithm to the model (e.g. the algorithm scores 5 for problems with local minima, model has a score of 3 for local minima, so add 15 to final score).

Anyone interested in trying to find a solution?

Maybrain

This will be brief, more details to follow! I've just established a project on googlecode called MayBrain. Ok, so the name is a bit lame, but it is trying to get at the project being the application of Mayavi, a powerful 3D visualisation program in Python, to look at brain data.

Very little of the plotting code is original, it's just patching together modules for loading, analysing and visualising the data. The hope is that it will be useful for those working in neurology to quickly visualise data, overlay several different kinds of data and create videos to share.

If you want to get involved, please do! I'm indebted to my brother, Tim, for kicking off the project and writing a lot of the code before I became involved.

Over the next few days and weeks, the plan is to add a few extra features, finish off the GUI and write  a wiki so that it can actually be useful to someone some day.

Resurrecting the blog

This blog has been out of commission for quite some time. There are various reasons for that, but now I find I have time and motivation to get it up and going again. However, it will look a bit different, at least that is the plan.

I'd like to blog about things I'm enthusiastic about. What does that mean? Well, as far as research goes, it's Open Access, but more than that open research (aka networked research). Most published science is behind closed doors, but so also is most active science. Why? This is a question I have been thinking about for some time now and here are the principal reasons that I see:

1. Inertia in the system. The changing nature of information availability has not been matched by a change in attitude to how information is shared. Scientists are very innovative in how they think about their work, but on the whole much less innovative in how they approach a project. In addition, those currently at the top are those who have thrived in the current system - why change?

2. Credit. Every culture has its currency. In research, it is the publication and the impact of that publication. This means that each piece of research is inexorably tied to a few people and most principally the first author on the resulting paper. This means two things. Firstly, the first author must do most of the work as they are the ones to get the most credit. Secondly, collaborations where two or more people contribute to a significant degree are not in anyone's interest because only one will get the credit.

3. Funding. What drives the above two is, of course, where the money comes from. Projects are funded, not people, and the basis of allocating the money is based on publications (ok, a few other things, but that's the main factor at play). Publish more, get more money.

Not all research should be out in the open. Some of it is commercially sensitive, but most is not. In fact, sharing resources and pooling skills should lead to much more rapid progress.

One of the aims of this rejuvenated blog will be to find out who is doing open science and see how successful are being with it. I also have one or two projects myself that I will put updates of on here. Let's see if the future can be a lot more open than the past.