Getting hold of citation data

Where do you get citation data from? This is a problem that I've found surprisingly difficult to answer. Below are a few possible sources and the problems I've encountered. My biggest problem has been time resolution. I want to know what date the citing and cited paper were published, but the downloads I have got my hands on only give the year in almost all cases. Pretty basic, public, but inaccessible information.

Perhaps the most obvious answer, especially if you are looking at impact factor, is Thomson Reuters' Web of Science. There's a few problems with this. Firstly, you need to be privileged enough to have access, which is not a huge barrier if you are somewhere in or near academia. You can do a cited article search, but the number of articles you can look at is limited to 500, meaning that only moderately large journals of work can be analysed. There is also the date resolution problem I mentioned above.

Crossref is a database contributed to by publishers. At least in this case access is free. There is a simple GUI, but I couldn't work out how to query articles that cite. You can use the API, but an expert tells me that the results are also limited to year in time resolution. In addition, the data is not complete: not all publishers submit data to Crossref and often with months worth of delays. It is not reliable for an up-to-the-minute, accurate picture.

You could try approaching publishers directly, but how many separate messages is that, and I simply don't have the time! Also, publishers treat the bibliography as part of the article which means that if the article is paywalled, so is the bibliography so you need to pay or get persmission, unless there is some kind of data-mining policy in place. This ends up in a minefield of permissions.

Google scholar do citations as well, but as far as I know there is no systematic way to download the data, and I have heard rumours that they are restricted on how they use the data, so probably couldn't release it even if they wanted to (see comments about publishers).

Is it too much to ask to have a comphrehensive database of citations? For better or worse, this is the basis for how  most researchers are assessed, so accurate data would seem to be a priority.

Just a final thought, what does and doesn't count as a citation is a much more complex question that it might at first seem. Google scholar, Crossref and Web of Science all give different numbers for citations. This is based on the selection of journals they use to source citations and probably also the quality of data - I suspect Thomson Reuters put a lot more work into verifying data than the others mentioned above, although I'm sure can't fix all mistakes.

The solution? To be honest, I have no idea beyond encouraging publishers to take Crossref deposition seriously and allowing users to access the publication date. Crossref data goes into Ithenticate, the standard software for detecting plagiarism, so a good motivation would be to persuade publishers that they can reduce misconduct and retractions by depositing quickly. Accurate citation data would be a nice side-product and of great benefit to those whose careers depend on it.

A couple of tweaks to the Impact Factor

My last post posed the question whether you could improve the impact factor. In this post, I want to put forward a proposal. It has two parts: the first is to apply a weighted average to citations, the second is to change the time resolution.

The Equation

Before we launch into that, here is something that I haven't seen in the literature, although I'd be glad if someone can point out a source: a robust equation for impact factor (that doesn't involve a lot of hand-wavy textual descriptions of the terms). If you're not too mathematically inclined, you can just skip the next section, although I truly believe that spending some time studying the equation can give some key insights into how impact factor works: it did for me, at least.

Consider a corpus of publications \(W\). For a subset \(J\in W\) there is a corresponding set \(C\) where \(C = \{c \in W | j \sim c\) for  some \(j \in J \}\), where \(a \sim b\) means that paper \(a\) is cited by \(b\). Then we define a general citation metric as

\( M = \frac{ f_{1} \left( \sum_{p\in J}{f_{2}} \sum_{c\in C}{f_{3}(c)} \right) }{||J||} \)

in particular, the Thomson Reuters impact factor for year \(y\) uses:

\( f_{3}(c) = 1\) if \( \lfloor t(c) \rfloor_{Y} = y \) and \( 0 \) otherwise

\(f_{2}(p) =  1 \)  if \( y-2 < \lfloor t(p) \rfloor_{Y} < y-1 \) and \(0\) otherwise

\(f_{1}\) is the identity function.

where we adopt the notation that \(\lfloor t(p) \rfloor_{Y}\) is the year of publication of a paper \(p \in W\) (by analogy to the floor function for real numbers: the date of publication is rounded down to the next full year).

So, let's move onto some possible modifications.

How much is a citation worth?

The underlying assumption of citations metrics is that there is a correlation between quality and citation. In the current calculation of impact factor, each citation counts for the same. A metric like the eigenfactor takes into account the impact factor of the citing journal, although to me that seems to just compound the uncertainty-by-proxy that impact factor introduces.

Consider four papers of the same age: one has 3 citations, one 5, another 100 and the fourth 102 citations. What is the difference in quality between these papers? The paper with 5 citations is probably (although not necessarily) of higher quality than the one with 3, but the difference between the papers with 100 and 102 citations is more marginal. In other words, the higher the number of citations gathered, the less each citation tells us about the quality of the paper.

What if each subsequent citation was worth slightly less? We can construct a geometric series: each citation is worth some fixed fraction of the previous citation. For example, let's suppose that the second citation is worth 0.9 times the first citation. A paper with three citations would have a score 1 + 0.9 + 0.81, slightly less than 3. Going back to high school mathematics, these series can be easily summed. Assume that the \(N\)th citation has a score of \(h\). Let \(d = h^{\frac{1}{N-1}}\), then the sum of scores of a paper with \(c\) citations will be

\( s(c; h,N) = \frac{d^{c} - 1.}{d-1.}. \)

We can pick a value \(N\) at which we judge a citation to be worth half (\(h=\) 0.5) of the first citation and can then sum across papers to get a final citation metric for the corpus of work we are looking at. My suggestion is to set the scale to have the tenth citation worth 0.5, although a more gradual scale might work better in practice. You could use the inverse of the the equation for \(d\) to calculate a number more equivalent to a number of citations.

When does a citation count?

A limitation of the calculation of impact factor is that in the two year period considered for the calculation, papers published at the beginning and the end of the period are treated equally. This means that an easy way to increase your impact factor in the short-term is to publish fewer papers in a given year. Also, journals that experience a rapid expansion will see a drop in impact factor, even if the quality of published papers is the same or marginally higher. This is a particular problem for relatively new journals.

I suspect that the choice of timeframe is a product of the time that impact factor was created in the 1970s. Not much of the process was automated and it would have been painstaking to work through all of the citations. It probably also accounts for why the values were, and still are, announced just once a year.

The distribution of citations over time is rather complex, so there is not a simple weighting that can be introduced to account for the different ages of papers. Why not, instead, compare papers at a fixed time after publication? The average time between publication and citation of papers in the current impact factor is eighteen months. Either eighteen months or two years would seem sensible options.

Given that, why not calculate the new citation metric in real time, instead of only announcing values once a year? These days, citations can predominantly be extracted automatically. I see few technical barriers to being able to achieve this.

What does the new impact factor look like?

Here is my proposed citation metric, which looks a lot like the impact factor, but should be more fair. Using the general equation above:

\( f_{3}(c) =  1 \) if \( p-c \leq 2\) and \(0\) otherwise

\(f_{2}(p) = \frac{d^{c} - 1.}{d-1.} \) if \( t-p \leq 2\) and \(0\) otherwise

What next?

There's a lot of comments to make about this, I won't go into a deep discussion here, I just want to introduce the concepts. For a start, the first equation above defines a whole family of citation metrics. You could also combine other data into the functions, such as that coming from altmetrics, to customize further.

The main point is that it may be an interesting idea, but it all needs testing! I need data and to see how robust the calculation is in different situations. That will be the subject of my next post on this topic.

What could be done to change the impact factor?

How we love and love to hate the impact factor. Scholars have an undwindling attraction to its simplicty, while at the same time bemoaning its oversimplicity and crude application. Its strength has also been its downfall, you could say. Except that no downfall seems to be iminent, despite the hopes of many.

So, what to do? There's no prospect of the revolutionaries who would ditch this all-pervasive metric succeeding, and nothing to replace it that comes close to holding the same kind of sway over grant bodies and promotion committees. Here's an option that I don't think has been given a serious airing as yet: let's try improving the impact factor. Revolution is not working, so let's try an evolutionary approach.

I have a couple of suggestions of my own, but I'll keep them under my hat for now because the aim of this post is to make you think. What would a 'better' impact factor look like? Here's a few criteria I think it has to follow:

1. It has to be intuitively simple. The beauty and attraction of the impact factor are that anyone gets a grasp of what it's trying to measure: if you publish in this journal, on average you will receive this many citations. Arguments aside about how well it achieves such a feat, you can't beat it for conciseness.

2. It must be a citation metric. Wonderful and interesting as alternative metrics are (e.g. Facebook likes, mentions on Twitter and the like), scholars rate success on how many other scholars mention your of work when they publish themselves. It's a high-brow popularity contest. Other metrics simply don't correlate well with citations.

3. It has to solve at least one of the flaws of the impact factor. Hopefully it's a given that any new proposal must be better than what is currently on offer.

4. It should give a fair reflection of a body of work. The impact factor is not an article level metric, and there are occasions where that is useful - usually where you have a great deal of data to analyse and need a brief summary. 

That's my criteria. Suggestions, additions, variations and comments are welcome.

I've got a new job!

After a decade of research, it's time for me to look for other pastures. My search has led me to the offices of MDPI in Basel, an open access publishing company, and taking up a role first as a trainee, then in due course a fully-fledged managing editor.

I'm obviously very excited about the move and the new opportunity. I passionately believe that all publicly funded research, in fact all research, should be made publicly available. Fundamental research is not not a commercial activity - that comes further down the line - and to me there seems little to be gained by hiding your results from others. There is much to be gained from spreading the results of science as far and wide as possible. Interested amateurs and those in developing countries have a huge amount to gain from open access, and it's difficult to see what the authors have to lose.

MDPI believes this as passionately as me. They also have a professional approach, aimed at making the high quality research freely available and are committed to sustainability in many different forms. I'm hoping I make a useful contribution!

Ok, so part of me misses the daily striving for a breakthrough that comes with a research project, but it's also so exciting to involved in the final steps of the research of so many, and to be surrounded by so much research from a host of different disciplines.

The blog will continue with contributions from time to time, but possibly with a more publishing-focused bent. I should also stress that the views here are entirely my own, although every effort will be made not to offend or upset my colleagues at MDPI - I figure that will make my life easier!

Maybrain update

It's time for an update on the brain  simulation project, Maybrain. The aim is to combine a few Python modules to be able to map and manipulate the connectome of a brain (i.e. neuronal nodes and their connections) and show in conjunction with isosurfaces of the skull and other data.

There's been a few important changes over the past couple of weeks. We implemented algorithms for degeneration, either by node spatial proximity or along edges connecting nodes. There's also a whole host of analysis and input/output functions that will no doubt continue to expand over time.

On a programming level, you can now call Maybrain as a Python module and there are maybrain.write and maybrain.analysis functions to neatly separate out the writing and analytical functions from those that actually change the connectome, which are still in maybrain.brainObj. All this is explained on the wiki quickstart guide, although other wiki changes are lagging behind the development a little at the moment!

Here's a taster of the degeneration functions on a 10 by 10 grid where each node is connected to its four nearest neighbours.

Getting a project off the ground

You have an idea? You think it's a good idea? That's great! How do you go about putting it into practice? This is a really crucial step, and in this  post we'll be discussing how to do that in an open way. What I mean is that you establish a project that others can be join, advise on, or be informed about as it progresses.

I've previously blogged about an open science workflow, and this is the first stage. As I mentioned before, when there is the potential for collaborating with others it's key to be absolutely clear which stage you are at. So

Step 1: Identify that you have a new problem to solve

This means taking some time to check that it really is novel, that no-one's solved it before and that no-one is close to solving it (the latter is not so easy, unless they are also being open about their work).

Step 2: Write down what the problem is

This serves two purposes. Firstly, it makes it clear in your mind exactly what it is you want to tackle. I would recommend using a SMART approach, i.e. make the statement of the problem: specific, measurable, achievable, realistic and time-scaled. This will lead to measures of success for later, allowing you to better monitor progress. Secondly, the statement can be shared with others and will make it much clearer to them what the goal is. This means that all those involved in the project are pulling in the same direction.

Step 3: Consult stakeholders

Are there people who will be affected by the progress or outcome of the project? Get their input at this early stage. That will make it much more fit for purpose. In my experience, others can frequently make key contributions that turn a mediocre idea into a great one. This stage can also involve contacting funding bodies, which might mean you have to define a few of your methods as well (there's an upcoming post about that stage).

My experience is that most of these stages happen, but not necessarily in the correct order and with the optimum effect on the project. Planning is usually rushed and the proposed outcome defined somewhere near the end. Interested parties want to change the direction of the project when it is half-way through, which can lead to tension and be costly.

This advice has a lot to do with communication. Ask yourself, can you successfully communicate your project? Have you done so?

This advice isn't rocket science, and my hope in spelling it out is that it can be used to improve project visibility and get lots of people excited about you and your great new idea.

Eating beef with a clear conscience?

Vegetarians and vegans often, quite justifiably, claim that their way of life is morally superior to those of us carnivores because the breeding of animals is, comparatively, a waste of resources. A cow's stomach produces vast quantities of methane, a greenhouse gas, plus it requires grassland on which to feed, further land to grow additional food for it to eat and finally live transport to a slaughterhouse.

A recent paper (Sustainability 2012, 4, 3279-3301; doi:10.3390/su4123279) explores these issues and gives estimates of what, exactly is the carbon footprint of beef cattle. The bad news is that this is still quite substantial (another article from the centre for sustainable systems shows a comparison). The good news, however, is that it can be greatly reduced. The article quotes values of reduction up to 50% for various regions of the world. Here's a graph of what they reckon to be the most reliable data (from Canada):

It shows a decrease, although the last couple of points raise the question of whether such a rate can be sustained in the future. Different kinds of feed make up a lot of the difference, including how the animal is 'finished' before slaughter, i.e. what it's fattened up on make a difference, as does how the herd is managed - running alongside a dairy herd can lead to exchange that can be beneficial.

A big challenge going forward concerns application of good practice worldwide. The regions with the smallest carbon footprint are the Americas and Europe, but South America, North Australia and Asia have some catching up to do.

It's not realistic to stop people from eating red meat, but it is nice to know that there are things that can be done to mitigate the effect of such a high-consuming industry. Combined with a shove in the right direction for us to eat a little less meat could have a tangible positive effect on our environment.