Getting hold of citation data

Where do you get citation data from? This is a problem that I've found surprisingly difficult to answer. Below are a few possible sources and the problems I've encountered. My biggest problem has been time resolution. I want to know what date the citing and cited paper were published, but the downloads I have got my hands on only give the year in almost all cases. Pretty basic, public, but inaccessible information.

Perhaps the most obvious answer, especially if you are looking at impact factor, is Thomson Reuters' Web of Science. There's a few problems with this. Firstly, you need to be privileged enough to have access, which is not a huge barrier if you are somewhere in or near academia. You can do a cited article search, but the number of articles you can look at is limited to 500, meaning that only moderately large journals of work can be analysed. There is also the date resolution problem I mentioned above.

Crossref is a database contributed to by publishers. At least in this case access is free. There is a simple GUI, but I couldn't work out how to query articles that cite. You can use the API, but an expert tells me that the results are also limited to year in time resolution. In addition, the data is not complete: not all publishers submit data to Crossref and often with months worth of delays. It is not reliable for an up-to-the-minute, accurate picture.

You could try approaching publishers directly, but how many separate messages is that, and I simply don't have the time! Also, publishers treat the bibliography as part of the article which means that if the article is paywalled, so is the bibliography so you need to pay or get persmission, unless there is some kind of data-mining policy in place. This ends up in a minefield of permissions.

Google scholar do citations as well, but as far as I know there is no systematic way to download the data, and I have heard rumours that they are restricted on how they use the data, so probably couldn't release it even if they wanted to (see comments about publishers).

Is it too much to ask to have a comphrehensive database of citations? For better or worse, this is the basis for how  most researchers are assessed, so accurate data would seem to be a priority.

Just a final thought, what does and doesn't count as a citation is a much more complex question that it might at first seem. Google scholar, Crossref and Web of Science all give different numbers for citations. This is based on the selection of journals they use to source citations and probably also the quality of data - I suspect Thomson Reuters put a lot more work into verifying data than the others mentioned above, although I'm sure can't fix all mistakes.

The solution? To be honest, I have no idea beyond encouraging publishers to take Crossref deposition seriously and allowing users to access the publication date. Crossref data goes into Ithenticate, the standard software for detecting plagiarism, so a good motivation would be to persuade publishers that they can reduce misconduct and retractions by depositing quickly. Accurate citation data would be a nice side-product and of great benefit to those whose careers depend on it.

What could be done to change the impact factor?

How we love and love to hate the impact factor. Scholars have an undwindling attraction to its simplicty, while at the same time bemoaning its oversimplicity and crude application. Its strength has also been its downfall, you could say. Except that no downfall seems to be iminent, despite the hopes of many.

So, what to do? There's no prospect of the revolutionaries who would ditch this all-pervasive metric succeeding, and nothing to replace it that comes close to holding the same kind of sway over grant bodies and promotion committees. Here's an option that I don't think has been given a serious airing as yet: let's try improving the impact factor. Revolution is not working, so let's try an evolutionary approach.

I have a couple of suggestions of my own, but I'll keep them under my hat for now because the aim of this post is to make you think. What would a 'better' impact factor look like? Here's a few criteria I think it has to follow:

1. It has to be intuitively simple. The beauty and attraction of the impact factor are that anyone gets a grasp of what it's trying to measure: if you publish in this journal, on average you will receive this many citations. Arguments aside about how well it achieves such a feat, you can't beat it for conciseness.

2. It must be a citation metric. Wonderful and interesting as alternative metrics are (e.g. Facebook likes, mentions on Twitter and the like), scholars rate success on how many other scholars mention your of work when they publish themselves. It's a high-brow popularity contest. Other metrics simply don't correlate well with citations.

3. It has to solve at least one of the flaws of the impact factor. Hopefully it's a given that any new proposal must be better than what is currently on offer.

4. It should give a fair reflection of a body of work. The impact factor is not an article level metric, and there are occasions where that is useful - usually where you have a great deal of data to analyse and need a brief summary. 

That's my criteria. Suggestions, additions, variations and comments are welcome.