Getting hold of citation data

Where do you get citation data from? This is a problem that I've found surprisingly difficult to answer. Below are a few possible sources and the problems I've encountered. My biggest problem has been time resolution. I want to know what date the citing and cited paper were published, but the downloads I have got my hands on only give the year in almost all cases. Pretty basic, public, but inaccessible information.

Perhaps the most obvious answer, especially if you are looking at impact factor, is Thomson Reuters' Web of Science. There's a few problems with this. Firstly, you need to be privileged enough to have access, which is not a huge barrier if you are somewhere in or near academia. You can do a cited article search, but the number of articles you can look at is limited to 500, meaning that only moderately large journals of work can be analysed. There is also the date resolution problem I mentioned above.

Crossref is a database contributed to by publishers. At least in this case access is free. There is a simple GUI, but I couldn't work out how to query articles that cite. You can use the API, but an expert tells me that the results are also limited to year in time resolution. In addition, the data is not complete: not all publishers submit data to Crossref and often with months worth of delays. It is not reliable for an up-to-the-minute, accurate picture.

You could try approaching publishers directly, but how many separate messages is that, and I simply don't have the time! Also, publishers treat the bibliography as part of the article which means that if the article is paywalled, so is the bibliography so you need to pay or get persmission, unless there is some kind of data-mining policy in place. This ends up in a minefield of permissions.

Google scholar do citations as well, but as far as I know there is no systematic way to download the data, and I have heard rumours that they are restricted on how they use the data, so probably couldn't release it even if they wanted to (see comments about publishers).

Is it too much to ask to have a comphrehensive database of citations? For better or worse, this is the basis for how  most researchers are assessed, so accurate data would seem to be a priority.

Just a final thought, what does and doesn't count as a citation is a much more complex question that it might at first seem. Google scholar, Crossref and Web of Science all give different numbers for citations. This is based on the selection of journals they use to source citations and probably also the quality of data - I suspect Thomson Reuters put a lot more work into verifying data than the others mentioned above, although I'm sure can't fix all mistakes.

The solution? To be honest, I have no idea beyond encouraging publishers to take Crossref deposition seriously and allowing users to access the publication date. Crossref data goes into Ithenticate, the standard software for detecting plagiarism, so a good motivation would be to persuade publishers that they can reduce misconduct and retractions by depositing quickly. Accurate citation data would be a nice side-product and of great benefit to those whose careers depend on it.

A couple of tweaks to the Impact Factor

My last post posed the question whether you could improve the impact factor. In this post, I want to put forward a proposal. It has two parts: the first is to apply a weighted average to citations, the second is to change the time resolution.

The Equation

Before we launch into that, here is something that I haven't seen in the literature, although I'd be glad if someone can point out a source: a robust equation for impact factor (that doesn't involve a lot of hand-wavy textual descriptions of the terms). If you're not too mathematically inclined, you can just skip the next section, although I truly believe that spending some time studying the equation can give some key insights into how impact factor works: it did for me, at least.

Consider a corpus of publications \(W\). For a subset \(J\in W\) there is a corresponding set \(C\) where \(C = \{c \in W | j \sim c\) for  some \(j \in J \}\), where \(a \sim b\) means that paper \(a\) is cited by \(b\). Then we define a general citation metric as

\( M = \frac{ f_{1} \left( \sum_{p\in J}{f_{2}} \sum_{c\in C}{f_{3}(c)} \right) }{||J||} \)

in particular, the Thomson Reuters impact factor for year \(y\) uses:

\( f_{3}(c) = 1\) if \( \lfloor t(c) \rfloor_{Y} = y \) and \( 0 \) otherwise

\(f_{2}(p) =  1 \)  if \( y-2 < \lfloor t(p) \rfloor_{Y} < y-1 \) and \(0\) otherwise

\(f_{1}\) is the identity function.

where we adopt the notation that \(\lfloor t(p) \rfloor_{Y}\) is the year of publication of a paper \(p \in W\) (by analogy to the floor function for real numbers: the date of publication is rounded down to the next full year).

So, let's move onto some possible modifications.

How much is a citation worth?

The underlying assumption of citations metrics is that there is a correlation between quality and citation. In the current calculation of impact factor, each citation counts for the same. A metric like the eigenfactor takes into account the impact factor of the citing journal, although to me that seems to just compound the uncertainty-by-proxy that impact factor introduces.

Consider four papers of the same age: one has 3 citations, one 5, another 100 and the fourth 102 citations. What is the difference in quality between these papers? The paper with 5 citations is probably (although not necessarily) of higher quality than the one with 3, but the difference between the papers with 100 and 102 citations is more marginal. In other words, the higher the number of citations gathered, the less each citation tells us about the quality of the paper.

What if each subsequent citation was worth slightly less? We can construct a geometric series: each citation is worth some fixed fraction of the previous citation. For example, let's suppose that the second citation is worth 0.9 times the first citation. A paper with three citations would have a score 1 + 0.9 + 0.81, slightly less than 3. Going back to high school mathematics, these series can be easily summed. Assume that the \(N\)th citation has a score of \(h\). Let \(d = h^{\frac{1}{N-1}}\), then the sum of scores of a paper with \(c\) citations will be

\( s(c; h,N) = \frac{d^{c} - 1.}{d-1.}. \)

We can pick a value \(N\) at which we judge a citation to be worth half (\(h=\) 0.5) of the first citation and can then sum across papers to get a final citation metric for the corpus of work we are looking at. My suggestion is to set the scale to have the tenth citation worth 0.5, although a more gradual scale might work better in practice. You could use the inverse of the the equation for \(d\) to calculate a number more equivalent to a number of citations.

When does a citation count?

A limitation of the calculation of impact factor is that in the two year period considered for the calculation, papers published at the beginning and the end of the period are treated equally. This means that an easy way to increase your impact factor in the short-term is to publish fewer papers in a given year. Also, journals that experience a rapid expansion will see a drop in impact factor, even if the quality of published papers is the same or marginally higher. This is a particular problem for relatively new journals.

I suspect that the choice of timeframe is a product of the time that impact factor was created in the 1970s. Not much of the process was automated and it would have been painstaking to work through all of the citations. It probably also accounts for why the values were, and still are, announced just once a year.

The distribution of citations over time is rather complex, so there is not a simple weighting that can be introduced to account for the different ages of papers. Why not, instead, compare papers at a fixed time after publication? The average time between publication and citation of papers in the current impact factor is eighteen months. Either eighteen months or two years would seem sensible options.

Given that, why not calculate the new citation metric in real time, instead of only announcing values once a year? These days, citations can predominantly be extracted automatically. I see few technical barriers to being able to achieve this.

What does the new impact factor look like?

Here is my proposed citation metric, which looks a lot like the impact factor, but should be more fair. Using the general equation above:

\( f_{3}(c) =  1 \) if \( p-c \leq 2\) and \(0\) otherwise

\(f_{2}(p) = \frac{d^{c} - 1.}{d-1.} \) if \( t-p \leq 2\) and \(0\) otherwise

What next?

There's a lot of comments to make about this, I won't go into a deep discussion here, I just want to introduce the concepts. For a start, the first equation above defines a whole family of citation metrics. You could also combine other data into the functions, such as that coming from altmetrics, to customize further.

The main point is that it may be an interesting idea, but it all needs testing! I need data and to see how robust the calculation is in different situations. That will be the subject of my next post on this topic.

What could be done to change the impact factor?

How we love and love to hate the impact factor. Scholars have an undwindling attraction to its simplicty, while at the same time bemoaning its oversimplicity and crude application. Its strength has also been its downfall, you could say. Except that no downfall seems to be iminent, despite the hopes of many.

So, what to do? There's no prospect of the revolutionaries who would ditch this all-pervasive metric succeeding, and nothing to replace it that comes close to holding the same kind of sway over grant bodies and promotion committees. Here's an option that I don't think has been given a serious airing as yet: let's try improving the impact factor. Revolution is not working, so let's try an evolutionary approach.

I have a couple of suggestions of my own, but I'll keep them under my hat for now because the aim of this post is to make you think. What would a 'better' impact factor look like? Here's a few criteria I think it has to follow:

1. It has to be intuitively simple. The beauty and attraction of the impact factor are that anyone gets a grasp of what it's trying to measure: if you publish in this journal, on average you will receive this many citations. Arguments aside about how well it achieves such a feat, you can't beat it for conciseness.

2. It must be a citation metric. Wonderful and interesting as alternative metrics are (e.g. Facebook likes, mentions on Twitter and the like), scholars rate success on how many other scholars mention your of work when they publish themselves. It's a high-brow popularity contest. Other metrics simply don't correlate well with citations.

3. It has to solve at least one of the flaws of the impact factor. Hopefully it's a given that any new proposal must be better than what is currently on offer.

4. It should give a fair reflection of a body of work. The impact factor is not an article level metric, and there are occasions where that is useful - usually where you have a great deal of data to analyse and need a brief summary. 

That's my criteria. Suggestions, additions, variations and comments are welcome.