A couple of tweaks to the Impact Factor

My last post posed the question whether you could improve the impact factor. In this post, I want to put forward a proposal. It has two parts: the first is to apply a weighted average to citations, the second is to change the time resolution.

The Equation

Before we launch into that, here is something that I haven't seen in the literature, although I'd be glad if someone can point out a source: a robust equation for impact factor (that doesn't involve a lot of hand-wavy textual descriptions of the terms). If you're not too mathematically inclined, you can just skip the next section, although I truly believe that spending some time studying the equation can give some key insights into how impact factor works: it did for me, at least.

Consider a corpus of publications \(W\). For a subset \(J\in W\) there is a corresponding set \(C\) where \(C = \{c \in W | j \sim c\) for  some \(j \in J \}\), where \(a \sim b\) means that paper \(a\) is cited by \(b\). Then we define a general citation metric as

\( M = \frac{ f_{1} \left( \sum_{p\in J}{f_{2}} \sum_{c\in C}{f_{3}(c)} \right) }{||J||} \)

in particular, the Thomson Reuters impact factor for year \(y\) uses:

\( f_{3}(c) = 1\) if \( \lfloor t(c) \rfloor_{Y} = y \) and \( 0 \) otherwise

\(f_{2}(p) =  1 \)  if \( y-2 < \lfloor t(p) \rfloor_{Y} < y-1 \) and \(0\) otherwise

\(f_{1}\) is the identity function.

where we adopt the notation that \(\lfloor t(p) \rfloor_{Y}\) is the year of publication of a paper \(p \in W\) (by analogy to the floor function for real numbers: the date of publication is rounded down to the next full year).

So, let's move onto some possible modifications.

How much is a citation worth?

The underlying assumption of citations metrics is that there is a correlation between quality and citation. In the current calculation of impact factor, each citation counts for the same. A metric like the eigenfactor takes into account the impact factor of the citing journal, although to me that seems to just compound the uncertainty-by-proxy that impact factor introduces.

Consider four papers of the same age: one has 3 citations, one 5, another 100 and the fourth 102 citations. What is the difference in quality between these papers? The paper with 5 citations is probably (although not necessarily) of higher quality than the one with 3, but the difference between the papers with 100 and 102 citations is more marginal. In other words, the higher the number of citations gathered, the less each citation tells us about the quality of the paper.

What if each subsequent citation was worth slightly less? We can construct a geometric series: each citation is worth some fixed fraction of the previous citation. For example, let's suppose that the second citation is worth 0.9 times the first citation. A paper with three citations would have a score 1 + 0.9 + 0.81, slightly less than 3. Going back to high school mathematics, these series can be easily summed. Assume that the \(N\)th citation has a score of \(h\). Let \(d = h^{\frac{1}{N-1}}\), then the sum of scores of a paper with \(c\) citations will be

\( s(c; h,N) = \frac{d^{c} - 1.}{d-1.}. \)

We can pick a value \(N\) at which we judge a citation to be worth half (\(h=\) 0.5) of the first citation and can then sum across papers to get a final citation metric for the corpus of work we are looking at. My suggestion is to set the scale to have the tenth citation worth 0.5, although a more gradual scale might work better in practice. You could use the inverse of the the equation for \(d\) to calculate a number more equivalent to a number of citations.

When does a citation count?

A limitation of the calculation of impact factor is that in the two year period considered for the calculation, papers published at the beginning and the end of the period are treated equally. This means that an easy way to increase your impact factor in the short-term is to publish fewer papers in a given year. Also, journals that experience a rapid expansion will see a drop in impact factor, even if the quality of published papers is the same or marginally higher. This is a particular problem for relatively new journals.

I suspect that the choice of timeframe is a product of the time that impact factor was created in the 1970s. Not much of the process was automated and it would have been painstaking to work through all of the citations. It probably also accounts for why the values were, and still are, announced just once a year.

The distribution of citations over time is rather complex, so there is not a simple weighting that can be introduced to account for the different ages of papers. Why not, instead, compare papers at a fixed time after publication? The average time between publication and citation of papers in the current impact factor is eighteen months. Either eighteen months or two years would seem sensible options.

Given that, why not calculate the new citation metric in real time, instead of only announcing values once a year? These days, citations can predominantly be extracted automatically. I see few technical barriers to being able to achieve this.

What does the new impact factor look like?

Here is my proposed citation metric, which looks a lot like the impact factor, but should be more fair. Using the general equation above:

\( f_{3}(c) =  1 \) if \( p-c \leq 2\) and \(0\) otherwise

\(f_{2}(p) = \frac{d^{c} - 1.}{d-1.} \) if \( t-p \leq 2\) and \(0\) otherwise

What next?

There's a lot of comments to make about this, I won't go into a deep discussion here, I just want to introduce the concepts. For a start, the first equation above defines a whole family of citation metrics. You could also combine other data into the functions, such as that coming from altmetrics, to customize further.

The main point is that it may be an interesting idea, but it all needs testing! I need data and to see how robust the calculation is in different situations. That will be the subject of my next post on this topic.