Relative score aggregation - harmonic mean?

Uncategorized
1

Hello,

Bit niche but I had a question about the way we aggregate forecast relative skill scores, and hoped people here would be well placed to help. (TLDR:) I wondered if the harmonic mean might be a useful average for summarising relative, proportional scores (i.e. relative WIS)?

(TL) background:

When evaluating forecast skill, sometimes we want to summarise forecast scores across an unbalanced sample of participant forecasters, across many different targets. We have been handling this by using the relative skill score. This compares individual forecaster scores each relative to another, for a particular forecast target, and then collates these for each forecaster and takes the geometric mean.

If* I understand correctly, when we get to summarising again across multiple forecasters’ relative scores, the method of aggregation isn’t obvious. Taking either arithmetic or geometric mean across relative scores, means averaging across ratios that are unbalanced (with potentially different sample size underlying each ratio). This might lose the propriety of the resulting summary. [*big if]

I was reading about the harmonic mean in a different context recently and wondered if we could use this to help. It better summarises across a set of proportions, by taking the reciprocal of the arithmetic mean of reciprocals.

I saw it explained by analogy to summarising “average speed” of a vehicle that varies its speed across lengths of the same distance (e.g. 20mph for 2 miles, 100mph for the next 2 miles). The arithmetic mean = (20mph + 100mph) / 4 miles; the harmonic mean = 4 miles / (1/20 + 1/20 + 1/100 + 1/100). It’s also used in finance, for averaging across an unbalanced portfolio of different companies’ price/earnings multiples.

This all reminded me of the issue of averaging relative skill scores, and I wondered if the harmonic mean might be a neat way to summarise across relWIS.

2

Hi - responding to this with a bit of a delay.

I don’t have a great intuition for the harmonic mean, but my experience with modes of aggregation other than arithmetic mean is that they break propriety of scores. We have an ongoing project where we looked into the geometric means of (task-wise) relative scores. That didn’t work well and I tend to think that whenever we take inverses of scores things somehow become weird. A related issue is that the harmonic mean is not well-defined if one score equals zero (unless there is some “natural” convention of handling that which I am not aware of?) We don’t see CRPS or WIS = 0 often, but in low-count settings (e.g., long stretches of zero values in off-season) we might.

But upon re-reading your post I wasn’t quite sure anymore whether I got your argument right - are you suggestiong harmonic mean to aggregate individual scores or to aggregate score averages across different strata or similar?

Have you ever tried this out in practice? If it somehow “behaves like one would like it to” it might be a helpful complement to regular score averages even if it is not proper.

1 Like