# Baseball Stats, Model Cards, and Forecasting Performance

**URL:** <https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322>\
**Category:** Project Proposals\
**Created:** [20 July 2025 14:14 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322 "2025-07-20T14:14:00Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [20 July 2025 14:14 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/1 "2025-07-20T14:14:00Z")

</div>

**TL;DR:** A baseball game in Atlanta led to some ideas for visual scorecards, performance-above-replacement metrics, and effort-normalised evaluation that seem useful/interesting.

I have been in Atlanta for the last few days teaching at SISMID ([course materials with some really good (not biased) new forecasting content and other tweaks](https://nfidd.github.io/sismid/)) with Nick Reich and Thomas Robacker. It was a lot of fun even if Atlanta is about 300% too hot for me.

Nick was kind enough to organise a trip to the Braves vs Yankees baseball game, which was super fun. Something that really got to me was the (famously) large amount of stats everywhere, especially on the big screens where new batters were shown with their summary stats (and another similar one for the pitcher).

 ![IMG_1674.HEIC](https://europe1.discourse-cdn.com/flex005/uploads/epinowcast/original/1X/bcf8231dccb178f0d9f381b37f4a13631fc57686.jpeg)

This got me thinking about how nice it would be to have something like this for infectious disease forecasting models that people could use in their READMEs to summarise performance, or forecast hubs could use as a way of visually summarising a model.

We then had to escape the stadium and naturally, this meant camping in a car park for a bit whilst all the traffic cleared. This gave me some time to think about how these score cards might interact with the kinds of model cards that [gen AI has started using](https://huggingface.co/docs/hub/en/model-cards). These are basically YAML formats that go in your README that contain model and performance metadata following certain standards.

I naturally then spent a lot of my free day in Atlanta thinking about this whilst zipping up and down the [beltline](https://beltline.org/visit/). The conclusion of this is that I think there is a fairly natural way to express this as an extension of `scoringutils` that outputs both a model card (i.e. YAML) and a scorecard (an image). Claude and I have been iterating on a [design document](https://claude.ai/public/artifacts/ffdff73d-f930-4351-9c92-58f27fe5dcdc) and I am very keen for feedback. Importantly, I am keen to know if there are examples of this kind of thing in the wild, as it’s really a very general and non-domain-specific concept, so it feels like there very much might be.

The score card prototype looks like:

```auto
┌─────────────────────────────────────────────────────────────────┐
│ [Logo] MODEL NAME PAR: +2.3%/+1.9% (overall) │ <- Header (Rows 1-3)
│ Team/Organization Nat: ▁▃█▂ (1-4w) │
│ Log: ▂▄█▃ (1-4w) │
├─────────────────────────────────────────────────────────────────┤
│ ┌──────────┬──────────┬──────────┬──────────┬────────────────┐│ <- Metrics (Rows 4-8)
│ │Coverage │ WIS │ Rel │ Bias │ Ensemble ││
│ │ 50%: 48%↓│ Nat: 42↑│ Skill │ -0.02 ↓ │ Contrib ││
│ │ 90%: 87%↓│ Log:0.38│Nat: 0.95↑│ │ Nat: +3.2%↑ ││
│ │ │ │Log: 0.87↑│ │ Log: +2.8%↑ ││
│ └──────────┴──────────┴──────────┴──────────┴────────────────┘│
├─────────────────────────────────────────────────────────────────┤
│ [Performance Timeline Graph - Model vs Others] │ <- Timeline (Rows 9-11)
├─────────────────────────────────────────────────────────────────┤
│ Forecasts: 127 | Since: 2023-01 | Target Coverage: 95% │ <- Footer (Row 12)
│ Best: 2-week ahead | Most consistent Q3 2024 │
└─────────────────────────────────────────────────────────────────┘

```

_A concern is that this is meant to be fun and as I have iterated it has become progressively less fun and more information dense. Definitely something to watch for. I think maybe adding the plots and logos will edge this back in the fun direction._

We also had a good chat about the types of stats that are widely used in baseball and the main one I thought was interesting was performance above replacement. This as I understand it is weighing all the actions players take vs the average action as a way of assessing value (I would like a version of this that is weighted to player cost - more on this in a second). There are things like this that we do (i.e. value to an ensemble where you look at ensemble performance with and without your model) but I think nothing quite like it. I think we could get closer if we looked at permutations of models in ensembles (as both @sbfnk and co and [Spencer Fox and co](https://wwwnc.cdc.gov/eid/article/30/9/24-0026_article) have been doing recently). So this would look like taking ensemble performance with our model and dividing by the mean performance of all possible ensembles created by removing our model and instead duplicating another component model. A caveat is I haven’t really thought through how this relates to other measures we already have (i.e. the ensemble with and without, just taking relative skill etc.). A nuance here is you might want to check replacement by some sub strata or categories (i.e. replacement across renewal models etc. but I think except in very large forecasting problems with lots of problems this is likely to run into issues.

If it does shake out as being useful, then as with baseball, I like the idea of trying to normalise against effort. I have had a bugbear for a while at how we confound model performance by modelling effort. All the forecasting I have done points to a relationship between having more time to spend reviewing, iterating and just looking at the forecasts and how you do in evaluations. This should mean that well-resourced teams that focus heavily on forecasting do very well, whilst less resourced teams do less well. There is also some interaction with model complexity as a more complex model takes more time and thus leaves less effort for the other forecasting tasks that might be the thing driving performance. This would bias us towards simpler models (it’s convenient I think this isn’t it). Another way to think about this is an “house effect” where all models from a team should have some common performance aspects and some that vary.

Something to note is we can also get at this doing model-based evaluation (i.e. adjusting for team size etc.) but this makes us need some fairly complex other ideas so having a good summary measure might be useful.

## Summary

For me, the baseball statistics displayed at the game highlighted three gaps in how we evaluate forecasting models:

- Visual scorecards could make model performance more accessible than current approaches.
- Performance-above-replacement metrics might better capture model value within ensembles than existing measures.
- Effort-normalised evaluation could address the confounding between model performance and development resources.

---

<div class="post-metadata">

**Author:** ![trobacker](https://avatars.discourse-cdn.com/v4/letter/t/898d66/32.png) [@trobacker](https://community.epinowcast.org/u/trobacker)\
**Post date:** [20 July 2025 15:39 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/2 "2025-07-20T15:39:33Z")

</div>

Very cool! I like the prototype score card, informative.

> _I think maybe adding the plots and logos will edge this back in the fun direction._

And emojis! 😆 As Claude would likely try to do (based on it’s model submission reports during SISMID). 🏆  
Perhaps even a metadata entry for team colors.

---

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [20 July 2025 20:14 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/3 "2025-07-20T20:14:42Z")

</div>

Yeah in the spec is the idea a team might have a colour scheme. I think it might be fun to add a hub or other colour scheme (i.e. across models) and a model/team colour scheme as well

---

<div class="post-metadata">

**Author:** ![nickreich](https://avatars.discourse-cdn.com/v4/letter/n/d2c977/32.png) [@nickreich](https://community.epinowcast.org/u/nickreich)\
**Post date:** [21 July 2025 02:25 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/4 "2025-07-21T02:25:18Z")

</div>

I’m excited about this idea!

Noting that Minsu Kim is working on this idea of “model importance” that is basically an implementation of a version of the “score all the permutations of models” idea. The core ideas are in [her recent preprint](https://arxiv.org/abs/2412.08916) which was just revised and resubmitted to Intl Journal of Forecasting.

And [an R package](https://github.com/mkim425/modelimportance) is in the works too…

I’ve been wondering for a while if we could create more interesting forecasting collaborations if the primary scoring metric was something like this. Instead of “make the best prediction you can” it’s more like “make the best prediction you can that no one else is already making”. Which I guess at an extreme could turn into trying to predict when and how other predictions will fail.

As another example of baseball stats “scorecards” check out these [umpire ratings](https://umpscorecards.com/data/single-game/777114) that are posted after every game.

Here’s the one from the game on Friday night:

 ![image](https://europe1.discourse-cdn.com/flex005/uploads/epinowcast/original/1X/1b3816bce91bb432be044ac36c157f4e69ee1e92.jpeg)

---

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [21 July 2025 12:57 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/5 "2025-07-21T12:57:52Z")

</div>

> [@nickreich](#):
>
> “score all the permutations of models” idea

Nice! Looking forward to trying out the package.,

I think this doesn’t extend to the replacement component mentioned here though? i.e. its the leave one or a subset out and normalise relative to the ensemble (or take difference) vs here leave one out and replace it with another for all permutations, then take the mean (or not if wanting a range) and use this to normalise the ensemble that includes the target model. The latter is closest, I think to the baseball metric but I haven’t thought deeply on tradeoffs.

> Which I guess at an extreme could turn into trying to predict when and how other predictions will fail.

Yes, I agree with the general idea and also with trying to push towards this extreme at least a little.

> [@nickreich](#):
>
> Here’s the one from the game on Friday night:

Oh, very cool. This has more of the information dense vibe that the design prototype has been heading towards.

---

<div class="post-metadata">

**Author:** ![nickreich](https://avatars.discourse-cdn.com/v4/letter/n/d2c977/32.png) [@nickreich](https://community.epinowcast.org/u/nickreich)\
**Post date:** [22 July 2025 02:19 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/6 "2025-07-22T02:19:14Z")

</div>

> [@samabbott](#):
>
> …vs here leave one out and replace it with another for all permutations…

Yes I agree that Minsu’s work is different in the way you describe. Although I might argue that this setup is more appropriate in our setting. In sports, if you don’t play there WILL be a replacement (unless I guess you get a red card in a football match). In ensemble forecasting, the models are “unique” and would not necessarily be replaced. So I’d argue that the more “true-to-life” metric is one that doesn’t replace a model with a generic one but just leaves it out.

---

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [4 August 2025 09:47 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/7 "2025-08-04T09:47:00Z")

</div>

Hmm I guess the replacement is the alternative baseline model I might submit. The issue you have if you don’t adjust for this you end up conflating the value of a given model with the value of an ensemble just having more random models that represent some error distribution i.e. how many models like analyses.

Also I would lightly push back on the idea that models are “unique” as often quite derivative of each other right especially in larger hubs. This replacement idea also I think starts maybe getting at that a bit as it adjusts for model weight in some sense to give the uniqueness value (or lol maybe it doesn’t who knows).

---

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [10 September 2025 12:12 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/8 "2025-09-10T12:12:56Z")

</div>

@kath-sherratt and I met today to talk about [A basket of baselines - #12 by samabbott](https://community.epinowcast.org/t/a-basket-of-baselines/325/12) as an off shoot of that conversation we thought that it would be really great to write a nice tightly scoped piece on the idea in here to evaluate models by value above replacement.

This would be a remix extension of some of the ideas from @nickreich and cos model importance work (with the extension here being replacement) and showing when it might be needed by looking at different ensemble sizes (under the assumption that as performance asymptotes with the addition of more models so should the difference betwee just looking at leave one out and performance above replacement).

We also thought there was an interesting discussion to be had in terms of what to replace with i.e all other models as a permutation, a baseline (hence the link to the basket of baselines), the ensemble of all other models (we thought this might end up being the same as the all permutation idea). There is a clear extension here to weighted replacements i.e more similar models or the extreme category version of this i.e. models from the same “family” for now we thought we would leave that to the discussion.

The plan is that we will draw up a short analysis plan over the next week or so and then reach out to interested parties from here to set up a meeting. Keen to hear peoples thoughts.

---

<div class="post-metadata">

**Author:** ![nickreich](https://avatars.discourse-cdn.com/v4/letter/n/d2c977/32.png) [@nickreich](https://community.epinowcast.org/u/nickreich)\
**Post date:** [10 September 2025 12:36 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/9 "2025-09-10T12:36:17Z")

</div>

I think once you get into weighting things get much more complicated. (How are the weights computed? if you have a model that is essentially given zero weight because it isn’t valuable, then it doesn’t matter what you replace it with?)

> [@samabbott](#):
>
> showing when it might be needed by looking at different ensemble sizes (under the assumption that as performance asymptotes with the addition of more models so should the difference betwee just looking at leave one out and performance above replacement).

I sometimes have trouble following the Sam’s rambly run-ons… But the idea is that the replacement value could be higher in settings with fewer models?

Excited that you all are thinking about this and happy to be involved in further discussions.

---

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [10 September 2025 12:55 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/10 "2025-09-10T12:55:23Z")

</div>

The idea is that ensemble without / ensemble with as a measure of performance is confounded by the size of the original ensemble as it mixes the unique contribution of that model vs the contribution of any model. I expect that as the ensemble size grows, the difference between performance above replacement and performance vs not being present at all will shrink given the assumed existence in an asymptotic relationship between ensemble size and performance.

> I think once you get into weighting things get much more complicated.

Yes, I agree. Here, the weighting is slightly different (its about weighting the permutations of the replacement not the overall ensemble) but has similar issues hence doing something simpler, i.e. replacement by models of a similar type vs any model would be an easier thing to look at.

There is also the wider point that for any ensemble with weighting, things get really complicated and we haven’t thought about it at all at this stage. That being said I think you might just weight the replacement by the same amount as the current model as a first pass.

---

<div class="post-metadata">

**Author:** ![kath-sherratt](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/kath-sherratt/32/165_2.png) [@kath-sherratt](https://community.epinowcast.org/u/kath-sherratt)\
**Post date:** [29 October 2025 00:50 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/11 "2025-10-29T00:50:08Z")

</div>

Exciting update - we are taking this on as a full investigation! We’ve drafted a pretty well fleshed out analysis plan to look at value above replacement. We’re planning to test a few different options for the proposed replacement model, and point towards lots of meaty ideas for taking this type of analysis further, in the discussion. See attached.

The team working on this is currently me, Sam, and Seb Funk at LSHTM, and Minsu Kim and Nick at UMass Amherst. All thoughts, feedback, and contributions very welcome - comment here to get involved / stay in the loop.

Next step is to work out the requirement for new code, versus building off Minsu’s existing work. Looking forward to getting this underway!

[Ensemble value over replacement.pdf](https://community.epinowcast.org/uploads/short-url/vfvNbrBqsufsJ8ZJVrqUITtHdCd.pdf) (147.7 KB)

---

<div class="post-metadata">

**Author:** ![kath-sherratt](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/kath-sherratt/32/165_2.png) [@kath-sherratt](https://community.epinowcast.org/u/kath-sherratt)\
**Post date:** [11 November 2025 23:00 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/12 "2025-11-11T23:00:53Z")

</div>

Adding the latest update 🙂

Sam and I caught up again on this today. We are generally very limited in capacity this side of the new year. The plan is now to

- Flesh out the current paper draft [timeline: the next ~2 weeks]
  - Focus is on getting detail into the methods section
  - Some extra limitations in comparison with methods from other fields (eg use of different model value metrics in ML)

- Reschedule the next team meeting (Minsu, Nick, Seb) [timeline: week of the 24th Nov]
  - As usual, anyone reading this and interested is welcome to join - just drop a message

- Plan a few days’ hackathon type to implement this [timeline: January]

---

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [19 January 2026 11:24 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/13 "2026-01-19T11:24:55Z")

</div>

@kath-sherratt and I were chatting a bit about things we could do here and we came up with another direction that I think is really quite (in the American sense!) promising.

So instead of attacking the performance above replacement directly as we have been discussing and then either ending up with an absolute or relative score we could instead think about this as performance added vs adding another model. That formulation is essentially the reverse I think but opens up a very clean implementation/narrative as long as we are happy to have a relative only score based on the point @nickreich made that model importance is just a transformation on the same scale as WIS/the original target. The downside is that the meaning of the score is perhaps not as clear to communicate.

So essentially what we could argue for is that you should calculate model importance to an ensemble, via whatever approach you fancy but probably the LOMO, and adjust this to remove the ensemble size issue. In principle, this leaves us with the same what do we replace with challenge as we currently have but here we can be guided by current practice (this made me think should we use any thing else when we normalise one score by another more generally like the ensemble or other ideas we have had for replacement) and use either a baseline or a pairwise tournament ([https://epiforecasts.io/scoringutils/reference/get\_pairwise\_comparisons.html](https://epiforecasts.io/scoringutils/reference/get_pairwise_comparisons.html)).

So this would look like either:

Performance above replacement (PAR) = (WIS of ensemble with model - WIS of ensemble without model) / (WIS of ensemble with baseline model - WIS of ensemble without baseline model)

or

PAR = model importance but using pairwise comparison

then that could also be scaled by the baseline as in the normal PC approach.

The main downside to this is that you lose the absolute option and the connection to baseball is a bit less clear (sad) but maybe the latter is avoidable if clearly arguing that comparing different adding in options is equiv to replacing?

TLDR: We could switch our approach to instead calculate model importance for all models and then either divide by i.e. a baseline model importance or do a pairwise tournament of model importance scores. This account for the ensemble size issue and give something like a performance above replacement score still I think.

---

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [30 January 2026 17:04 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/14 "2026-01-30T17:04:32Z")

</div>

The current plan is to frame this around this kind of thing where we look at both of these ideas:

- We think there is a problem measuring model importance as suggested in … as … has shown that ensembles typically improve as ensemble grows up to some …

- In order to account for this there are two possible approaches. 1. is to modify the MI metric so that it accounts for ensemble size and 2. is to use a relative form of MI where the MI for one model is normalised by the MI from other models which contribute to ensembles of comparable size.

- Sports analytics/baseball blah blah commonly evaluates the contribution of a player to a team and so may provide useful insights into this problem.

- One metric WAR which scores the contribution of player relative to a replacement player is particularly appealing.

- In this paper we adapt these ideas into a modified MI measure, performance above replace (PAR), which compares performance of a given model in an ensemble to the performance of some set of replacements models as well as approaches for directly normalising MI to account for ensemble size using the MI of other ensemble models. We compare model rankings from these approaches to rankings form unmodelled MI in two cases studies…

- We seek to provide guidance on measures to use for assessing a models contribution to a forecast ensemble…

---

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [16 February 2026 21:34 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/15 "2026-02-16T21:34:55Z")

</div>

@seabbs-bot and I did some design work on what a scoringutils style workflow could look like with PAR and model importance support.

> <https://github.com/epiforecasts/scoringutils/issues/1121>
>
> \## Summary
> 
> Measure how much each model contributes to ensemble performance, fol…lowing Kim et al. (2024).
> 
> Three methods:
> \- \*\*LOMO\*\*: remove each model, measure score difference
> \- \*\*Shapley\*\*: marginal contributions across all subsets
> \- \*\*PAR\*\*: replace each model with a baseline, measure score difference
> 
> \### API
> 
> \`\`\`r
> \# Create a method object (per-method constructor)
> method \<- importance\_lomo()
> method \<- importance\_shapley()
> method \<- importance\_par(baseline = "naive")
> 
> \# Modular: three steps, same method object passed throughout
> ensembles \<- ensemble\_permutations(data, method = method, ensemble\_fn = mean\_quantile)
> ensemble\_scores \<- score(ensembles)
> importance \<- ensemble\_importance(ensemble\_scores, method = method, metric = "wis")
> 
> \# Convenience wrapper
> importance \<- ensemble\_importance\_score(data, method = method, metric = "wis")
> \`\`\`
> 
> Per-method constructors (like \`geom\_point()\` / \`geom\_line()\` in ggplot2) return a method object containing both the permutation and importance functions.
> The same object is passed to \`ensemble\_permutations()\` and \`ensemble\_importance()\`, so the method is defined once.
> \`ensemble\_permutations()\` also accepts a bare permutation function for use cases beyond importance.
> 
> \### How it works
> 
> 1. \`ensemble\_permutations()\` creates ensemble variants as a new forecast object with informative model names (e.g. \`"ensemble\_-\_A"\` = A removed, \`"ensemble\_A-\>R"\` = A replaced by R)
> 2. \`score()\` scores the ensembles — any metric, any transform
> 3. \`ensemble\_importance()\` parses model names to reconstruct the mapping and compute importance diffs
> 
> \### Note on memory
> 
> The output of \`ensemble\_permutations()\` holds all ensemble variants in memory.
> For Shapley with N models this is 2^N - 1 variants, each containing the full set of forecast rows.
> This can be RAM-intensive for large forecast datasets.
> The convenience wrapper could mitigate this by scoring and discarding variants incrementally, but the modular API necessarily materialises everything.
> 
> \---
> 
> \## Motivation
> 
> Beyond ranking models by their individual scores, we want to measure how much each model \*contributes\* to ensemble performance.
> This is inspired by baseball's Wins Above Replacement (WAR) and discussed in \[this community post\](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322).
> 
> Kim et al. (2024) formalise this as model importance metrics for forecast ensembles.
> The key finding is that importance correlates with accuracy but reveals additional value: models that are not individually accurate but offer a unique perspective can still play an important role in an ensemble.
> A model is rewarded if it is uncorrelated with others, and penalised if the other models are already correlated with each other.
> 
> \## Design detail
> 
> \### Method objects
> 
> Per-method constructors return a list containing both functions needed for the pipeline:
> 
> \`\`\`r
> importance\_lomo \<- function() {
> list(
> permutations = permutations\_lomo,
> importance = compute\_importance\_lomo
> )
> }
> 
> importance\_par \<- function(baseline = "naive") {
> list(
> permutations = function(data, ensemble\_fn) {
> permutations\_par(data, ensemble\_fn, baseline = baseline)
> },
> importance = compute\_importance\_par
> )
> }
> \`\`\`
> 
> Each constructor can accept method-specific arguments (e.g. \`baseline\` for PAR).
> The method object is just a list — no S3 class overhead needed.
> 
> \`ensemble\_permutations()\` also accepts a bare permutation function instead of a method object.
> This supports using the permutation step for purposes beyond importance (e.g. ensemble diagnostics, subset analysis) without needing to define a full method object.
> 
> \### Step 1: \`ensemble\_permutations()\`
> 
> Returns a new forecast object containing only the ensemble variants (not the original models).
> 
> \`\`\`r
> ensembles \<- ensemble\_permutations(
> data,
> method = importance\_lomo(),
> ensemble\_fn = mean\_quantile
> )
> \`\`\`
> 
> \- \`method\`: a method object (from a constructor like \`importance\_lomo()\`) or a bare permutation function (for use cases beyond importance).
> \- \`ensemble\_fn(data\_subset) -\> forecast\`: constructs an ensemble from a subset of models. Default is equally weighted mean at each quantile level. User-supplied for flexibility.
> 
> Ensemble variants are named with informative names encoding the construction:
> \- Removal: \`"ensemble\_-\_A"\` means model A was dropped (LOMO, Shapley)
> \- Replacement: \`"ensemble\_A-\>R"\` means model A was replaced by R (PAR)
> \- Multiple removals: \`"ensemble\_-\_A\_-\_B"\` (Shapley)
> \- Full ensemble: \`"ensemble"\`
> 
> The function validates that generated names do not collide with existing model names.
> 
> \### Step 2: \`score()\` and optionally \`summarise\_scores()\`
> 
> Standard scoringutils pipeline.
> Any metric, any \`transform\_forecasts()\`, any scoring perspective works.
> This connects naturally to the aggregation-and-transformation framework (#1120).
> 
> No extra metadata columns are added, so there is nothing special to preserve through the pipeline.
> 
> \### Step 3: \`ensemble\_importance()\`
> 
> \`\`\`r
> importance \<- ensemble\_importance(ensemble\_scores, method = method, metric = "wis")
> \`\`\`
> 
> \- \`method\`: the same method object passed to step 1.
> \- \`metric\`: which score column to use for the importance calculation.
> 
> The importance function uses model names to identify variants and reconstruct the mapping.
> Can sit downstream of either \`score()\` or \`summarise\_scores()\`.
> 
> \### Convenience wrapper
> 
> \`\`\`r
> importance \<- ensemble\_importance\_score(
> data,
> method = importance\_lomo(),
> ensemble\_fn = mean\_quantile,
> metric = "wis"
> )
> \`\`\`
> 
> Wraps all three steps.
> Could score and discard variants incrementally to reduce memory usage.
> 
> \## Model naming convention
> 
> The full ensemble is \`"ensemble"\`.
> Two syntaxes cover all methods:
> \- Removal: \`"ensemble\_-\_X"\` — model X was dropped (LOMO, Shapley)
> \- Replacement: \`"ensemble\_X-\>R"\` — model X was replaced by R (PAR)
> 
> \### LOMO (3 models A, B, C)
> 
> Produces N+1 = 4 ensemble variants.
> 
> \`\`\`
> model | ...
> "A" | (original)
> "B" |
> "C" |
> "ensemble" | (full ensemble)
> "ensemble\_-\_A" | (A removed)
> "ensemble\_-\_B" | (B removed)
> "ensemble\_-\_C" | (C removed)
> \`\`\`
> 
> Importance(A) = score(ensemble\_-\_A) - score(ensemble).
> 
> \### Shapley (3 models A, B, C)
> 
> Produces 2^N - 1 = 7 non-empty subset variants.
> Only practical for ~10 models or fewer.
> 
> \`\`\`
> model | ...
> "A" | (original)
> "B" |
> "C" |
> "ensemble" | (full: {A,B,C})
> "ensemble\_-\_A" | ({B,C})
> "ensemble\_-\_B" | ({A,C})
> "ensemble\_-\_C" | ({A,B})
> "ensemble\_-\_A\_-\_B" | ({C})
> "ensemble\_-\_A\_-\_C" | ({B})
> "ensemble\_-\_B\_-\_C" | ({A})
> \`\`\`
> 
> For each model i, compute Shapley-weighted marginal contributions across all pairs (S, S∪{i}) with weights \`|S|!(n-|S|-1)!/n!\`.
> 
> \### PAR (3 models A, B, C, replacement R)
> 
> Produces N+1 = 4 variants.
> Uses replacement syntax (\`-\>\`) rather than removal syntax (\`-\`).
> 
> \`\`\`
> model | ...
> "A" | (original)
> "B" |
> "C" |
> "ensemble" | (full ensemble)
> "ensemble\_A-\>R" | (A replaced by R)
> "ensemble\_B-\>R" | (B replaced by R)
> "ensemble\_C-\>R" | (C replaced by R)
> \`\`\`
> 
> Importance(A) = score(ensemble\_A-\>R) - score(ensemble).
> 
> \## Design notes
> 
> \### Why method objects
> 
> We explored several approaches to avoid passing method information separately to the upstream and downstream functions:
> \- \*\*Infer from model names\*\*: LOMO uses \`\_-\_\` (removal) and PAR uses \`-\>\` (replacement), so they are distinguishable. But inferring method from name parsing feels brittle.
> \- \*\*Column in the data\*\* (e.g. \`.importance\_method\`): the downstream function still needs to know which importance calculation to use, so it is effectively a second method pass — just implicit.
> \- \*\*Attribute on the data\*\*: fragile in R, silently dropped by \`rbind()\`, \`merge()\`, and other common operations.
> \- \*\*Pass method twice as separate functions\*\*: works but verbose and easy to mismatch.
> 
> A method object from a per-method constructor bundles permutation and importance functions together.
> The user defines the method once and passes the same object throughout.
> Method-specific configuration (e.g. PAR baseline) lives in the constructor.
> 
> \### Why two naming syntaxes
> 
> Removal (\`\_-\_\`) and replacement (\`-\>\`) are the two fundamental operations across all methods.
> LOMO and Shapley remove models; PAR replaces them.
> Using distinct syntax for each operation makes names unambiguous and human-readable.
> 
> \## Connection to multivariate scoring
> 
> For multivariate importance, the ensemble function needs to produce joint samples (not just marginal quantiles) so that energy/variogram scores can be computed in step 2.
> The importance computation in step 3 stays the same.
> 
> \## Scope
> 
> Start with:
> 1. \`importance\_lomo()\` method constructor
> 2. \`ensemble\_permutations()\` and \`ensemble\_importance()\`
> 3. \`ensemble\_importance\_score()\` convenience wrapper
> 4. \`importance\_par()\` method constructor
> 
> Later extensions:
> \- \`importance\_shapley()\` method constructor
> \- Multivariate ensemble creation (joint samples)
> \- Memory-efficient incremental scoring in the convenience wrapper
> 
> Since ensemble variants are just models with distinct names, \`get\_pairwise\_comparisons()\` works on the scored output without any special handling.
> 
> \## Packaging options
> 
> This functionality could live in different places:
> 
> \- \*\*Part of scoringutils\*\*: natural home given the tight integration with \`score()\` and forecast objects. Keeps the dependency graph simple.
> \- \*\*Standalone package depending on scoringutils\*\*: same modular API (\`ensemble\_permutations()\` → \`score()\` → \`ensemble\_importance()\`) plus the method constructors and convenience wrapper. The package would depend on scoringutils for \`score()\` and forecast objects but have its own release cycle.
> \- \*\*Collaborate with Kim et al.'s \[modelimportance\](https://github.com/mkim425/modelimportance) package\*\*: an existing R package by our collaborators that implements LOMO and LASOMO (their name for all-subsets with equal or permutation-based/Shapley weights). It supports multiple metrics (WIS, log score, AE, RSE) selected per output type, and provides useful missing-forecast handling in \`model\_importance\_summary()\`. It is designed for the hubverse ecosystem (\`hubUtils\`, \`hubEnsembles\`, \`hubEvals\`; input is \`model\_out\_tbl\`), with ensemble building, scoring, and importance computed together in \`model\_importance()\`. The main difference from our proposal is architectural: \`modelimportance\` wraps the full pipeline internally, whereas we propose separating the steps so that \`score()\` sits in the middle and any metric, transform, or scoring perspective can be applied. There may be an opportunity to collaborate — either by having \`modelimportance\` use our modular functions under the hood, or by providing a shared core that both packages build on.
> 
> No decision needed now — worth considering once the API design is settled.
> 
> \## References
> 
> \- Kim, M., Ray, E. L., and Reich, N. G. (2024). Beyond forecast leaderboards: Measuring individual model importance based on contribution to ensemble accuracy. https://arxiv.org/abs/2412.08916
> \- \[Community discussion on PAR and model cards\](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322)
> \- \[Kim et al. modelimportance package\](https://github.com/mkim425/modelimportance)
> \- \[Kim et al. replication code\](https://github.com/mkim425/replication\_model-importance)
> \- #1120 (aggregation-and-transformation framework)
> \- #1122 (handling missing forecasts before summarisation)
> 
> This was opened by a bot. Please ping @seabbs for any questions.

@nickreich @kath-sherratt @sbfnk aside from design stuff there is also the question of where this fits with the in dev `modelimportance` package (are we wedded to calling this thing model importance vs ensemble importance, ensemble contribution etc I find it a bit confusing).

Note that here I have a split off issue for scoring imputation support for scoring utils which I think would be a nice utility to have anyway and maybe useful for some other research questions.

---

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [24 February 2026 11:47 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/16 "2026-02-24T11:47:34Z")

</div>

I just skim read: [https://www.medrxiv.org/content/medrxiv/early/2026/02/18/2026.02.12.26346156.1.full.pdf](https://www.medrxiv.org/content/medrxiv/early/2026/02/18/2026.02.12.26346156.1.full.pdf)

by @jack @mariatang @jonathon.mellor and others which uses a performance above replacement based approach to assess model contribution to an ensemble. They argue that this is justified by a functional ANOVA decomposition (I haven’t found more detail on this yet in the paper but I am reading the ref -\_interested in hearing more).

There isn’t a justification in here for why this approach vs the model importance approach of @nickreich, Kim etc I wonder if it is the same justification as we have been following (i.e ensemble size confounding).

As a note I much prefer this being talked about as ensemble contribution vs model contribution as the latter just makes me think to what.

This is all very related to our proposal so perhaps there is an interest in having a chat about this a bit?

Will circle back when I have read it in more detail.

Also @jack the code link 404s: [https://github.com/jcken95/subensemble-evaluation](https://github.com/jcken95/subensemble-evaluation).

_Update I did some git stalking and this is just a name type.The repo lives here:_ [GitHub - jcken95/sub-ensemble-evaluation: Code supporting the manuscript "Evaluation of short-term multi-target respiratory forecasts over winter 2024-25 in England using sub-ensemble contribution analyses"](https://github.com/jcken95/sub-ensemble-evaluation)

_Great file name here:_ [sub-ensemble-evaluation/src/R/prj/nowcast/whooping.R at 8591e534fb237654dfe0a11604143a86ceebd7a5 · jcken95/sub-ensemble-evaluation · GitHub](https://github.com/jcken95/sub-ensemble-evaluation/blob/8591e534fb237654dfe0a11604143a86ceebd7a5/src/R/prj/nowcast/whooping.R)

---

<div class="post-metadata">

**Author:** ![jonathon.mellor](https://avatars.discourse-cdn.com/v4/letter/j/bbce88/32.png) [@jonathon.mellor](https://community.epinowcast.org/u/jonathon.mellor)\
**Post date:** [10 March 2026 14:00 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/17 "2026-03-10T14:00:26Z")

</div>

Thanks @samabbott, would be good to have a discussion with @jack and anyone else who’s keen.

---

<div class="post-metadata">

**Author:** ![samabbott](https://dub1.discourse-cdn.com/flex005/user_avatar/community.epinowcast.org/samabbott/32/5_2.png) [@samabbott](https://community.epinowcast.org/u/samabbott)\
**Post date:** [11 March 2026 16:52 UTC](https://community.epinowcast.org/t/baseball-stats-model-cards-and-forecasting-performance/322/18 "2026-03-11T16:52:19Z")

</div>

That sounds great to me! Do you folks want to propose a time that works for you next week? Tuesday or Wednesday from 2 is clear for me if either of those slots have something in them that work for you?