Design sketch: Julia equivalents of scoringRules and scoringutils — feedback welcome

Project Proposals
1

I’ve been doing some design work (with Claude) on what a Julia version of the probabilistic forecast scoring ecosystem would look like. There’s currently no Julia package for proper scoring rules, and I think the language is a natural fit given multiple dispatch on Distributions.jl types.

The proposal is two packages:

  1. ScoringRules.jl — atomic scoring functions (CRPS, LogS, DSS, Energy Score) dispatching on Distributions.jl types, with closed-form specialisations where available and numerical fallbacks otherwise. Lightweight, AD-compatible. ( Proposal: ScoringRules.jl — proper scoring rules for probabilistic forecasts in Julia · Issue #5 · EpiAware/.github · GitHub )
  2. ForecastScoring.jl — the tabular workflow layer (transform → score → aggregate → compare), built on DataFrames.jl and ScoringRules.jl. Typed forecast objects, validation, model comparison with statistical tests, and plotting. Julia equivalent of scoringutils. (Proposal: ForecastScoring.jl — tabular forecast evaluation workflow (scoringutils equivalent) · Issue #6 · EpiAware/.github · GitHub)

I think this is something I could largely churn through with an AI agent team, so the barrier to getting it done is relatively low. But I’d appreciate design feedback before diving in — particularly on the API and how it should integrate with the wider Julia ecosystem. I guess I should connect with the scoringRules folks as well as this would be very much a point LLM at their work and have it copy it to another language kind of job.

Also happy to have volunteers if anyone wants to get involved.

1 Like
2

I have started doodling out an ecosystem structure over in the EpiAware org is anyone has any ideas please throw them down. I am not worrying too much how everything would connect yet as that is kind of for later but the assumption is that the main modelling layer either as something packages work with or something they use is Turing.jl

3

@sambrand are you aware of anyting like this in Julia already?

4

@mstapper ‘s pkg here GitHub - ManuelStapper/ForecastBaselines.jl: Repo to collect baseline models is probably the closest I can find but obviously pretty different in intent.

5

Yeah I also found that one and had a look comment in the issue. Its obviously with the intent of doign something (forecasting baselines) and also is using a struct dispatch approach that I was thinking we might save for a scoringutils adaption i.e do the base as functional with multiple dispathc for the types of i.e crps and then a struct based version to support a transform, score, aggregate like workflow.