I’d be interested in people in this forum’s view on this pre-print:
From metric to action: An evaluation framework to translate infectious disease forecasts into policy decisions, especially as I know there’s a lot of ongoing thought in this space.
My main take away is it’s great there’s a framework and someone’s fleshed out ideas that have been kicking about the last few years, such as in this paper When are predictions useful? A new method for evaluating epidemic forecasts - PMC.
A reflection I have as a “within system” practitioner is that the “non-trivial” part of getting decision makers to articulate how they make decisions in a way that can be put into a framework is extremely tricky, which could have been engaged with more in the paper.
To provoke some discussion, should REV replace WIS/rWIS/log-WIS?
Perhaps a future talk?
3 Likes
Yes I agree this is interesting. I thought they would be a good person for the seminar. If anyone knows them a connect would be handy or I will cold email.
From what I have read so far there is a lot in here that chimes with things we have been thinking about (I am a fan that there is in general a lot in here).
Another similar one that comes to mind is from @nickreich and co: https://academic.oup.com/jrsssa/advance-article/doi/10.1093/jrsssa/qnae136/7927906
how they make decisions in a way that can be put into a framework is extremely tricky
Yes, my external experience is the same. Another issue is that the more you tune your metrics for a decision the more you have to do custom model selection for any given use. This is one of the reasons I am such a fan of a basket of measures and trying to give a concise summary of what they mean (this is one of the reasons I like the fun baseball scorecard idea
which could have been engaged
My critique here is probably that this skates over some of the reasons this isn’t done and also the other work that exists on this but hey can’t have everything and its a big field!
3 Likes
I like this preprint. Would be interested to hear a talk on it. To respond to “should REV replace WIS and its variants”? If you are like us (California), and focused on a limited geography, potentially in very limited use cases? If you have a defined threshold and sense of C/L ratios they discuss. But those are hard enough to estimate and by their own admission will vary widely by disease/scenario/decision-maker. For hubs and more generalized forecasting efforts, it doesn’t seem possible.
It’s a hard problem so I appreciate their efforts to chip away at it.