Thanks for the great talk @kejohnson9! Just one follow-up question on continuous model evaluation, as we ran out of time in the meeting…
You said that regular model evaluation helped you during the model development process. As I understood, the evaluation was based on real-world forecasting performance. On the one hand, I can see how this is great because it explicitly represents the end goal of your work and it evaluates the full interplay of your model components on actual data, so you cannot trick yourself using simulation etc. On the other hand, I wonder how informative forecast scores can be for identifying your model’s limitations in the early development stage. I could imagine that at this stage, forecasts might be pretty bad, but this can be due to various factors - and combinations of factors. How does this help you to get insights like “Oh, maybe we should weigh site-level Rt variation by catchment size” etc. I guess I am just missing concrete examples of how forecast performance can highlight individual problem areas of a model as complex as yours.