@kath-sherratt and I met today to talk about this. We got a bit side tracked talking about performance above replacement which I think we are going to focus on in the short term ( Baseball Stats, Model Cards, and Forecasting Performance - #8 by samabbott ). In terms of taking ideas forward here we saw a few projects:
- Making surrogate models from ensembles of baselines as a way of understanding performance
- Decomposition of models into key features and doing dimension reduction
- Making surrogate models by ensembles of decomposed baselines
- Using a “basket of baselines” as the baseline model
The curent plan is that @kath-sherratt is going to take forward a project using the first and last ideas as part of a wider fellowship application. Her very cool idea here is to do so as an expert elicitation exercise. If anyone wants to be involved with that reach out to her. We see third project as probably being an extension of the first one and so needs to wait. We see the decomposition idea as being independent and a nice project for i.e. a PhD student if anyone has anyone in mind. @kath-sherratt please correct, fill in anything I have missed as you think is needed.