@kath-sherratt and I met today to talk about A basket of baselines - #12 by samabbott as an off shoot of that conversation we thought that it would be really great to write a nice tightly scoped piece on the idea in here to evaluate models by value above replacement.
This would be a remix extension of some of the ideas from @nickreich and cos model importance work (with the extension here being replacement) and showing when it might be needed by looking at different ensemble sizes (under the assumption that as performance asymptotes with the addition of more models so should the difference betwee just looking at leave one out and performance above replacement).
We also thought there was an interesting discussion to be had in terms of what to replace with i.e all other models as a permutation, a baseline (hence the link to the basket of baselines), the ensemble of all other models (we thought this might end up being the same as the all permutation idea). There is a clear extension here to weighted replacements i.e more similar models or the extreme category version of this i.e. models from the same “family” for now we thought we would leave that to the discussion.
The plan is that we will draw up a short analysis plan over the next week or so and then reach out to interested parties from here to set up a meeting. Keen to hear peoples thoughts.