I’ve been thinking about this too. I think there’s still room for both the top-down and bottom-up approaches to suggesting baselines.
(Top-down: evaluating across existing models’ error, considering predictability/other factors (with due respect to the chat over in the that thread), and breaking down the variance into identifiable components (eg with PCA). Or bottom up: doing the rounds of expert elicitation from different modelling families, identifying some minimum standards (sufficient conditions) for a “valid” model from each, and building with that.)
Idealistically, these two approaches would kinda meet in the middle in a model confidence set…
I guess the following question would then be for what.
Definitely agree. Perhaps that is a good motivation for why we need a basket. Maybe links back to your post (since I just came across it again) Asking your mum if your idea is good