This was interesting. While watching the video, I thought may be you want to restrict models as tools for explanations by the scientists for the scientists. But then you also recommend Lake et al. (2017) which goes ahead and discusses meta-modeling as a stepping stone to humanlike intelligent machines - or models as tools for explanations by the machines for the machines.
1. I tend to love both mathematical elegance as well as software elegance. On the latter, may be our views differ; mine being that you can have software elegance without mathematical elegance, and mathematical elegance is much easier to convey and communicate through human language than software elegance. I suspect that mathematical principles are easier to spot in isolated phenomena. However, putting them together into a whole seems to involve principles that might be hard to capture in familiar mathematics; yet the same principles might be appreciable (and perhaps even implicitly) by software developers or programming language developers. The closest day-to-day analogy might be that actually learning to drive a bicycle / dance / any procedural skill is easier than figuring out the principles involved. This is a nitpick; I agree with the overall idea of trying to keep things simple and using rigor to derive explanations and predictions.
2. In the context of the Characters Challenge, but also more generally, I had come across the argument that humans learn fast because they employ match-based learning rather than the error-based learning that traditional NNs employ. And there are other arguments that come up in the context of Adaptive Resonance Theory; I wonder what are your thoughts on this theory.
Lotta's point about the similarity between the phenomena in causal judgement manipulation and the weights in Rescorla-Wagner also interests me.