Elizabeth Gibney’s Nature article, Could machine learning fuel a reproducibility crisis in science?, is an intriguing exploration about reproducibility in disciplines that use Machine Learning with a particular focus on computational reproducibility.
The challenges of training data from the same period or even including data in both training and evaluation data, or data leakage, are challenging ones. I am more of a software than machine learning person but it does suggest that training and critical thinking in this area is required. It does sound as if there is a deeper move to this.
The article gestures to the looming AI winter and the legitimacy of the results. I seem to recall the coming AI winter before. The second part is more problematic. Machine Learning, in many forms, is embedded into creating the epistemic object, which makes the potential crisis more challenging of results cannot be trusted.
