Exploring Best Test Size, Number of Folds, and Repeated Hold-Out

Study gains in guaranteed, supported, and mean accuracy; discover biases

Photo by Elena Mozhvilo on Unsplash

While people say between 10% and 30% is a good test size, it’s very problem-dependent. A good starting point is what I call the guaranteed accuracy — accuracy minus two standard deviations. Using a representative, but not fine-tuned…