MCCV

Requirements:

• parameter mccv_parameter_name set to the parameter name of which of loop over.
• parameter mccv_parameter_values set to a list of values. On each iteration, the parameters specified by 'mccv_parameter_name' will be set to one of the values in the list.
• parameter mccv_test_fraction set to a number between (0.0, 1.0) which is the faction of point to hold out from the original dataset for testing.
• parameter mccv_num_trials set to the number of trials to run per parameter value. The larger the number of trial, the more confident the results.

MCCV stands for Monte-Carlo Cross Validation and is a technique designed to increase the confidence in a result based on probabilistic laws. The core ideas is simple; do multiple runs using a different random seed. Your confidence in the results can be quantified using the mean and variance of the set of runs.