Model Tuning#

Hyperparameter tuning is a crucial step in the process of training machine learning models. Examples of hyperparameters include learning rate, batch size, the number of hidden layers in a neural network, and the number of trees in a random forest.

The goal of hyperparameter tuning is to find the optimal set of hyperparameters that maximizes the performance of a model on a given task. The process involves experimenting with different hyperparameter values and assessing their impact on the model’s performance. The performance is typically evaluated using a validation set or through cross-validation.

Modeva integrates several common techniques used in hyperparameter tuning:

  • models.ModelTuneGridSearch: Grid search is a brute-force approach where a predefined set of hyperparameter values is specified, and the model is trained and evaluated for all possible combinations. While comprehensive, it can be computationally expensive. The grid search is implemented based on scikit-learn’s GridSearchCV.

  • models.ModelTuneRandomSearch: Random search involves randomly selecting a combination of hyperparameter values from a predefined range. This method is more efficient than grid search and often yields similar or better results. The random search is implemented based on scikit-learn’s RandomizedSearchCV.

  • models.ModelTunePSO: PSO (Particle Swarm Optimization) search is a population-based optimization algorithm inspired by the social behavior of birds. It is a metaheuristic optimization algorithm that can be used to find the optimal hyperparameters for a model.

ModelTune Usage#

Each of the model tuning methods support several key parameters:

  • param_distributions: This parameter allows you to define the search space through a dictionary. Keys in the dictionary represent parameter names (as strings), and corresponding values are lists of values to be explored. It is usually set as a list of candidate values, e.g., {‘kernel’: [‘linear’, ‘rbf’], ‘alpha’: np.linspace(0, 1, 20)}. For random search, you can also specify distributions using scipy.stats` distributions, e.g., {‘alpha’: scipy.stats.uniform(0, 1)}.

  • metric: Determines the evaluation metrics for model tuning. It can be a single metric or a list of metrics. For regression tasks, options include “MSE,” “MAE,” and “R2.” For classification tasks, metrics including “ACC,” “AUC,” “F1,” “LogLoss,” and “Brier” are available. The default is “MSE” for regression and “ACC” for classification.

  • n_iter: Relevant only when the search method is ‘randomized,’ specifying the number of runs or iterations during the search process.

  • cv: Defines the number of folds for cross-validation. The default is 5.

from scipy.stats import uniform, randint
from modeva.models import ModelTuneRandomSearch
model = mz.get_model("XGB-Depth2")
param_space = {"learning_rate": uniform(0.001, 0.3),
            "n_estimators": randint(100, 1000)}
hpo = ModelTuneRandomSearch(dataset=ds, model=model)
result = hpo.run(param_distributions=param_space,
                n_iter=10,
                metric="MSE",
                cv=5)
result.table

Examples#

On this page