Parallelize hyperparameter tuning with scikit-learn and MLflow
Note
The managed MLflow integration with Databricks on Google Cloud requires Databricks Runtime for Machine Learning 9.1 LTS or above.
This notebook shows how to use Hyperopt to parallelize hyperparameter tuning calculations. It uses the SparkTrials class to automatically distribute calculations across the cluster workers. It also illustrates automated MLflow tracking of Hyperopt runs so you can save the results for later.
After you perform the actions in the last cell in the notebook, your MLflow UI should display:
