[FEATURE] Hyperband

[colligant]

Hi! I was wondering if the Hyperband hyperparameter algorithm is something you want implemented.

I'm willing to spend some time working on it if there's interest.

[RobertTLange]

Hi Thomas,
thanks for reaching out 🤗 Yes, I would definitely be interested in adding an implementation.

Right now mle-hyperopt focuses on the sequential/batch + full evaluation setting. I shied away from HyperBand for the first version due to it being a successive halving algorithm and thereby restricted to iterative learning paradigms. In principle though it should not be too hard to implement ask_search/tell_search methods such that they perform the ranking and pruning of evaluated configurations. The checkpoint maintenance (interrupting training runs, reloading checkpoints, etc.) will ultimately have to be handled by the user. Although we have to think about how to store the intermediate results. It could potentially make sense to add an auxiliary parameters num_train_steps and/or to keep the checkpoint path around in the parameter configurations.

What do you think? Do you want to give it a shot? I should have some time in the next weeks to actively support you.
Have a nice evening,
Rob

[RobertTLange]

@colligant - let me know if are you still interested in a working together on an implementation? If not (no worries), I would work on it during the Christmas season.

[colligant]

Hey! Sorry about not getting back to you. I ended up deciding that trying to integrate hyperband into this package would take up too much of my work time, so I wrote a small python package to tune hyperparameters using the slurm workload manager (https://github.com/TravisWheelerLab/shopty). I should have a few days in mid January to work on this, but no worries if you want to get started sooner.

[RobertTLange]

Hi! Sorry for me taking so long. No worries. shopty looks very cool! I have written something similar in the form of the mle-scheduler. It supports local, Slurm, Grid Engine, SSH server and GCP job management and is meant to integrate nicely with the other pieces of my little ecosystem. Maybe you have some recommendations! Seems like we are interested in the same type of ML tools 😃

@ hyperband implementation: I just added 3 new search classes: HalvingSearch, HyperbandSearch and PBTSearch which follow all the same simple ask/tell API. Here is an example Hyperband instantiation:

from mle_hyperopt import HyperbandSearch

strategy = HyperbandSearch(real={"lrate": {"begin": 0.1,
                                        "end": 0.5,
                                        "prior": "uniform"}},
                           integer={"batch_size": {"begin": 1,
                                                   "end": 5,
                                                   "prior": "log-uniform"}},
                           categorical={"arch": ["mlp", "cnn"]},
                           search_config={"max_resource": 27,
                                          "eta": 3},
                           seed_id=42,
                           verbose=True)

configs = strategy.ask()
# Get scores and a list of ckpts for later train continuation
...
strategy.tell(configs, scores, ckpts)

You can find more info at the end of the colab notebook and in this example script. I will add some more explanatory text in the following days. Let me know what you think!

[RobertTLange]

@colligant - this is now featured in v0.0.5 Closing the issue for now. Feel free to reopen if you find a bug/something is missing.