Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mlos_bench: grid search support #688

Closed
bpkroth opened this issue Feb 23, 2024 · 1 comment · Fixed by #690
Closed

mlos_bench: grid search support #688

bpkroth opened this issue Feb 23, 2024 · 1 comment · Fixed by #690

Comments

@bpkroth
Copy link
Contributor

bpkroth commented Feb 23, 2024

In benchmarking only mode, sometimes we actually do want to use grid search to evaluate all possibilities for comparison sake.

This can also be useful when comparing more efficient optimizer search strategy performance as well.

We should be able to implement this as an additional mlos_bench optimizer in combination with the storage backend to check which ones have already been evaluated.
@bpkroth
Copy link
Contributor Author

bpkroth commented Feb 23, 2024

This btw would be a great place for parallel trial execution (#380)

@bpkroth bpkroth mentioned this issue Feb 26, 2024
Merged
@motus motus closed this as completed in #690 Mar 7, 2024
motus added a commit that referenced this issue Mar 7, 2024
@bpkroth @motus
Add GridSearch Optimizer support (#690)
e881987 
Closes #688

- Introduces `GridSearchOptimizer` to `mlos_bench`
- Generates and stores a set of `tuple(dict.values())` from
`ConfigSpace` to track elements of the config grid to search
- If `max_iterations` > `len(grid)` can refill the grid if desired
(e.g., by calling `suggest()` after `not_converged()` returns `False`.
- If `max_iterations` < `len(grid)` (i.e., we don't have enough
iterations to complete the grid) will issue a warning.

---------

Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add GridSearch Optimizer support bpkroth/MLOS
1 participant
@bpkroth