Skip to content

Commit 7306df7

Browse files
LGroLGro
authored
add basic docs for optimizers and loops (#125)
* add basic docs for optimizers and loops * add badges --------- Signed-off-by: Grossberger Lukas (CR/AIR2.2) <Lukas.Grossberger@de.bosch.com>
1 parent ebbb2a3 commit 7306df7

14 files changed

+151
-60
lines changed

‎CONTRIBUTING.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ request model. For large contributions we do encourage you to file a ticket in
55
the GitHub issues tracking system prior to any code development to coordinate
66
with the blackboxopt development team early in the process. Coordinating up
77
front helps to avoid frustration later on.
8-
Please follow the conventiones outlined by the pre-commit hooks mentioned in the
8+
Please follow the conventions outlined by the pre-commit hooks mentioned in the
99
repository README and add tests for the functionality you would like to contribute.
1010

1111
Your contribution must be licensed under the Apache-2.0 license, the license

‎README.md

+3-14
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22

33
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
44
[![CI/CD](https://github.com/boschresearch/blackboxopt/workflows/ci-cd-pipeline/badge.svg)](https://github.com/boschresearch/blackboxopt/actions?query=workflow%3Aci-cd-pipeline+branch%3Amain)
5+
[![PyPI - Wheel](https://img.shields.io/pypi/wheel/blackboxopt)](https://pypi.org/project/blackboxopt/)
6+
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/blackboxopt)](https://pypi.org/project/blackboxopt/)
7+
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
58

69
Various blackbox optimization algorithms with a common interface along with useful
710
helpers like parallel optimization loops, analysis and visualization scripts.
@@ -64,20 +67,6 @@ For HTML test coverage reports run
6467
poetry run pytest tests/ --cov --cov-report html:htmlcov
6568
```
6669

67-
### Custom Optimizers
68-
69-
When you develop an optimizer based on the interface defined as part of
70-
`blackboxopt.base`, you can use `blackboxopt.testing` to directly test whether your
71-
implementation follows the specification by adding a test like this to your test suite.
72-
73-
```python
74-
from blackboxopt.testing import ALL_REFERENCE_TESTS
75-
76-
@pytest.mark.parametrize("reference_test", ALL_REFERENCE_TESTS)
77-
def test_all_reference_tests(reference_test):
78-
reference_test(CustomOptimizer, custom_optimizer_init_kwargs)
79-
```
80-
8170
## Building Documentation
8271

8372
Make sure to install _all_ necessary dependencies:

‎blackboxopt/__init__.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
__version__ = "5.0.3"
1+
__version__ = "5.0.4"
22

33
from parameterspace import ParameterSpace
44

‎docs/index.md

+3-43
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22

33
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
44
[![CI/CD](https://github.com/boschresearch/blackboxopt/workflows/ci-cd-pipeline/badge.svg)](https://github.com/boschresearch/blackboxopt/actions?query=workflow%3Aci-cd-pipeline+branch%3Amain)
5+
[![PyPI - Wheel](https://img.shields.io/pypi/wheel/blackboxopt)](https://pypi.org/project/blackboxopt/)
6+
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/blackboxopt)](https://pypi.org/project/blackboxopt/)
7+
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
58

69
The `blackboxopt` Python package contains blackbox optimization algorithms with a common
710
interface, along with useful helpers like parallel optimization loops, analysis and
@@ -26,49 +29,6 @@ implementation.
2629
BOHB is provided as a cleaner replacement of the former implementation in
2730
[HpBandSter](https://github.com/automl/HpBandSter).
2831

29-
#### Fidelities for BOHB & Hyperband
30-
31-
You can calculate the fidelity schedule resulting from these parameters:
32-
33-
<script>
34-
function calculateFidelitiesBOHB() {
35-
const min_fidelity = document.getElementById('minFidelityBOHB').value;
36-
const max_fidelity = document.getElementById('maxFidelityBOHB').value;
37-
const eta = document.getElementById('etaBOHB').value;
38-
39-
const max_num_stages = 1 + Math.floor(
40-
Math.log(max_fidelity / min_fidelity) / Math.log(eta)
41-
);
42-
const num_configs_first_stage = Math.ceil(Math.pow(eta, max_num_stages - 1));
43-
const num_configs_per_stage = Array.from({ length: max_num_stages }, (_, i) =>
44-
Math.floor(num_configs_first_stage / Math.pow(eta, i))
45-
);
46-
const fidelities_per_stage = Array.from({ length: max_num_stages }, (_, i) =>
47-
max_fidelity / Math.pow(eta, max_num_stages - 1 - i)
48-
);
49-
50-
document.getElementById('fidelitiesBOHB').innerHTML = `Fidelities: ${fidelities_per_stage}`;
51-
}
52-
</script>
53-
<table>
54-
<tr>
55-
<td>min_fidelity</td>
56-
<td><input type="text" id="minFidelityBOHB"></td>
57-
</tr>
58-
<tr>
59-
<td>max_fidelity</td>
60-
<td><input type="text" id="maxFidelityBOHB"></td>
61-
</tr>
62-
<tr>
63-
<td>eta</td>
64-
<td><input type="text" id="etaBOHB"></td>
65-
</tr>
66-
<tr>
67-
<td></td><td><button onclick="calculateFidelitiesBOHB();">Submit</button></td>
68-
</tr>
69-
</table>
70-
<p id="fidelitiesBOHB"></p>
71-
7232
### Optimization Loops
7333

7434
As part of the `blackboxopt.optimization_loops` module compatible implementations for

‎docs/optimization-loops/dask-distributed.md

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Dask Distributed Optimization Loop
2+
3+
In case you are working with [dask](https://github.com/dask/dask/), this optimization
4+
loop can help you run `blackboxopt` based optimization leveraging your dask cluster.
5+
See also the corresponding [example](../../examples/dask-distributed) for more
6+
details.
7+
8+
::: blackboxopt.optimization_loops.dask_distributed

‎docs/optimization-loops/overview.md

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Optimization Loops
2+
3+
We include a handful optimization loop implementations for different scenarios from a
4+
simple [sequential](sequential.md) loop to one [distributed](dask-distributed.md)
5+
potentially across nodes in a cluster setup.
6+
Additionally, a set of [reference tests](testing.md) are included for
7+
anyone extending the selection of optimization loops as part of a contribution to
8+
`blackboxopt` or in a separate project.

‎docs/optimization-loops/sequential.md

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Sequential Optimization Loop
2+
3+
::: blackboxopt.optimization_loops.sequential

‎docs/optimization-loops/testing.md

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Reference Tests
2+
3+
To test an optimization loop implementation across various reference scenarios, follow:
4+
5+
```python
6+
import pytest
7+
from blackboxopt.optimization_loops.testing import ALL_REFERENCE_TESTS
8+
9+
@pytest.mark.parametrize("reference_test", testing.ALL_REFERENCE_TESTS)
10+
def test_all_reference_tests(reference_test):
11+
reference_test(custom_optimization_loop, {"opt_loop_specific_kwarg": 123})
12+
```
13+
14+
where you can include custom keyword arguments that are passed to the optimization loop
15+
calls in the reference tests.

‎docs/optimizers/bohb.md

+60
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# BOHB Optimizer
2+
3+
BOHB performs robust and efficient hyperparameter optimization at scale by combining the
4+
speed of Hyperband searches with the guidance and guarantees of convergence of Bayesian
5+
Optimization.
6+
Instead of sampling new configurations at random, BOHB uses kernel density estimators to
7+
select promising candidates.
8+
9+
This implementation is meant to supersede the initial release of
10+
[HpBandSter](https://github.com/automl/HpBandSter/).
11+
12+
13+
## Fidelities
14+
15+
Here you can calculate the fidelity schedule resulting from BOHB's hyper-parameters:
16+
17+
<script>
18+
function calculateFidelitiesBOHB() {
19+
const min_fidelity = document.getElementById('minFidelityBOHB').value;
20+
const max_fidelity = document.getElementById('maxFidelityBOHB').value;
21+
const eta = document.getElementById('etaBOHB').value;
22+
23+
const max_num_stages = 1 + Math.floor(
24+
Math.log(max_fidelity / min_fidelity) / Math.log(eta)
25+
);
26+
const num_configs_first_stage = Math.ceil(Math.pow(eta, max_num_stages - 1));
27+
const num_configs_per_stage = Array.from({ length: max_num_stages }, (_, i) =>
28+
Math.floor(num_configs_first_stage / Math.pow(eta, i))
29+
);
30+
const fidelities_per_stage = Array.from({ length: max_num_stages }, (_, i) =>
31+
max_fidelity / Math.pow(eta, max_num_stages - 1 - i)
32+
);
33+
34+
document.getElementById('fidelitiesBOHB').innerHTML = `Fidelities: ${fidelities_per_stage}`;
35+
}
36+
</script>
37+
<table>
38+
<tr>
39+
<td>min_fidelity</td>
40+
<td><input type="text" id="minFidelityBOHB"></td>
41+
</tr>
42+
<tr>
43+
<td>max_fidelity</td>
44+
<td><input type="text" id="maxFidelityBOHB"></td>
45+
</tr>
46+
<tr>
47+
<td>eta</td>
48+
<td><input type="text" id="etaBOHB"></td>
49+
</tr>
50+
<tr>
51+
<td></td>
52+
<td><button onclick="calculateFidelitiesBOHB();">Calculate</button></td>
53+
</tr>
54+
</table>
55+
<p id="fidelitiesBOHB"></p>
56+
57+
58+
## Reference
59+
60+
::: blackboxopt.optimizers.bohb

‎docs/optimizers/botorch.md

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# BoTorch Base Optimizer
2+
3+
This optimizer is a basic Gaussian Process based Bayesian optimization implementation
4+
leveraging BoTorch in a way that is compatible with the `blackboxopt` interface.
5+
While this is a functional optimizer, it is more intended as a basis for other BoTorch
6+
based optimizer implementations.
7+
8+
## Reference
9+
10+
::: blackboxopt.optimizers.botorch_base

‎docs/optimizers/space-filling.md

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Space Filling Optimizer
2+
3+
The `SpaceFilling` optimizer is a
4+
[Sobol sequence](https://en.wikipedia.org/wiki/Sobol_sequence) based optimizer that
5+
covers the search space based on a quasi-random low-discrepancy sequence.
6+
This strategy requires a larger budget for evaluations but can be a good initial
7+
approach to get to know the optimization problem at hand.
8+
While this implementation follows the overall interface including the specification and
9+
reporting of objectives and their values, the actual objective values are
10+
inconsequential for the underlying Sobol sequence and do not guide the optimization.
11+
12+
## Reference
13+
14+
::: blackboxopt.optimizers.space_filling

‎docs/optimizers/testing.md

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Reference Tests
2+
3+
When you develop an optimizer based on the interface defined as part of
4+
`blackboxopt.base`, you can use `blackboxopt.testing` to directly test whether your
5+
implementation follows the specification by adding a test like this to your test suite:
6+
7+
```python
8+
import pytest
9+
from blackboxopt.testing import ALL_REFERENCE_TESTS
10+
11+
@pytest.mark.parametrize("reference_test", ALL_REFERENCE_TESTS)
12+
def test_all_reference_tests(reference_test):
13+
reference_test(CustomOptimizer, optional_optimizer_init_kwargs)
14+
```

‎mkdocs.yml

+10
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,16 @@ nav:
3131
- Overview: examples/overview.md
3232
- examples/dask-distributed.md
3333
- examples/multi-objective-multi-param.md
34+
- Optimizers:
35+
- Space Filling: optimizers/space-filling.md
36+
- BOHB: optimizers/bohb.md
37+
- BoTorch: optimizers/botorch.md
38+
- optimizers/testing.md
39+
- Optimization Loops:
40+
- Overview: optimization-loops/overview.md
41+
- Sequential: optimization-loops/sequential.md
42+
- Dask Distributed: optimization-loops/dask-distributed.md
43+
- optimization-loops/testing.md
3444
- ...
3545

3646
plugins:

‎pyproject.toml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[tool.poetry]
22
name = "blackboxopt"
3-
version = "5.0.3"
3+
version = "5.0.4"
44
description = "A common interface for blackbox optimization algorithms along with useful helpers like parallel optimization loops, analysis and visualization scripts."
55
readme = "README.md"
66
repository = "https://github.com/boschresearch/blackboxopt"

0 commit comments

Comments
 (0)