Improved Docker and cloud support

Author: anders-kiaer

.github/workflows/webviz-config.yml

@@ -74,6 +74,14 @@ jobs:
           webviz docs --portable ./docs_build --skip-open
           webviz schema
 
+      - name: 🐳 Build Docker example image
+        run: |
+          export GIT_POINTER_WEBVIZ_CONFIG=$GITHUB_REF
+          webviz build ./examples/basic_example.yaml --portable ./some_portable_app
+          pushd ./some_portable_app
+          docker build .
+          rm -rf ./some_portable_app
+
       - name: 🚢 Build and deploy Python package
         if: github.event_name == 'release' && matrix.python-version == '3.6'
         env:

CHANGELOG.md

@@ -21,7 +21,22 @@ generated docs when having type hinted return value of plugin `__init__` functio
 
 ## [0.2.3] - 2020-11-26
 
+### Changed
+- [#318](https://github.com/equinor/webviz-config/pull/318) - Ad-hoc plugins not
+supported anymore (i.e. plugins not being part of a Python project with `setup.py`).
+This enables us to reserve the configuration file syntax `prefix.PluginName` to later
+have a way for the user to specify which plugin project `PluginName` should be taken
+from, in cases where multiple plugin projects provide a plugin with the same name.
+Requiring a formal Python project also enables useful features in the framework
+(see the `Added` section for this PR). 
+
 ### Added
+- [#318](https://github.com/equinor/webviz-config/pull/318) - `webviz-config`
+now facilitates automatically including necessary plugin projects as dependencies
+in generated Docker setup. Private repositories are also supported, however the
+Docker build process would then need to be given deploy keys as environment variable
+secrets. First iteration of automatically creating a corresponding
+[Radix config](https://www.radix.equinor.com/) is also added.
 - [#337](https://github.com/equinor/webviz-config/pull/337) - New generic plugin to
 generate a [Pivot table](https://en.wikipedia.org/wiki/Pivot_table) based on
 [Dash Pivottable](https://github.com/plotly/dash-pivottable).

CONTRIBUTING.md

@@ -8,9 +8,10 @@
   - [User provided arguments](#user-provided-arguments)
   - [Data input](#data-input)
     - [Deattaching data from its original source](#deattaching-data-from-its-original-source)
-  - [Custom ad-hoc plugins](#custom-ad-hoc-plugins)
 - [Run tests](#run-tests)
 - [Build documentation](#build-documentation)
+  - [Improve plugin documentation](#improve-plugin-documentation)
+- [Make your plugin project available](#make-your-plugin-project-available)
 
 ## Creating a new plugin
 
@@ -385,7 +386,7 @@ The core of `webviz-config` will do the following:
 2) If the user asks for a portable version, it will
    1) Before writing the actual dash code, it will run all decorated functions
       with the given argument combinations. The resulting dataframes are stored
-      in a folder `./webviz_storage` as parquet files.
+      in a folder `./resources/webviz_storage` as parquet files.
    2) It writes the webviz-dash code (as usual), but this the decorated
       functions will return the dataframe from the stored `.parquet` files,
       instead of running the actual function code.
@@ -457,58 +458,6 @@ signature is not necessary, however if you do you will get a
 instance representing the absolute path to the configuration file that was used, and/or
 a boolean value stating if the Webviz application running is a portable one.
 
-### Custom ad-hoc plugins
-
-It is possible to create custom plugins which still can be included through
-the configuration file, which could be useful for quick prototyping.
-
-As an example, assume someone on your project has made the Python file
-
-```python
-import dash_html_components as html
-from webviz_config import WebvizPluginABC
-
-
-class OurCustomPlugin(WebvizPluginABC):
-
-    def __init__(self, title: str):
-
-        super().__init__()
-
-        self.title = title
-
-    @property
-    def layout(self):
-        return html.Div([
-                         html.H1(self.title),
-                         'This is just some ordinary text'
-                        ])
-```
-
-If this is saved such that it is available through e.g. a
-[module](https://docs.python.org/3/tutorial/modules.html)
-`ourmodule`, the user can include the custom plugin the same way as a standard
-plugin, with the only change of also naming the module:
-```yaml
-title: Simple Webviz example
-
-pages:
-
-  - title: Front page
-    content:
-      - ourmodule.OurCustomPlugin:
-          title: Title of my custom plugin
-```
-
-Note that this might involve appending your `$PYTHONPATH` environment
-variable with the path where your custom module is located. The same principle
-applies if the custom plugin is saved in a package with submodule(s),
-```yaml
-    ...
-      - ourpackage.ourmodule.OurCustomPlugin:
-          ...
-```
-
 ## Run tests
 
 To run tests it is necessary to first install the [selenium chrome driver](https://github.com/SeleniumHQ/selenium/wiki/ChromeDriver).
@@ -584,3 +533,39 @@ $$\alpha = \frac{\beta}{\gamma}$$
 
 Example of auto-built documentation for `webviz-config` can be seen
 [here on github](https://equinor.github.io/webviz-config/).
+
+## Make your plugin project available
+
+It is strongly recommended to store your plugin project code base in a `git` solution,
+e.g. [in a new GitHub repository](https://github.com/new). If possible, it is also
+_highly_ recommended to let it be an open source repository. This has several advantages:
+ - Others can reuse your work - and help each other achieve better code quality, add more features and share maintenance
+ - You will from the beginning make sure you keep a good separation between visualization code and data loading/processing code.
+ - It becomes easier for your plugin users to use it in e.g. Docker and in cloud hosting.
+
+`webviz-config` will make sure that portable builds, which uses one (or more) plugins
+from your plugin project, includes installation instructions for the project also in
+the Docker build instructions. In order for `webviz-config` to do this, you should add
+[`project_urls` metadata](https://packaging.python.org/guides/distributing-packages-using-setuptools/#project-urls)
+in the project's `setup.py`. An example:
+```
+    project_urls={
+        "Download": "https://pypi.org/project/webviz-config",
+        "Source": "https://github.com/equinor/webviz-config",
+    },
+```
+The following logic applies when the user creates a portable application:
+- If `Download` is specified *and* the plugin project version installed is available
+  on PyPI, the Dockerfile will `pip install` the same version from PyPI.
+- Otherwise the Dockerfile will install from the `Source` url. From the `setuptools_scm`
+  provided version, the correct commit/version/tag will be installed.
+
+Note that if you are a developer and working on a fork, you might want to temporarily
+override the `Source` url to your fork. You can do this by specifying an environment
+variable `SOURCE_URL_NAME_PLUGIN_PROJECT=https://urlyourfork`. If you also want to
+explicitly state git pointer/reference (thereby not use the one derived from
+`setuptools_scm` version) you can set the environment variable
+`GIT_POINTER_NAME_PLUGIN_PROJECT`.
+
+For private repositories, a GitHub SSH deploy key will need to be provided to the Docker
+build process (see instructions in `README` created with the portable application).

INTRODUCTION.md

@@ -113,3 +113,19 @@ If you are using Visual Studio Code, we recommend [Red Hat's YAML extension](htt
 }
 ```
 to your `settings.json` file, you will get help from the editor on YAML files following the namepatterns to the right (might have to restart the editor after updating the settings).
+
+#### Deployment
+
+When you have created a portable Webviz application, you are approximately one command away of either creating a new application in cloud (or updating an existing application).
+
+##### Azure
+
+Automatic deployment to Azure Web app service is very much possible, and only a community PR away (most of the machinery and Azure CLI wrappers are already in place through the Radix deployment feature).
+
+##### Radix
+
+[Radix is an open source hosting framework](https://github.com/equinor/radix-platform/) by Equinor. If you execute
+```bash
+webviz deploy radix ./your_portable_app owner/reponame --initial
+```
+Webviz will do the following:

setup.py

@@ -82,20 +82,24 @@ def get_long_description() -> str:
         "pandas>=1.0",
         "pyarrow>=0.16",
         "pyyaml>=5.1",
+        "requests>=2.20",
         "tqdm>=4.8",
         "importlib-metadata>=1.7; python_version<'3.8'",
         "typing-extensions>=3.7; python_version<'3.8'",
         "webviz-core-components>=0.1.0",
     ],
-    tests_require=TESTS_REQUIRES,
-    extras_require={"tests": TESTS_REQUIRES},
+    extras_require={
+        "tests": TESTS_REQUIRES,
+        "deploy": ["azure-cli"],
+    },
     setup_requires=["setuptools_scm~=3.2"],
     python_requires="~=3.6",
     use_scm_version=True,
     zip_safe=False,
     project_urls={
         "Documentation": "https://equinor.github.io/webviz-config",
         "Download": "https://pypi.org/project/webviz-config",
+        "Source": "https://github.com/equinor/webviz-config",
         "Tracker": "https://github.com/equinor/webviz-config/issues",
     },
     classifiers=[

tests/test_plugin_init.py

@@ -30,7 +30,7 @@ def __init__(self, entry_points, name):
 def test_no_warning():
     globals_mock = {}
     with warnings.catch_warnings(record=True) as warn:
-        metadata = load_webviz_plugins_with_metadata(
+        metadata, _ = load_webviz_plugins_with_metadata(
             [dist_mock1, dist_mock3], globals_mock
         )
         assert len(warn) == 0, "Too many warnings"
@@ -43,7 +43,7 @@ def test_no_warning():
 def test_warning_multiple():
     globals_mock = {}
     with warnings.catch_warnings(record=True) as warn:
-        metadata = load_webviz_plugins_with_metadata(
+        metadata, _ = load_webviz_plugins_with_metadata(
             [dist_mock1, dist_mock2], globals_mock
         )
 

tests/test_plugin_metadata.py

@@ -1,13 +1,13 @@
 import webviz_config
-from webviz_config.plugins import metadata
+from webviz_config.plugins import plugin_project_metadata
 
 
 def test_webviz_config_metadata():
-    meta = metadata["BannerImage"]
 
-    assert meta["dist_name"] == "webviz-config"
-    assert meta["dist_version"] == webviz_config.__version__
+    metadata = plugin_project_metadata["webviz-config"]
 
-    assert meta["documentation_url"] == "https://equinor.github.io/webviz-config"
-    assert meta["download_url"] == "https://pypi.org/project/webviz-config"
-    assert meta["tracker_url"] == "https://github.com/equinor/webviz-config/issues"
+    assert metadata["dist_version"] == webviz_config.__version__
+    assert metadata["documentation_url"] == "https://equinor.github.io/webviz-config"
+    assert metadata["download_url"] == "https://pypi.org/project/webviz-config"
+    assert metadata["source_url"] == "https://github.com/equinor/webviz-config"
+    assert metadata["tracker_url"] == "https://github.com/equinor/webviz-config/issues"

webviz_config/_build_webviz.py

@@ -10,13 +10,16 @@
 
 from ._config_parser import ParserError
 from ._write_script import write_script
+from ._dockerize import create_docker_setup
 from .themes import installed_themes
 from .utils import terminal_colors
 
 BUILD_FILENAME = "webviz_app.py"
 STATIC_FOLDER = pathlib.Path(__file__).resolve().parent / "static"
 
 
+# Restructure here before merge.
+# pylint: disable=too-many-branches
 def build_webviz(args: argparse.Namespace) -> None:
 
     if args.theme not in installed_themes:
@@ -28,13 +31,10 @@ def build_webviz(args: argparse.Namespace) -> None:
         build_directory = args.portable.resolve()
         build_directory.mkdir(parents=True)
 
-    shutil.copytree(STATIC_FOLDER / "assets", build_directory / "assets")
-
-    for filename in ["README.md", "Dockerfile", ".dockerignore"]:
-        shutil.copy(STATIC_FOLDER / filename, build_directory)
+    shutil.copytree(STATIC_FOLDER / "assets", build_directory / "resources" / "assets")
 
     for asset in installed_themes[args.theme].assets:
-        shutil.copy(asset, build_directory / "assets")
+        shutil.copy(asset, build_directory / "resources" / "assets")
 
     (build_directory / "theme_settings.json").write_text(
         installed_themes[args.theme].to_json()
@@ -71,14 +71,18 @@ def build_webviz(args: argparse.Namespace) -> None:
                 f"{terminal_colors.END}"
             )
 
-        non_default_assets = write_script(
+        non_default_assets, plugin_metadata = write_script(
             args, build_directory, "webviz_template.py.jinja2", BUILD_FILENAME
         )
 
         for asset in non_default_assets:
-            shutil.copy(asset, build_directory / "assets")
+            shutil.copy(asset, build_directory / "resources" / "assets")
 
-        if not args.portable:
+        if args.portable:
+            for filename in ["README.md", ".dockerignore", ".gitignore"]:
+                shutil.copy(STATIC_FOLDER / filename, build_directory)
+            create_docker_setup(build_directory, plugin_metadata)
+        else:
             run_webviz(args, build_directory)
 
     finally:

webviz_config/_config_parser.py

@@ -2,11 +2,11 @@
 import sys
 import pathlib
 import inspect
-import typing
+from typing import Dict, List, Optional
 
 import yaml
 
-from . import plugins as standard_plugins
+import webviz_config.plugins
 from .utils import terminal_colors
 from .utils._get_webviz_plugins import _get_webviz_plugins
 
@@ -17,7 +17,7 @@ def _call_signature(
     plugin_name: str,
     kwargs: dict,
     config_folder: pathlib.Path,
-    contact_person: typing.Optional[dict] = None,
+    contact_person: Optional[dict] = None,
 ) -> tuple:
     # pylint: disable=too-many-branches,too-many-statements
     """Takes as input the name of a plugin together with user given arguments
@@ -31,7 +31,9 @@ def _call_signature(
       * If there is type mismatch between user given argument value, and type
         hint in __init__ signature (given that type hint exist)
     """
-    argspec = inspect.getfullargspec(getattr(standard_plugins, plugin_name).__init__)
+    argspec = inspect.getfullargspec(
+        getattr(webviz_config.plugins, plugin_name).__init__
+    )
 
     if argspec.defaults is not None:
         required_args = argspec.args[: -len(argspec.defaults)]
@@ -90,7 +92,7 @@ def _call_signature(
 
             if expected_type == pathlib.Path:
                 kwargs[arg] = (config_folder / pathlib.Path(kwargs[arg])).resolve()
-            elif expected_type == typing.List[pathlib.Path]:
+            elif expected_type == List[pathlib.Path]:
                 kwargs[arg] = [
                     (config_folder / pathlib.Path(patharg)).resolve()
                     for patharg in kwargs[arg]
@@ -126,7 +128,9 @@ class ParserError(Exception):
 
 class ConfigParser:
 
-    STANDARD_PLUGINS = [name for (name, _) in _get_webviz_plugins(standard_plugins)]
+    STANDARD_PLUGINS = [
+        name for (name, _) in _get_webviz_plugins(webviz_config.plugins)
+    ]
 
     def __init__(self, yaml_file: pathlib.Path):
 
@@ -151,8 +155,9 @@ def __init__(self, yaml_file: pathlib.Path):
             ).with_traceback(sys.exc_info()[2])
 
         self._config_folder = pathlib.Path(yaml_file).parent
-        self._page_ids: typing.List[str] = []
+        self._page_ids: List[str] = []
         self._assets: set = set()
+        self._plugin_metadata: Dict[str, dict] = {}
         self._used_plugin_packages: set = set()
         self.clean_configuration()
 
@@ -281,7 +286,10 @@ def clean_configuration(self) -> None:
                     self._config_folder,
                 )
 
-                self.assets.update(getattr(standard_plugins, plugin_name).ASSETS)
+                self._assets.update(getattr(webviz_config.plugins, plugin_name).ASSETS)
+                self._plugin_metadata[
+                    plugin_name
+                ] = webviz_config.plugins.plugin_metadata[plugin_name]
 
     @property
     def configuration(self) -> dict:
@@ -294,3 +302,10 @@ def shared_settings(self) -> dict:
     @property
     def assets(self) -> set:
         return self._assets
+
+    @property
+    def plugin_metadata(self) -> Dict[str, dict]:
+        """Returns a dictionary of plugin metadata, and only for
+        plugins included in the configuration file.
+        """
+        return self._plugin_metadata

webviz_config/_deployment/__init__.py

@@ -0,0 +1 @@
+from .radix import radix_deployment