Issue with local load_profile and %verdi magic

[chrisjsewell]

Locally running the docs build, and thus the tutorial notebook execution, there is now (I don't believe this was originally the case) an issue; that the Config does not appear to be "loaded" with the profile, and so verdi tries to load a local default profile.

nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
%verdi node show 1
------------------

---------------------------------------------------------------------------
InvalidOperation                          Traceback (most recent call last)
/var/folders/t2/xbl15_3n4tsb1vr_ccmmtmbr0000gn/T/ipykernel_79897/2387781979.py in <module>
----> 1 get_ipython().run_line_magic('verdi', 'node show 1')

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2362                 kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2363             with self.builtin_trap:
-> 2364                 result = fn(*args, **kwargs)
   2365             return result
   2366 

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/decorator.py in fun(*args, **kw)
    230             if not kwsyntax:
    231                 args, kw = fix(args, kw, sig)
--> 232             return caller(func, *(extras + args), **kw)
    233     fun.__name__ = func.__name__
    234     fun.__doc__ = func.__doc__

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

~/Documents/GitHub/aiida_core_develop/aiida/tools/ipython/ipython_magics.py in verdi(self, line, local_ns)
     74         profile = get_profile()
     75         obj = AttributeDict({'config': config, 'profile': profile})
---> 76         return verdi(shlex.split(line), prog_name='%verdi', obj=obj, standalone_mode=False)  # pylint: disable=too-many-function-args,unexpected-keyword-arg
     77 
     78     @magic.needs_local_scope

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/click/core.py in __call__(self, *args, **kwargs)
   1128     def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
   1129         """Alias for :meth:`main`."""
-> 1130         return self.main(*args, **kwargs)
   1131 
   1132 

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/click/core.py in main(self, args, prog_name, complete_var, standalone_mode, windows_expand_args, **extra)
   1052         try:
   1053             try:
-> 1054                 with self.make_context(prog_name, args, **extra) as ctx:
   1055                     rv = self.invoke(ctx)
   1056                     if not standalone_mode:

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/click/core.py in make_context(self, info_name, args, parent, **extra)
    918 
    919         with ctx.scope(cleanup=False):
--> 920             self.parse_args(ctx, args)
    921         return ctx
    922 

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/click/core.py in parse_args(self, ctx, args)
   1611             ctx.exit()
   1612 
-> 1613         rest = super().parse_args(ctx, args)
   1614 
   1615         if self.chain:

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/click/core.py in parse_args(self, ctx, args)
   1376 
   1377         for param in iter_params_for_processing(param_order, self.get_params(ctx)):
-> 1378             value, args = param.handle_parse_result(ctx, opts, args)
   1379 
   1380         if args and not ctx.allow_extra_args and not ctx.resilient_parsing:

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/click/core.py in handle_parse_result(self, ctx, opts, args)
   2332 
   2333             try:
-> 2334                 value = self.process_value(ctx, value)
   2335             except Exception:
   2336                 if not ctx.resilient_parsing:

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/click/core.py in process_value(self, ctx, value)
   2288 
   2289     def process_value(self, ctx: Context, value: t.Any) -> t.Any:
-> 2290         value = self.type_cast_value(ctx, value)
   2291 
   2292         if self.required and self.value_is_missing(value):

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/click/core.py in type_cast_value(self, ctx, value)
   2276             return tuple(convert(x) for x in check_iter(value))
   2277 
-> 2278         return convert(value)
   2279 
   2280     def value_is_missing(self, value: t.Any) -> bool:

~/Documents/GitHub/aiida_core_develop/.tox/py38-docs-clean/lib/python3.8/site-packages/click/types.py in __call__(self, value, param, ctx)
     80     ) -> t.Any:
     81         if value is not None:
---> 82             return self.convert(value, param, ctx)
     83 
     84     def get_metavar(self, param: "Parameter") -> t.Optional[str]:

~/Documents/GitHub/aiida_core_develop/aiida/cmdline/params/types/profile.py in convert(self, value, param, ctx)
     68 
     69         if self._load_profile:
---> 70             load_profile(profile.name)
     71 
     72         ctx.obj.profile = profile

~/Documents/GitHub/aiida_core_develop/aiida/manage/configuration/__init__.py in load_profile(profile, allow_switch)
    151     """
    152     from aiida.manage import get_manager
--> 153     return get_manager().load_profile(profile, allow_switch)
    154 
    155 

~/Documents/GitHub/aiida_core_develop/aiida/manage/manager.py in load_profile(self, profile, allow_switch)
    124 
    125         if self._profile and self.profile_storage_loaded and not allow_switch:
--> 126             raise InvalidOperation(
    127                 f'cannot switch to profile {profile.name!r} because profile {self._profile.name!r} storage '
    128                 'is already loaded and allow_switch is False'

InvalidOperation: cannot switch to profile 'quicksetup' because profile 'myprofile' storage is already loaded and allow_switch is False

[Deepanshu7777777]

hey @chrisjsewell It seems that the verdi command is trying to load a default profile but it cannot find any profiles in the ~/.aiida/config.json file. One solution could be to create a new profile using the verdi profile command and set it as the default profile. Here are the steps:

javascript :
verdi profile new

php:
verdi profile setdefault

[chrisjsewell]

Heya @Deepanshu7777777 %verdi is designed to use the profile already loaded in the ipython kernel. It should probably just except, if none has been loaded

[sphuber]

Just to mention that I had a related issue in #5940 . The tests were running fine, but the RTD build was failing, because the tutorial notebook was excepting in the verdi process list call. The exception was:

---------------------------------------------------------------------------
ProfileConfigurationError                 Traceback (most recent call last)
Cell In[11], line 1
----> 1 get_ipython().run_line_magic('verdi', 'process list')

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/IPython/core/interactiveshell.py:2414, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2412     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2413 with self.builtin_trap:
-> 2414     result = fn(*args, **kwargs)
   2416 # The code below prevents the output from being displayed
   2417 # when using magics with decodator @output_can_be_silenced
   2418 # when the last Python token in the expression is a ';'.
   2419 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/aiida/tools/ipython/ipython_magics.py:76, in AiiDALoaderMagics.verdi(self, line, local_ns)
     74 profile = get_profile()
     75 obj = AttributeDict({'config': config, 'profile': profile})
---> 76 return verdi(shlex.split(line), prog_name='%verdi', obj=obj, standalone_mode=False)

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/click/core.py:1130, in BaseCommand.__call__(self, *args, **kwargs)
   1128 def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
   1129     """Alias for :meth:`main`."""
-> 1130     return self.main(*args, **kwargs)

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/click/core.py:1055, in BaseCommand.main(self, args, prog_name, complete_var, standalone_mode, windows_expand_args, **extra)
   1053 try:
   1054     with self.make_context(prog_name, args, **extra) as ctx:
-> 1055         rv = self.invoke(ctx)
   1056         if not standalone_mode:
   1057             return rv

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/click/core.py:1657, in MultiCommand.invoke(self, ctx)
   1655         sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
   1656         with sub_ctx:
-> 1657             return _process_result(sub_ctx.command.invoke(sub_ctx))
   1659 # In chain mode we create the contexts step by step, but after the
   1660 # base command has been invoked.  Because at that point we do not
   1661 # know the subcommands yet, the invoked subcommand attribute is
   1662 # set to ``*`` to inform the command that subcommands are executed
   1663 # but nothing else.
   1664 with ctx:

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/click/core.py:1657, in MultiCommand.invoke(self, ctx)
   1655         sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
   1656         with sub_ctx:
-> 1657             return _process_result(sub_ctx.command.invoke(sub_ctx))
   1659 # In chain mode we create the contexts step by step, but after the
   1660 # base command has been invoked.  Because at that point we do not
   1661 # know the subcommands yet, the invoked subcommand attribute is
   1662 # set to ``*`` to inform the command that subcommands are executed
   1663 # but nothing else.
   1664 with ctx:

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/click/core.py:1404, in Command.invoke(self, ctx)
   1401     echo(style(message, fg="red"), err=True)
   1403 if self.callback is not None:
-> 1404     return ctx.invoke(self.callback, **ctx.params)

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/click/core.py:760, in Context.invoke(_Context__self, _Context__callback, *args, **kwargs)
    758 with augment_usage_errors(__self):
    759     with ctx:
--> 760         return __callback(*args, **kwargs)

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/aiida/cmdline/utils/decorators.py:73, in with_dbenv.<locals>.wrapper(wrapped, _, args, kwargs)
     70 except (IntegrityError, ConfigurationError, LockedProfileError) as exception:
     71     echo.echo_critical(str(exception))
---> 73 return wrapped(*args, **kwargs)

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/aiida/cmdline/commands/cmd_process.py:97, in process_list(all_entries, group, process_state, process_label, paused, exit_status, failed, past_days, limit, project, raw, order_by, order_dir)
     94 echo.echo(f'\nTotal results: {len(projected)}\n')
     95 print_last_process_state_change()
---> 97 if not get_daemon_client().is_daemon_running:
     98     echo.echo_warning('the daemon is not running', bold=True)
     99 else:
    100     # Second query to get active process count
    101     # Currently this is slow but will be fixed with issue #2770
    102     # We place it at the end so that the user can Ctrl+C after getting the process table.

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/aiida/engine/daemon/client.py:80, in get_daemon_client(profile_name)
     71 """Return the daemon client for the given profile or the current profile if not specified.
     72 
     73 :param profile_name: Optional profile name to load.
   (...)
     77 :raises aiida.common.ProfileConfigurationError: if the given profile does not exist.
     78 """
     79 profile = get_manager().load_profile(profile_name)
---> 80 return DaemonClient(profile)

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/aiida/engine/daemon/client.py:98, in DaemonClient.__init__(self, profile)
     96 self._profile = profile
     97 self._socket_directory: str | None = None
---> 98 self._daemon_timeout: int = config.get_option('daemon.timeout', scope=profile.name)

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/aiida/manage/configuration/config.py:457, in Config.get_option(self, option_name, scope, default)
    454 default_value = option.default if default and option.default is not NO_DEFAULT else None
    456 if scope is not None:
--> 457     value = self.get_profile(scope).get_option(option.name, default_value)
    458 else:
    459     value = self.options.get(option.name, default_value)

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/aiida/manage/configuration/config.py:309, in Config.get_profile(self, name)
    306 if not name:
    307     name = self.default_profile_name
--> 309 self.validate_profile(name)
    311 return self._profiles[name]

File ~/checkouts/readthedocs.org/user_builds/aiida-core/envs/5940/lib/python3.8/site-packages/aiida/manage/configuration/config.py:291, in Config.validate_profile(self, name)
    288 from aiida.common import exceptions
    290 if name not in self.profile_names:
--> 291     raise exceptions.ProfileConfigurationError(f'profile `{name}` does not exist')

ProfileConfigurationError: profile `myprofile` does not exist

The offending line that was introduced by the PR was

self._daemon_timeout: int = config.get_option('daemon.timeout', scope=profile.name)

The original code:

self._daemon_timeout: int = config.get_option('daemon.timeout')

ran without issues. It is not clear to me why the Config.validate_profile would except but the rest of the code would run fine, which would suggest that a profile is configured and exists.

Maybe this can help in finding the cause for the build failing locally but not on ReadTheDocs.

[sphuber]

Found the problem. The notebook creates a temporary profile on the fly and loads it. However, it is not actually added to the Config. So as soon as there is code that validates the existence of the profile, as Config.validate_profile does, it will fail, because the loaded profile is not actual part of the config. The problem in the example I posted above then can be solved by making sure that the temporary profile is properly added to the config, as it would have been when created using verdi quicksetup/setup.

The cause of the OP of this issue is very similar. Except there, when %verdi node show is called, the real causes is that, in invoking verdi, the --profile option is used with the default (which is set to the default profile, which is not the one created temporarily in the notebook) and so ProfileParamType is called which will attempt to load the profile. But since this is different from the already loaded profile, the InvalidOperation is raised. The solution of the previous example won't help here. Even if the temporary profile is part of the config, it will still try to load the default profile. So here, the notebook should either make the temp profile the default as well, or we change the %verdi magic to always call the subcommand of verdi and thereby circumventing the ProfileParamType.convert call entirely.