Runtime error during startup since new release

[Jeblii]

Description

Since the new release, we have noticed that a RuntimeError is raised whereas previously it did not. We are running jobs on databricks with Kedro pipeline; some jobs using python wheels and others as a notebook. We initialize the pipeline in the following manner

configure_project(self.package_name)
        with KedroSession.create(
            env: str =self.env,
            conf_source: pathlib.Path =self.conf_source,
            extra_params: Optional[dict] =self.runtime_params,
        ) as session:

Running the pipeline results in the following error:

RuntimeError: Pipeline encountered errors during the run: ["Failed to read the file: /databricks/driver/pyproject.toml.\n[Errno 2] No such file or directory: '/databricks/driver/pyproject.toml'"]<Title>

Expected Result

No error raised. I am not sure of the reason why a pyproject.toml has to be supplied on databricks.

Your Environment

• Kedro version used (pip show kedro or kedro -V): 0.19.7
• Python version used (python -V): 3.9
• Operating system and version: Databricks 12.2 LTS

[merelcht]

Hi @Jeblii, I happened to be testing things on Databricks yesterday, and I noticed the same message in my logs, but it's not actually causing failures for me. Are you sure there isn't another error that's the real issue?

[Jeblii]

My apologies! I wasn't fully clear in my opening message. It indeed does not raise actively, but it seems to log the message with level Error. We noticed that this wasn't the case before the latest release. Is this the expected behaviour?

[merelcht]

@Jeblii , thanks for clarifying! This definitely isn't expected. I've created a ticket to address it: kedro-org/kedro-plugins#804. We'll prioritise to fix this.

[merelcht]

Fixed in kedro-org/kedro-plugins#806