Multiple files pipeline

[Panicum]

Hello,

I am creating larger RAG pipeline, with multiple classes and I would like to split them into multiple code files for better readability and also for future other pipelines that will use some of the same classes (like connection to DB etc.).

But I can't upload such pipeline to open-webui. I tired to split my pipeline into files with each class, save them into one directory in /pipelines/my_big_pipeline/ openwebui docker image and upload main pipeline file with Pipeline class by openweb-ui interface or by adding it to /pipelines directory with imports from /pipelines/my_big_pipeline/ classes. But it isn't working, I am keep getting error:

WARNING:root:No Pipeline class found in main_pipeline
Error loading module: main_pipeline

Anyone was trying something like this and knows the solution to this problem?

[TheWasilingMan]

I have been trying for a long time to get around this error when having a pipeline with more than one file. I have been unable to figure it out. Have you had any luck figuring out an alternative?

[lpadula]

Hello,

I have the same problem and would like to know how to resolve it. Would it be necessary to submit a feature request to the repository https://github.com/open-webui/open-webuito address this issue?

[vladislavdonchev]

Hello,

I have the same problem and would like to know how to resolve it. Would it be necessary to submit a feature request to the repository https://github.com/open-webui/open-webuito address this issue?

The below code is confirmed and working for automatically downloading, deploying and importing additional source files:

def download_latest_code():
    response = requests.get("https://drive.google.com/uc?export=download&id=YOUR_FILE_ID",
                            headers={}, stream=True)
    response.raise_for_status()

    with open("pipelines-integration-latest.zip", 'wb') as file:
        for chunk in response.iter_content(chunk_size=8192):
            if chunk:
                file.write(chunk)
    print(f"Downloaded latest version.")


def deploy_latest_code():
    with zipfile.ZipFile("pipelines-integration-latest.zip", 'r') as zip_ref:
        deployment_path= os.path.join(os.path.dirname(os.curdir), 'my_pipeline')
        zip_ref.extractall(deployment_path)
    pip(['install', '-r', os.path.join(deployment_path, 'requirements.txt')])

    current_dir = os.getcwd()
    for root, _, files in os.walk(deployment_path):
        for file in files:
            # Special file handling.
            pass

    print(f"Deployed latest version.")


if os.getenv("DEVELOPMENT") is None:
    download_latest_code()
    # TODO Only if there is a change.
    deploy_latest_code()

from your.downloaded.package import module

print("DYNAMIC IMPORTS SUCCESSFUL!")

Simply put it in your main Pipeline file. You can also automate it to get the sources from a local host. Hope this helps!