|
| 1 | +# Jobq, workflow submission and scheduling |
| 2 | + |
| 3 | +Jobq is a tool that aims to make it easy for you to create, submit and schedule workflows. |
| 4 | + |
| 5 | +To get teams shipping fast, we keep the definition of the workflows inside a familiar environment: Python. |
| 6 | +You wrap all your logic into a function that you then decorate with the `@job` decorator. |
| 7 | +In this decorator you will define the environment and hardware requirements. |
| 8 | + |
| 9 | +You then use the `jobq` CLI to submit the workflow, stop it, or retrieve logs. |
| 10 | + |
| 11 | +The `jobq` backed is built on top of Kubernetes and is installed in the cluster, orchestrates the workflow management, and interfaces to the client CLI calls. |
| 12 | + |
| 13 | +Let us walk through an example. |
| 14 | +The example expects a set-up cluster and API server. |
| 15 | + |
| 16 | +## Workflow definition in your Python files |
| 17 | + |
| 18 | +To restructure your Python script into a workflow that can be executed by `jobq` you need to reorganize your logic such that everything is called from one top level function. |
| 19 | +Then you decorate this function with the `jobq.job` decorator. |
| 20 | +The decorator takes two arguments, `image` and `options`. |
| 21 | +You use the `image` argument to submit `ImageOptions`, a wrapper specifying the image, that is the environment your code will execute in. |
| 22 | +The `ImageOptions` take a `spec`, that is a path to a `.yaml` or `Dockerfile` which outlines the environment creation, as well as the name and tag of the image. |
| 23 | + |
| 24 | +The other argument to the `job` decorator are `options`. |
| 25 | +Here you specify the desired resources using the `ResourceOptions` wrapper, can attach labels and specify the `SchedulingOptions`. |
| 26 | + |
| 27 | +A defined job looks similar to this |
| 28 | +```python |
| 29 | +# quickstart.py |
| 30 | +from pathlib import Path |
| 31 | +from jobq import ImageOptions, JobOptions, ResourceOptions, SchedulingOptions, job |
| 32 | + |
| 33 | +@job( |
| 34 | + options=JobOptions( |
| 35 | + labels={"type": "hello-world@quickstart"}, |
| 36 | + resources=ResourceOptions(memory='1Gi', cpu='1'), |
| 37 | + scheduling=SchedulingOptions( |
| 38 | + priority_class='background', queue_name="user-queue" |
| 39 | + ), |
| 40 | + image=ImageOptions( |
| 41 | + spec=Path('dockerfile/path/relative/to/job'), |
| 42 | + name="quickstart/hello-world", |
| 43 | + tag="latest" |
| 44 | + ) |
| 45 | + ) |
| 46 | +) |
| 47 | +def quickstart(): |
| 48 | + print("Hello, World!") |
| 49 | +``` |
| 50 | + |
| 51 | +## Interface with your workflows using the `jobq` CLI |
| 52 | +Now that we have the job defined let us execute it. |
| 53 | +You can execute the job via the CLI,`jobq submit quickstart.py`. |
| 54 | +This execution is locally, which may be useful for debugging and testing. |
| 55 | +However, you probably want to execute the job on the remote cluster. |
| 56 | +In this case you can add the `--mode kueue` flag to submit the workflow to the cluster queue (orchestrated by Kubernetes Kueue) like so: |
| 57 | +`jobq submit --mode kueue quickstart.py`. |
| 58 | + |
| 59 | +This returns the job id which we can use to fetch or stream the logs using `jobq logs <job id>`. |
| 60 | + |
| 61 | +## Setting up the `jobq` Backend and API server |
| 62 | +In this quickstart guide we use Minikube to run a Kubernetes cluster locally. |
| 63 | +In case you want to deploy the `jobq` backend on your own cluster, follow the instructions [here](/link/to/clustersetup). |
| 64 | + |
| 65 | +Follow the [ instructions ](https://minikube.sigs.k8s.io/docs/start/) to install Minikube in the Minikube docs. You will also need a container or virtual machine manager like [Docker](https://www.docker.com/get-started/). |
| 66 | +Then, open a terminal and start Minikube with `minikube start`. |
| 67 | + |
| 68 | +Next, you need to ensure that the registry add-on is enabled. |
| 69 | +Do so by running `minikube addons enable registry`. |
| 70 | +The registry is where the cluster saves the job images. |
0 commit comments