Sglang benchmark test

[stbaione]

Description

Create a nightly workflow for SGLang Benchmark test that enables running a Shortfin server and benchmarking from SGLang, using the bench_serving script.

bench_serving Invocations

The bench_serving script is ran with various request-rate arguments:

• python -m sglang.bench_serving --backend shortfin --num-prompt 10 --base-url http://localhost:8000 --tokenizer=<tokenizer_path> --request-rate 1 --output-file <tmp_dir>/shortfin_10_1.jsonl
• python -m sglang.bench_serving --backend shortfin --num-prompt 10 --base-url http://localhost:8000 --tokenizer=<tokenizer_path> --request-rate 2 --output-file <tmp_dir>/shortfin_10_1.jsonl
• python -m sglang.bench_serving --backend shortfin --num-prompt 10 --base-url http://localhost:8000 --tokenizer=<tokenizer_path> --request-rate 4 --output-file <tmp_dir>/shortfin_10_1.jsonl
• python -m sglang.bench_serving --backend shortfin --num-prompt 10 --base-url http://localhost:8000 --tokenizer=<tokenizer_path> --request-rate 8 --output-file <tmp_dir>/shortfin_10_1.jsonl
• python -m sglang.bench_serving --backend shortfin --num-prompt 10 --base-url http://localhost:8000 --tokenizer=<tokenizer_path> --request-rate 16 --output-file <tmp_dir>/shortfin_10_1.jsonl
• python -m sglang.bench_serving --backend shortfin --num-prompt 10 --base-url http://localhost:8000 --tokenizer=<tokenizer_path> --request-rate 32 --output-file <tmp_dir>/shortfin_10_1.jsonl

After the test is finished running, we upload the html output from pytest to gh-pages. The subdirectory is set to ./llm/sglang, so the results should be accessible from the browser at /llm/sglang/index.html in gh-pages.

This also includes a refactor of the existing integration test. Most of the methods for downloading a model/tokenizer, exporting to mlir, compiling to vmfb, and starting a shortfin server have been moved to a utils.py file.

[marbre]

I am aware that this is still a draft, however, some things to consider while getting it into a shape that can be merged.

[marbre]

Some first comments. I think there was an agreement to add a new top-level folder @ScottTodd?

[stbaione]

Some first comments. I think there was an agreement to add a new top-level folder @ScottTodd?

I moved the integration and benchmark tests out of /build_tools to a new top-level /app_tests directory:

#476 (comment)

[marbre]

The token cannot be accessed from an outside PR / PR from a fork. Thus, if dropping the pull_request trigger, it should be fine to merge (but can do another review round if you would like me to).

[stbaione]

The token cannot be accessed from an outside PR / PR from a fork. Thus, if dropping the pull_request trigger, it should be fine to merge (but can do another review round if you would like me to).

Gotcha, that makes sense. Removed pull_request trigger. Yeah, if you have the time, another review round would be great

[marbre]

I think this is fine to land and to iterate on as needed.