v0.2.0

New features

• Support .bin, .pt, .pth extensions
• Add Starcoder 2 GGUF
• 🔥 PagedAttention - beating llama.cpp running GGUF plus all the throughput benefits 😉
• Optimized performance and memory usage

Rust MSRV

MSRV of mistral.rs v0.2.0 is 1.75.

What's Changed

• Fix SWA order (flip it) for Gemma 2 by @EricLBuehler in #554
• Support .bin, .pt, .pth extensions by @EricLBuehler in #557
• Update readme by @EricLBuehler in #558
• Fix Starcoder 2 ISQ by @EricLBuehler in #559
• Update deps by @EricLBuehler in #560
• Add the starcoder2 GGUF arch by @EricLBuehler in #561
• Readme update for starcoder2 gguf by @EricLBuehler in #562
• Fix PyPI release trigger by @EricLBuehler in #566
• Optimize multi-batch and inference performance with PagedAttention by @EricLBuehler in #552
• [Breaking] Version 0.2.0 by @EricLBuehler in #527
• Paged attention support for vision models by @EricLBuehler in #567
• Automatically use paged attn on cuda, get memory size by @EricLBuehler in #568
• Add docs link for vision loader by @EricLBuehler in #570
• Add matching for valid model weight names by @EricLBuehler in #571
• Remove ensure about no paged attn for vision models by @EricLBuehler in #573
• Add percentage utilization support to paged attn by @EricLBuehler in #574
• Include block engine in paged attn metadata by @EricLBuehler in #576
• Update deps and sync Candle by @EricLBuehler in #578
• Optimize CLIP model by @EricLBuehler in #579
• Use softmax_last_dim in CLIP by @EricLBuehler in #580
• Fix method of calculating paged attn with util percent by @EricLBuehler in #581
• Handle windows in paged attn build by @EricLBuehler in #577
• Warn instead of error when paged attn not supported by @EricLBuehler in #583
• Warn instead of error when paged attn for adapters not supported by @EricLBuehler in #584
• Add support for lm_head to adapter models by @EricLBuehler in #586
• Add default plotly feature by @EricLBuehler in #587
• Improve memory handling of PagedAttention with GGUF by @EricLBuehler in #590
• Fix Windows build on cuda w/ PagedAttention by @EricLBuehler in #589
• Update cuda kernels build.rs on windows by @EricLBuehler in #591
• Bump version to 0.2.0 and update docs by @EricLBuehler in #582

Full Changelog: v0.1.26...v0.2.0

Install mistralrs-server 0.2.0

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.2.0/mistralrs-server-installer.sh | sh

Download mistralrs-server 0.2.0

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.1.26

What's Changed

• Reference cargo dist in readme by @EricLBuehler in #536
• Fix xlora table in readme by @EricLBuehler in #537
• Warning about device mapping for anymoe by @EricLBuehler in #538
• Add updater and installer programs to cargo dist functionality by @EricLBuehler in #540
• Device mapping for AnyMoE by @EricLBuehler in #539
• Add tracking of memory usage by @EricLBuehler in #392
• Fix memory avail for cuda by @EricLBuehler in #543
• Add LLaVA Support by @chenwanqq in #484
• Device mapping fix for normal llama by @EricLBuehler in #549
• Add benchmark for a6000 by @EricLBuehler in #550
• Bump version to 0.1.26 by @EricLBuehler in #551

Full Changelog: v0.1.25...v0.1.26

Install mistralrs-server 0.1.26

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/EricLBuehler/mistral.rs/releases/download/v0.1.26/mistralrs-server-installer.sh | sh

Download mistralrs-server 0.1.26

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.1.25

Summary

• Added AnyMoE
• Added the Starcoder 2 model
• Vision interactive mode
• X-LoRA and LoRA support for Gemma 2
• Optimized local loading speed
• Release using cargo dist

MSRV

cargo msrv

MSRV of mistral.rs is 1.75.0.

What's Changed

• Add a vision interactive mode by @EricLBuehler in #468
• Add support for X-LoRA/LoRA to Gemma 2 by @EricLBuehler in #499
• Add adapter activation support for Gemma 2 LoRA by @EricLBuehler in #500
• Decrease MSRV to 1.75.0 by @EricLBuehler in #501
• Fix docs with preprocessor config by @EricLBuehler in #502
• Allow for optional model field in OpenAI requests by @EricLBuehler in #504
• Support streaming for completion by @EricLBuehler in #508
• Select the best device in examples by @EricLBuehler in #507
• Handle Metal error from bf16 autodtype selection by @EricLBuehler in #509
• Use chat messages for the rust examples and show T/s by @EricLBuehler in #510
• Avoid f64 to f32 cast in phirope for dtype on metal by @EricLBuehler in #512
• Handle errors in rust examples by @EricLBuehler in #513
• AnyMoE: Build an MoE model from anything, quickly by @EricLBuehler in #476
• chore: update paths.rs by @eltociear in #516
• Add AnyMoE support for vision models by @EricLBuehler in #515
• Support images in AnyMoE dataset by @EricLBuehler in #517
• Handle metal case for nonzero and bitwise ops by @EricLBuehler in #518
• Support saving and loading AnyMoE gating layer by @EricLBuehler in #519
• Implement the Starcoder 2 model architecture by @EricLBuehler in #522
• Docs for Starcoder 2 by @EricLBuehler in #524
• Starcoder2 docs by @EricLBuehler in #525
• Export loader for starcoder2 by @EricLBuehler in #526
• Support the new phi3 models by @EricLBuehler in #530
• Source the AnyMoE dataset from a structured JSON file by @EricLBuehler in #531
• Create an AnyMoE loss graph by @EricLBuehler in #532
• Add AnyMoE demo video by @EricLBuehler in #533
• Add judgment for whether model_id is a local path to speed up local model loading by @chenwanqq in #523
• Update docs by @EricLBuehler in #534
• release using cargo-dist by @kranurag7 in #480
• Bump version to 0.1.25 by @EricLBuehler in #535

New Contributors

• @kranurag7 made their first contribution in #480

Full Changelog: v0.1.24...v0.1.25

Download mistralrs-server 0.1.25

File	Platform	Checksum
mistralrs-server-aarch64-apple-darwin.tar.xz	Apple Silicon macOS	checksum
mistralrs-server-x86_64-apple-darwin.tar.xz	Intel macOS	checksum
mistralrs-server-x86_64-unknown-linux-gnu.tar.xz	x64 Linux	checksum

v0.1.24

Patch release, please update

What's Changed

• Bump version to 0.1.24 by @EricLBuehler in #497

Full Changelog: v0.1.23...v0.1.24

v0.1.23

What's Changed

• Improve and update docs by @EricLBuehler in #477
• Progress bar and logging when loading repeating layers by @EricLBuehler in #479
• Update deps by @EricLBuehler in #483
• Optimize decoding by removing redundant qkv transpose by @EricLBuehler in #487
• Fixes and tweak docs, logging for local loading by @EricLBuehler in #489
• Add the Gemma 2 model by @EricLBuehler in #490
• Update demo video by @EricLBuehler in #491
• Utilize new quantize_onto qtensor api by @EricLBuehler in #492
• Update deps by @EricLBuehler in #493
• Bump version to 0.1.23 by @EricLBuehler in #495

Full Changelog: v0.1.22...v0.1.23

v0.1.22

What's Changed

• Remove erroneously flaky CI test by @EricLBuehler in #466
• NVCC flags support for mistralrs_core build by @EricLBuehler in #469
• Prevent divide by zero in cuda kernel by @joshpopelka20 in #471
• Better cuda build.rs linking of stdc++ by @EricLBuehler in #472
• Remove some unnecessary &muts by @EricLBuehler in #473
• Fix arg order for pdoc by @EricLBuehler in #474
• Bump version to 0.1.22 by @EricLBuehler in #475

Full Changelog: v0.1.21...v0.1.22

v0.1.21

What's Changed

• Expose idefics2 loader by @EricLBuehler in #450
• Try auto dtypes based on compute cap by @EricLBuehler in #453
• Fix dtype error for logit bias by @EricLBuehler in #454
• Fix sequence prompt len for Phi3-V by @EricLBuehler in #455
• Tune threshold for matmul via f16 by @EricLBuehler in #457
• Improve short/long scaling precision for LongRope by @EricLBuehler in #458
• Fix LongRope models position ids calculation by @EricLBuehler in #459
• Update deps by @EricLBuehler in #460
• Improve handling of errors in auto dtype selection by @EricLBuehler in #461
• Add support for cross-gpu device mapping by @EricLBuehler in #462
• Bump version to 0.1.21 by @EricLBuehler in #463

Full Changelog: v0.1.20...v0.1.21

v0.1.20

What's Changed

• Fix with docker images by fixing use of pyo3 by @EricLBuehler in #440
• Update readme with docker info by @EricLBuehler in #441
• Add Cargo.lock file by @EricLBuehler in #442
• Fix causal masks dtype by @EricLBuehler in #443
• Add support for Idefics 2 by @EricLBuehler in #309
• Bump to version 0.1.20 by @EricLBuehler in #449

Full Changelog: v0.1.19...v0.1.20

v0.1.19

What's Changed

• Format readme by @EricLBuehler in #427
• Remove multiple tracing initializations and init outside of mistralrs-core by @EricLBuehler in #428
• Run clippy by @EricLBuehler in #429
• adding reboot functionality by @gregszumel in #378
• Lower memory spike when loading with ISQ on CUDA by @EricLBuehler in #433
• Fix failing docs workflow by @EricLBuehler in #435
• Remove unused line in dockerignore by @EricLBuehler in #436
• Improve Auto dtype determination by @EricLBuehler in #438
• Bump version to 0.1.19 by @EricLBuehler in #439

Full Changelog: v0.1.18...v0.1.19

v0.1.18

What's Changed

• Switch to minijinja's pycompat mode by @mitsuhiko in #421
• chore: update speculative.rs by @eltociear in #423
• Bump to new commit of candle with cudarc 0.11.5 by @EricLBuehler in #424
• Use rev key instead of commit to get rid of warning by @EricLBuehler in #425
• Add nonzero and bitwise operators by @chenwanqq in #422
• Fix Python deps, base64 impl, add examples by @EricLBuehler in #426

New Contributors

• @mitsuhiko made their first contribution in #421
• @chenwanqq made their first contribution in #422

Full Changelog: v0.1.17...v0.1.18