Vq

Vq (v[ector] q[uantizer]) is a Rust library that implements several popular vector quantization algorithms including binary, scalar, and product quantization algorithms. It provides a simple, efficient API for data compression that helps reduce memory usage and computational overhead.

Features

• Implemented Algorithms:

  • Binary Quantization (BQ)
  • Scalar Quantization (SQ)
  • Product Quantization (PQ)
  • Optimized Product Quantization (OPQ)
  • Tree-structured Vector Quantization (TSVQ)
  • Residual Vector Quantization (RVQ)

• Parallelized vector operations for large vectors using Rayon.

• Flexible quantization algorithm implementations that support using various distance metrics such as Euclidean, Cosine, Manhattan distances.

• Support for quantizing vectors of f32 to f16 (using half) or u8 data types.

• Simple, intuitive, and uniform API for all quantization algorithms.

Installation

cargo add vq

Vq requires Rust 1.83 or later.

Documentation

Find the latest documentation on docs.rs.

Check out vq_examples.rs the tests directory for detailed examples of using Vq.

Quick Example

Here's a simple example using the SQ algorithm to quantize a vector:

use vq::sq::ScalarQuantizer;
use vq::vector::Vector;

fn main() {
    // Create a scalar quantizer for values in the range [0.0, 1.0] with 256 levels.
    let quantizer = ScalarQuantizer::fit(0.0, 1.0, 256);

    // Create an input vector.
    let input = Vector::new(vec![0.1, 0.5, -0.8, -0.3, 0.9]);

    // Quantize the input vector.
    let quantized_input = quantizer.quantize(&input);

    println!("Quantized input vector: {}", quantized_input);
}

Performance

Check out the notebooks directory for information on how to evaluate the performance of the implemented algorithms. Additionally, see the content of src/bin directory for the scripts used for the evaluation.

On a ThinkPad T14 laptop with an Intel i7-1355U CPU and 32GB of RAM, the performance of the PQ algorithm for quantizing one million vectors of 128 dimensions (into 16 subspaces with 256 centroids per subspace) is as follows:

• Training Time: 232.5 seconds
• Quantization Time: 34.1 seconds
• Reconstruction Error: 0.02
• Recall@10: 0.19

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for details on contributing.

License

Vq is available under the terms of either of the following licenses:

• MIT License (LICENSE-MIT)
• Apache License, Version 2.0 (LICENSE-APACHE)