-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
c3e8c0a
commit 002fa5e
Showing
1 changed file
with
58 additions
and
82 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,121 +1,97 @@ | ||
# BRRO Compressor | ||
|
||
Version: 0.5 Released: 30/11/2023 | ||
|
||
## Major Changes | ||
|
||
### 0.5 | ||
|
||
- Added Polynomial Compressor (with 2 variants) | ||
- Created and Integrated a proper file type (wbro) | ||
- Benchmarks of the different compressors | ||
- Integration testing | ||
- Several fixes and cleanups | ||
|
||
## Description | ||
**BRRO Compressor** is an advanced signal-based compression tool designed to provide better compression ratios than traditional methods. It performs an initial analysis of the input signal to select the best compression technique for each segment. | ||
|
||
BRRO Compressor is a compressor that relies on the characteristics of a signal to provide a far greater compression that currently existing ones. BRRO relies on different techniques based on a initial analysis of the signal to use the best suited method for compressor for that specific signal segment. | ||
|
||
For a detailed description on the compressor methods and logic check `BRRO.md`. | ||
|
||
## Getting Started with BRRO Compressor | ||
For detailed information on the compression methods used, see [BRRO.md](paper/BRRO.md). | ||
|
||
## Major Changes | ||
### Version 0.5 - Released 30/11/2023 | ||
- Added Polynomial Compressor (with 2 variants). | ||
- Introduced `wbro` file type. | ||
- Added benchmarks for various compressors. | ||
- Integration testing implemented. | ||
- Various fixes and code cleanups. | ||
|
||
## Getting Started | ||
### Prerequisites | ||
|
||
- Ensure you have [Rust](https://www.rust-lang.org/tools/install) and Cargo installed on your system. | ||
- Install Rust and Cargo on your system. | ||
|
||
### Installation | ||
|
||
1. Clone the repository: | ||
```bash | ||
git clone https://github.com/instaclustr/fft-compression | ||
cd fft-compression | ||
``` | ||
|
||
```bash | ||
git clone https://github.com/instaclustr/fft-compression | ||
cd fft-compression | ||
``` | ||
2. Build the project: | ||
```bash | ||
cargo build --release | ||
``` | ||
|
||
### Usage | ||
```bash | ||
cargo build --release | ||
``` | ||
|
||
Currently BRRO relies on have Raw BRRO files generated by our prometheus remote endpoint. This would work as input for the compressor. | ||
|
||
Compressor usage: | ||
## Usage | ||
BRRO Compressor currently requires raw BRRO files generated by a Prometheus remote endpoint as input. | ||
|
||
### Command-Line Options | ||
``` | ||
Usage: brro-compressor [OPTIONS] <INPUT> | ||
|
||
Arguments: | ||
<INPUT> input file | ||
<INPUT> Input file | ||
|
||
Options: | ||
--compressor <COMPRESSOR> [default: auto] [possible values: auto, fft, constant, polynomial] | ||
-u Uncompresses the input file/directory | ||
-h, --help Print help | ||
-V, --version Print version | ||
--compressor <COMPRESSOR> | ||
Select a compressor, default is auto [default: auto] [possible values: auto, noop, fft, constant, polynomial, idw] | ||
-e, --error <ERROR> | ||
Sets the maximum allowed error for the compressed data, must be between 0 and 50. Default is 5 (5%). 0 is lossless compression 50 will do a median filter on the data. In between will pick optimize for the error [default: 5] | ||
-u | ||
Uncompresses the input file/directory | ||
-c, --compression-selection-sample-level <COMPRESSION_SELECTION_SAMPLE_LEVEL> | ||
Samples the input data instead of using all the data for selecting the optimal compressor. Only impacts speed, might or not increased compression ratio. For best results use 0 (default). Only works when compression = Auto. 0 will use all the data (slowest) 6 will sample 128 data points (fastest) [default: 0] | ||
--verbose | ||
Verbose output, dumps everysample in the input file (for compression) and in the ouput file (for decompression) | ||
-h, --help | ||
Print help | ||
-V, --version | ||
Print version | ||
``` | ||
#### Compress a File | ||
|
||
### Compress a File | ||
To compress a file using the BRRO Compressor, run: | ||
|
||
```bash | ||
brro-compressor <input-file> | ||
``` | ||
|
||
#### Decompress a File | ||
|
||
To decompress a file, use the following command: | ||
|
||
### Decompress a File | ||
To decompress a file, use: | ||
```bash | ||
brro-compressor -u <input-file> | ||
``` | ||
|
||
## Programs and description | ||
|
||
This repository contains one main program and other programs that serve different purposes, some are for just testing, others do some actual work. | ||
|
||
### flac-server | ||
## Programs and Descriptions | ||
|
||
**NOTE**: Remote read is currently NOT working, as it depends on FLAC files that are no longer generated. | ||
Requires a Prometheus server to collect samples. Supports read and write operations with Prometheus. | ||
|
||
Needs a prometheus server. We need it to get our samples out. Supports read and write from prometheus. | ||
|
||
Launch the `flac-server` and set it as your remote endpoint for prometheus, example below. | ||
|
||
```YAML | ||
Example configuration: | ||
```yaml | ||
# Remote read and Write | ||
remote_write: | ||
- url: "http://localhost:9201/api/write" | ||
- url: "http://localhost:9201/api/write" | ||
|
||
remote_read: | ||
- url: "http://localhost:9201/api/read" | ||
read_recent: true | ||
name: "flac_server" | ||
- url: "http://localhost:9201/api/read" | ||
read_recent: true | ||
name: "flac_server" | ||
``` | ||
Make Prometheus server a source of your grafana and check the data. | ||
### brro_optimizer | ||
Maybe the most important tool at this point, it picks a WAV file from the datasets described below and optimizes it into a way that we might see a meaningful compression into FLAC. | ||
The tool also has options to dump the output of the file as a single sample per period, instead of the 4 channels. This is good to obtain the data as it was feed into the flac-server. | ||
The code performs optimizations based on file name, so renaming might cause issues. | ||
Usage (Getting raw samples): `./brro_optimizer infile.wav --dump-raw > file.raw` | ||
Usage (Getting optimized samples): `./brro_optimizer infile.wav --dump-optimized > file.raw` | ||
Usage (Generate a optimized file): `./brro_optimizer -w infile.wav` | ||
|
||
If you set the ENV Variable for Debug it will output what it is doing. | ||
|
||
### Matlab folder | ||
|
||
Exploratory code. Should be removed. | ||
Add the Prometheus server as a data source in Grafana to visualize the data. | ||
## Roadmap | ||
- Update `flac-server` to support reading and writing WBRO/BRO files. | ||
- Implement streaming compression/decompression. | ||
- Introduce automated compressor selection. | ||
- Implement frame expansion (allowing new data to be appended to existing frames). | ||
|
||
## Contributing | ||
Please see [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines. | ||
|
||
1. Update `flac-server` to read/write WBRO/BRO files. | ||
2. Streaming compression/decompression | ||
3. Automated compressor selection | ||
4. Frame expansion (Allowing new data to be appended to existing frames) | ||
## License | ||
This project is licensed under the [Apache License](LICENSE). |