Skip to content

Commit

Permalink
issue-98, readme file was improved
Browse files Browse the repository at this point in the history
  • Loading branch information
DoodgeMatvey committed Oct 15, 2024
1 parent c3e8c0a commit 002fa5e
Showing 1 changed file with 58 additions and 82 deletions.
140 changes: 58 additions & 82 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,121 +1,97 @@
# BRRO Compressor

Version: 0.5 Released: 30/11/2023

## Major Changes

### 0.5

- Added Polynomial Compressor (with 2 variants)
- Created and Integrated a proper file type (wbro)
- Benchmarks of the different compressors
- Integration testing
- Several fixes and cleanups

## Description
**BRRO Compressor** is an advanced signal-based compression tool designed to provide better compression ratios than traditional methods. It performs an initial analysis of the input signal to select the best compression technique for each segment.

BRRO Compressor is a compressor that relies on the characteristics of a signal to provide a far greater compression that currently existing ones. BRRO relies on different techniques based on a initial analysis of the signal to use the best suited method for compressor for that specific signal segment.

For a detailed description on the compressor methods and logic check `BRRO.md`.

## Getting Started with BRRO Compressor
For detailed information on the compression methods used, see [BRRO.md](paper/BRRO.md).

## Major Changes
### Version 0.5 - Released 30/11/2023
- Added Polynomial Compressor (with 2 variants).
- Introduced `wbro` file type.
- Added benchmarks for various compressors.
- Integration testing implemented.
- Various fixes and code cleanups.

## Getting Started
### Prerequisites

- Ensure you have [Rust](https://www.rust-lang.org/tools/install) and Cargo installed on your system.
- Install Rust and Cargo on your system.

### Installation

1. Clone the repository:
```bash
git clone https://github.com/instaclustr/fft-compression
cd fft-compression
```

```bash
git clone https://github.com/instaclustr/fft-compression
cd fft-compression
```
2. Build the project:
```bash
cargo build --release
```

### Usage
```bash
cargo build --release
```

Currently BRRO relies on have Raw BRRO files generated by our prometheus remote endpoint. This would work as input for the compressor.

Compressor usage:
## Usage
BRRO Compressor currently requires raw BRRO files generated by a Prometheus remote endpoint as input.

### Command-Line Options
```
Usage: brro-compressor [OPTIONS] <INPUT>

Arguments:
<INPUT> input file
<INPUT> Input file

Options:
--compressor <COMPRESSOR> [default: auto] [possible values: auto, fft, constant, polynomial]
-u Uncompresses the input file/directory
-h, --help Print help
-V, --version Print version
--compressor <COMPRESSOR>
Select a compressor, default is auto [default: auto] [possible values: auto, noop, fft, constant, polynomial, idw]
-e, --error <ERROR>
Sets the maximum allowed error for the compressed data, must be between 0 and 50. Default is 5 (5%). 0 is lossless compression 50 will do a median filter on the data. In between will pick optimize for the error [default: 5]
-u
Uncompresses the input file/directory
-c, --compression-selection-sample-level <COMPRESSION_SELECTION_SAMPLE_LEVEL>
Samples the input data instead of using all the data for selecting the optimal compressor. Only impacts speed, might or not increased compression ratio. For best results use 0 (default). Only works when compression = Auto. 0 will use all the data (slowest) 6 will sample 128 data points (fastest) [default: 0]
--verbose
Verbose output, dumps everysample in the input file (for compression) and in the ouput file (for decompression)
-h, --help
Print help
-V, --version
Print version
```
#### Compress a File

### Compress a File
To compress a file using the BRRO Compressor, run:

```bash
brro-compressor <input-file>
```

#### Decompress a File

To decompress a file, use the following command:

### Decompress a File
To decompress a file, use:
```bash
brro-compressor -u <input-file>
```

## Programs and description

This repository contains one main program and other programs that serve different purposes, some are for just testing, others do some actual work.

### flac-server
## Programs and Descriptions

**NOTE**: Remote read is currently NOT working, as it depends on FLAC files that are no longer generated.
Requires a Prometheus server to collect samples. Supports read and write operations with Prometheus.

Needs a prometheus server. We need it to get our samples out. Supports read and write from prometheus.

Launch the `flac-server` and set it as your remote endpoint for prometheus, example below.

```YAML
Example configuration:
```yaml
# Remote read and Write
remote_write:
- url: "http://localhost:9201/api/write"
- url: "http://localhost:9201/api/write"

remote_read:
- url: "http://localhost:9201/api/read"
read_recent: true
name: "flac_server"
- url: "http://localhost:9201/api/read"
read_recent: true
name: "flac_server"
```
Make Prometheus server a source of your grafana and check the data.
### brro_optimizer
Maybe the most important tool at this point, it picks a WAV file from the datasets described below and optimizes it into a way that we might see a meaningful compression into FLAC.
The tool also has options to dump the output of the file as a single sample per period, instead of the 4 channels. This is good to obtain the data as it was feed into the flac-server.
The code performs optimizations based on file name, so renaming might cause issues.
Usage (Getting raw samples): `./brro_optimizer infile.wav --dump-raw > file.raw`
Usage (Getting optimized samples): `./brro_optimizer infile.wav --dump-optimized > file.raw`
Usage (Generate a optimized file): `./brro_optimizer -w infile.wav`

If you set the ENV Variable for Debug it will output what it is doing.

### Matlab folder

Exploratory code. Should be removed.
Add the Prometheus server as a data source in Grafana to visualize the data.
## Roadmap
- Update `flac-server` to support reading and writing WBRO/BRO files.
- Implement streaming compression/decompression.
- Introduce automated compressor selection.
- Implement frame expansion (allowing new data to be appended to existing frames).

## Contributing
Please see [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines.

1. Update `flac-server` to read/write WBRO/BRO files.
2. Streaming compression/decompression
3. Automated compressor selection
4. Frame expansion (Allowing new data to be appended to existing frames)
## License
This project is licensed under the [Apache License](LICENSE).

0 comments on commit 002fa5e

Please sign in to comment.