@naveen521kk/react-native-tokenizer

Port of Huggingface's Tokenizer written in Rust for React Native. Only supports Android for now.

Installation

npm install @naveen521kk/react-native-tokenizer

Usage

See example for a full example.

import { encode } from '@naveen521kk/react-native-tokenizer';

// ...
// modelPath is the path to the model directory
const result = encode(modelPath, "Hello World!");

Contributing

See the contributing guide to learn how to contribute to the repository and the development workflow.

License

MIT

---

Made with create-react-native-library