Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
no risky patterns detected
Type: Agent skill (SKILL.md) · Category: Programming Languages & Frameworks · Source: https://github.com/dabit3/sonic-agent/tree/HEAD/optional-skills/mlops/huggingfac
| Capabilities (fewer powerful permissions = higher) | 50/50 |
| Maintenance | 26/35 |
| Documentation | 15/15 |
Always review files before installing. How scores work. Request a correction