data-prep
Everything on Ground Truth tagged “data-prep” — 1 item.
GigaToken Tool
An open-source native BPE tokenizer optimized with SIMD byte scanning and instruction-level parallelism, best used for fast offline corpus preparation and bulk token counting rather than end-to-end serving speedups.