Skip to content
Better HN
Top
New
Best
Ask
Show
Jobs
Search
⌘K
0 comments
No comments yet.
Show HN: FlashTokenizer – 10x faster C++ tokenizer for Python | Better HN
Show HN: FlashTokenizer – 10x faster C++ tokenizer for Python
(opens in new tab)
(github.com)
5 points
springkim
11mo ago
0 comments
Share
I built a tokenizer in C++ with a Python binding that outperforms HuggingFace tokenizers by 10x on large inputs. It's optimized for minimal memory usage and latency.
Benchmarks and comparison included in README. Would love feedback or contributions!