Show HN: ctoc – Count Tokens of Code (cloc for the LLM Era) (opens in new tab)

Hi HN, I'm building a CLI tool for counting tokens of code. I’m excited to share this project and would love to hear your feedback!

----

Token counts play a key role in shaping a Large Language Model's (LLM) memory and conversation history. They're vital for prompt engineering and token cost estimation. Various strategies in prompt engineering (e.g., contextual filtering and reranking) predominantly aim at token compression to counteract LLM's context size limit.

ctoc provides a lightweight tool for analyzing codebases at the token level. It incorporates all the features of cloc. (You can use ctoc in a cloc-consistent manner.)

Built on top of gocloc, ctoc is extremely fast.