Show HN: We built an AI tool for working with massive LLM chat log datasets (opens in new tab)

(hyperparam.app)

16 pointsplatypii4mo ago1 comments

There’s an important problem with AI that nobody’s talking about. AI’s entire lifecycle is tons of data in for training, and an even larger amount of text data out. Traditional tools can’t handle the sheer volume of text, leaving teams overwhelmed and unable to make their data work for them.

Today we’re launching Hyperparam, a browser-native app for exploring and transforming multi-gigabyte datasets in real time. It combines a fast UI that can stream huge unstructured datasets with an army of AI agents that can score, label, filter, and categorize them. Now you can actually make sense of AI-scale data instead of drowning in it.

Example: Using the chat, ask Hyperparam’s AI agent to score every conversation in a 100K-row dataset for sycophancy, filter out the worst responses, adjust prompts, regenerate, and export your dataset V2. It all runs in one browser tab with no waiting and no lag.

It’s free while it’s in beta if you want to try it on your own data.

1 comments

platypiiOP4mo ago

I started Hyperparam one year ago because I knew that the world of data was changing, and existing tools like Python and Jupyter Notebooks were not built for the scale of LLM data. The weights of LLMs may be tensors, but the input and output of LLMs are massive piles of text.

No human has the patience to sift through all that text, so we need better tools to help us understand and analyze it. That's why I built Hyperparam to be the first tool specifically designed for working with LLM data at scale. No one else seemed to be solving this problem.

j / k navigate · click thread line to collapse