Ask HN: How do you analyze conversations with AI agents in your products?

We have config defining the various topics of interests – so in your case that would be "things users hate" and "things users love" - then as a nightly job we'll get an LLM to extract values based on those topics for any active conversations that occurred during the previous 24 hours.

When that job finishes we have a list of conversation ids paired with the raw extracted values from conversations.

We then run a second job to analyse what we've extracted, asking the LLM to dedupe, categorise, tag and count the extracted values. So "I hate your website" and "your site sucks" might become "doesn't like website" with a category of "website feedback".

Then when clients view their analysis we show them those deduped values for each topic of interest, ordered by the the total count. We also maintain links back to the conversation so they can dig deeper to understand the context. They can filter on tags and categories.

Like I say, probably better approaches, but this works well enough for us.