I’ve been working on an AI agent system over the past year that connects to internal company tools like Slack, GitHub, Notion, etc, to help investigate production incidents. The agent needs context, so we built a system that ingests this data, processes it, and builds a structured knowledge graph (kind of a mix of RAG and GraphRAG).
What we didn’t expect was just how much infra work that would require.
We ended up:
- Using LlamaIndex's OS abstractions for chunking, embedding and retrieval.
- Adopting Chroma as the vector store.
- Writing custom integrations for Slack/GitHub/Notion. We used LlamaHub here for the actual querying, although some parts were a bit unmaintained and we had to fork + fix. We could’ve used Nango or Airbyte tbh but eventually didn't do that.
- Building an auto-refresh pipeline to sync data every few hours and do diffs based on timestamps. This was pretty hard as well.
- Handling security and privacy (most customers needed to keep data in their own environments).
- Handling scale - some orgs had hundreds of thousands of documents across different tools.
It became clear we were spending a lot more time on data infrastructure than on the actual agent logic. I think it might be ok for a company that interacts with customers' data, but definitely we felt like we were dealing with a lot of non-core work.
So I’m curious: for folks building LLM apps that connect to company systems, how are you approaching this? Are you building it all from scratch too? Using open-source tools? Is there something obvious we’re missing?
Would really appreciate hearing how others are tackling this part of the stack.
