Show HN: LLMFlows – LangChain alternative for explicit and transparent apps (opens in new tab)

(github.com)

49 pointsspstoyanov2y ago22 comments

Hi HN! Over the last several weekends, I've been building LLMFlows as an alternative to langchain.

There's been a lot of discussion on the shortcomings of langchain in the past few weeks, but when I first tried it in March, I thought there are 3 main problems: 1. Too many abstractions 2. Hidden prompts and opinionated logic in chains which makes it hard to customize 3. Hard to debug

This inspired me to try and build a framework that solves these 3 issues, and therefore I started building LLFlows with the "philosophy" of being "simple, explicit, and transparent."

A few weekends later, I think I finally managed to reach a state where I feel it's ready to be shared.

I would love to hear your feedback! Thank you!

22 comments

adamgordonbell2y ago

This looks nice!

I frequently am putting together complex chains of prompts and found microsoft guidance to be quite nice. Have you looked at it?

It feels like the abstraction I'd like the most is some way to combine few shot prompting, voting and chain-of-thought.

I have to create a prompt chain like this fairly often: write a description for this thing, given these previous input and outputs as examples (few shot). Do it N times with high randomness, maybe with expert personas, then review the solutions one by one with pros and cons (chain of thought) and then use all that to create a better description (final answer).

Right now, I always have to write out and connect all the steps, but its fairly rote and I think other prompting chains have a similar repetitiveness to them.

https://github.com/microsoft/guidance

spstoyanovOP2y ago

Thank you for your feedback! And thank you for sharing guidance!

You might find this guide useful to your use case:

https://llmflows.readthedocs.io/en/latest/user_guide/Async%2...

It shows how you can connect multiple similar flowsteps to a previous flowstep and they will all run in parallel.

adamgordonbell2y ago

Very cool!

CharlesW2y ago

I think you've explained the "LangChain alternative" motivation pretty well, but as someone who hasn't yet worked with LLMs from a development perspective, what are the primary benefits of a framework like this?

Does the framework help with building the vector store? If not, are there complementary frameworks you use and/or would recommend? I'd think one would want something that watches a collection of source documents (with lots of supported formats) and updates the vector store as those change.

One of the primary benefits of a framework seems to be to act as an abstraction for LLMs and vector stores. I see LLMFlows supports Pinecone. Does it support others (e.g. pgvector)? Does it support LLMs other than OpenAI LLMs?

spstoyanovOP2y ago

Thank you for your feedback!

Building apps with LLMs is fundamentally an exercise of manipulating strings and making API calls (Both LLM and vector db APIs). When apps are more complex and start including multiple LLM calls and prompts that can change dynamically some additional challenges start to emerge. e.g figuring out:

- When was a particular LLM called?

- How much time did it take?

- What were the input variables?

- What did the prompt look like?

- What was the exact configuration of each LLM?

- How many times did we retry the request?

- What was the raw data the API returned?

- What was exactly returned from the vector store and fed into an LLM?

- How many tokens were used?

- What was the final result for each call?

- How do you make API calls in parallel?

I think a framework like this should provide abstractions that allow people to focus on the important part like prompt engineering and productionizing the app and worry less about how to figure out stuff like this.

Right now LLMFlows supports only OpenAI models and Pinecone but I am working on classes for Chroma and Weaviate. And I would like to also provide support for Bard and Claude once I get access.

CharlesW2y ago

> I think a framework like this should provide abstractions that allow people to focus on the important part like prompt engineering and productionizing the app and worry less about how to figure out stuff like this.

I appreciate the detailed explanation, thank you!

If you'll permit another noob question, can/should frameworks like this include source attribution for responses? Based on articles like https://jamesg.blog/2023/04/02/llm-prompts-source-attributio... I'd guess the strict answer would be no, and that any strategies you might use to elicit them might hallucinate citations.

1 more reply

politelemon2y ago

> What did the prompt look like?

This is the main thing I'm struggling with langchain. I needed to see this info for troubleshooting. I don't know if it's easily visible and I'm being thick, or it's just hidden away.

perone2y ago

https://feste.readthedocs.io/en/latest/design.html, this seems to be what Feste already does, and it offers automatic batching as well.

kveykva2y ago

We're working on similar things! If you'd like to chat sometime I'd be interested. My HN username is also my twitter handle.

netdur2y ago

I am trying to do the same but using C++ instead of python... wonder if someone else also trying it!

habitue2y ago

I'm really glad people are iterating on the abstraction for these systems. This is exactly the place where creativity is really useful

politelemon2y ago

I see this only works with OpenAI at the moment. Any plans to include other sources, in thinking llama, bedrock.

spmurrayzzz2y ago

Many of the open source inference solutions you'd use for llama ship with an openai-compatible interface (e.g. oobabooga/text-generation-web-ui, huggingface/text-generation-inference, etc). Probably can just change the API host and it would work.

spstoyanovOP2y ago

Yes, I would love to get some feedback on what people want to see - right now I am thinking Bard and Claude. What do you want to have?

politelemon2y ago

For gpt, both OpenAI and Azure OpenAI, I think they are similar implementation, you just have to provide some additional parameters like deployment name and different URL.

For non gpt options, I like Claude and Amazon titan. Claude is definitely good.

I'm also currently learning to use llama.cpp I see it has a langchain integration and it's very fast with GPU. So it's useful to be but as a hobbyist.

I've got no experience with bard though. Is it any good?

1 more reply

CharlesW2y ago

Support for llama.cpp seems like it'd be really interesting for folks who don't want to send data back and forth to LLMaaS providers. https://github.com/ggerganov/llama.cpp

1 more reply

carterschonwald2y ago

Looks very nicely thought out.

spstoyanovOP2y ago

Thank you for the kind words!

j / k navigate · click thread line to collapse

22 comments

adamgordonbell2y ago

This looks nice!

I frequently am putting together complex chains of prompts and found microsoft guidance to be quite nice. Have you looked at it?

It feels like the abstraction I'd like the most is some way to combine few shot prompting, voting and chain-of-thought.

Right now, I always have to write out and connect all the steps, but its fairly rote and I think other prompting chains have a similar repetitiveness to them.

https://github.com/microsoft/guidance

spstoyanovOP2y ago

Thank you for your feedback! And thank you for sharing guidance!

You might find this guide useful to your use case:

https://llmflows.readthedocs.io/en/latest/user_guide/Async%2...

It shows how you can connect multiple similar flowsteps to a previous flowstep and they will all run in parallel.

adamgordonbell2y ago

Very cool!

CharlesW2y ago

spstoyanovOP2y ago

Thank you for your feedback!

- When was a particular LLM called?

- How much time did it take?

- What were the input variables?

- What did the prompt look like?

- What was the exact configuration of each LLM?

- How many times did we retry the request?

- What was the raw data the API returned?

- What was exactly returned from the vector store and fed into an LLM?

- How many tokens were used?

- What was the final result for each call?

- How do you make API calls in parallel?

Right now LLMFlows supports only OpenAI models and Pinecone but I am working on classes for Chroma and Weaviate. And I would like to also provide support for Bard and Claude once I get access.

CharlesW2y ago

I appreciate the detailed explanation, thank you!

1 more reply

politelemon2y ago

> What did the prompt look like?

This is the main thing I'm struggling with langchain. I needed to see this info for troubleshooting. I don't know if it's easily visible and I'm being thick, or it's just hidden away.

perone2y ago

https://feste.readthedocs.io/en/latest/design.html, this seems to be what Feste already does, and it offers automatic batching as well.

kveykva2y ago

We're working on similar things! If you'd like to chat sometime I'd be interested. My HN username is also my twitter handle.

netdur2y ago

I am trying to do the same but using C++ instead of python... wonder if someone else also trying it!

habitue2y ago

I'm really glad people are iterating on the abstraction for these systems. This is exactly the place where creativity is really useful

politelemon2y ago

I see this only works with OpenAI at the moment. Any plans to include other sources, in thinking llama, bedrock.

spmurrayzzz2y ago

spstoyanovOP2y ago

Yes, I would love to get some feedback on what people want to see - right now I am thinking Bard and Claude. What do you want to have?

politelemon2y ago

For gpt, both OpenAI and Azure OpenAI, I think they are similar implementation, you just have to provide some additional parameters like deployment name and different URL.

For non gpt options, I like Claude and Amazon titan. Claude is definitely good.

I'm also currently learning to use llama.cpp I see it has a langchain integration and it's very fast with GPU. So it's useful to be but as a hobbyist.

I've got no experience with bard though. Is it any good?

1 more reply

CharlesW2y ago

Support for llama.cpp seems like it'd be really interesting for folks who don't want to send data back and forth to LLMaaS providers. https://github.com/ggerganov/llama.cpp

1 more reply

carterschonwald2y ago

Looks very nicely thought out.

spstoyanovOP2y ago

Thank you for the kind words!

j / k navigate · click thread line to collapse