IBM Granite: A Family of Open Foundation Models for Code Intelligence (opens in new tab)

You run most of these models in something that wraps them in an HTTP API. I use Ollama, which I think is the most popular but I’m not in a great position to judge. My impression is that it handles running models on CPU better.

So you’d basically install Ollama, download one of the versions of this model off HuggingFace, create a Modelfile since this isn’t in the default Ollama repo, and then Ollama can answer prompts with the model. Modelfiles are very simple, based on Dockerfiles. It takes like 15 seconds to make one if you aren’t messing with the various parameters.

Once it’s in Ollama, just get one of the various GPT plugins for VSCode and give it the Ollama URL (http://localhost:11434 by default). I use continue.dev but there are many.

Continue takes over the tab autocomplete with the LLM, and has a chat window on the right where you can use keyboard shortcuts to copy code into the prompt and ask it to edit/generate code or ask questions about existing code.

https://github.com/TabbyML/tabby can run self-hosted AI coding assistants. I tried it a while ago and it worked with Nvim pretty easily. There is a VS code extension too. The extension will just sort of "read" with you and provide suggestions from time to time. Anytime the suggestion is good you can press some key (<TAB> by default) to accept it. It's basically autocomplete on steroids.

If you like Emacs (I use both Emacs and VSCode, for slightly different coding use cases), then the Emacs elamma [1]package is very nice. It is set up out of the box to use Ollama and to use M-x commands for code completion, summarization, and dozens of other useful functions. I love it, your mileage may vary.

[1] https://github.com/s-kostyaev/ellama

WizardCoder, StarCoder, CodeLlama?

Is this a segway for IBM to release Terraform specific LLMs so I never have to write that hot garbage ever again? Sign me up IBM!

IBM made $60 billion in revenue last year. Where do you think it all came from? The same companies/governments that buy their overpriced crap are going to buy these new LLMs as well.

Enterprises think differently. They want data provenance, privacy, ability to mitigate/transfer risk etc. If IBM is willing to offer that, there will be enterprises that bite.

IBM do a mixture of shovelware and extremely hardcore tech so they could honestly go either way with this.

When they pitch potential clients for their services, their slides on LLM, AI, ML, etc., must be their own. Whether they use it or not for the services does not matter. These are like the side projects that service companies release to help them close their clients.

Same reason they jumped on the clown bandwagon, it's the kind of offering it's expected to have when you're a company like that. Huge size, leading research departments, big enterprise customers.

They've been doing "AI" for ages. Notably Watson over the last couple of decades or so.

> ... models that have no chance of competing with ...

I've not seen any proper evaluations for Granite against, say, Llama or Mistral.

Until we do it's probably too early to say they can't compete, at least in some areas where others perform poorly.

>I wonder why companies like IBM are jumping on the LLM bandwagon and training/releasing models that have no chance of competing with Llama/Mistral

Did you even read the benchmarks they post on that link? Assuming they're not outright lying, their 8B model is superior to Llama/Mistral models of the same size for coding tasks.

prob getting some reputation in AI space will help them to sell watsonx. tbf, watson predates Transformers paper.

On the other hand i spend my time wondering why people like you think someone should just throw away their ideas simply because there's already someone in the niche.