Show HN: Nexa SDK – Build powerful and efficient AI apps on edge devices (opens in new tab)

Hey HN! Alex and Zack here from Nexa AI. We're excited to share something we've been working on.

Our journey began with the Octopus series --- action models for mobile AI agents (https://huggingface.co/NexaAIDev/Octopus-v2). We focused on making sub-billion parameter models excel at function calling, making high accurate and fast function-calling possible on mobile and edge devices. But as we delved into developing full-fledged on-device applications, we hit a roadblock.

We realized that optimizing for function calling (tool-use) alone wasn't enough. Building powerful on-device AI apps requires a diverse set of tools: language models with domain expertise, speech processing, image generation, embedding models and more. That's when we decided to create Nexa SDK --- a comprehensive toolkit that brings together everything developers need to build powerful and efficient AI applications that run entirely on-device.

Here's what Nexa SDK offers:

  - Support for both ONNX and GGML models.
  - An integrated conversion engine for making custom GGML Quantized Models for different device hardware requirements.
  - An inference engine that supports language models, image generation models, TTS, audio generation models, and Vision-Language Models. 
  - An OpenAI-compatible API server with optimization in function calling.
  - A Streamlit UI for rapid prototyping.
  - An intuitive CLI for easy model management.
  - Backend optimizations for latency and power consumption on edge devices.

We've designed Nexa SDK to be the go-to solution for developers pushing the boundaries of what's possible with on-device AI applications and AI on edge devices.

To showcase its capabilities, we've built several demo apps running entirely on your device (https://github.com/NexaAI/nexa-sdk/tree/main/examples):

  - AI soulmate with uncensored model and audio-in/audio-out interaction.
  - A quick interface for uploading and chatting with PDFs like your personal finance documents.
  - A meeting transcription app supporting multiple languages and real-time translation.

We're proud to share that the winner of yesterday's (Sep 7) House AGI "AI PC/ GenAI Goes Local" hackathon used Nexa SDK to build a local semantic image search (https://github.com/asl3/deja-view).

But we're just getting started! There are lots of exciting developments in our pipeline, and we can't wait to share them with you soon!

Check it out: (https://github.com/NexaAI/nexa-sdk)

Docs: (https://docs.nexaai.com/)

If you're excited about the future of on-device AI, we'd really appreciate your support. A star on our GitHub repo goes a long way in helping us reach more developers!

Cheers,

Alex & Zack