Shimmy – lightweight OpenAI API server

Shimmy is a 4.8MB single-binary that provides 100% OpenAI-compatible endpoints for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free.

This is free and open source software.

Features include:

• Compatible with OpenAI SDKs and Tools:
  • No code changes needed – just change the API endpoint:
  • Any OpenAI client: Python, Node.js, curl, etc.
  • Development applications: Compatible with standard SDKs
  • VSCode Extensions: Point to http://localhost:11435
  • Cursor Editor: Built-in OpenAI compatibility
  • Continue.dev: Drop-in model provider.
• Zero Configuration Required:
  • Automatically finds models from Hugging Face cache, Ollama, local dirs.
  • Auto-allocates ports to avoid conflicts.
  • Auto-detects LoRA adapters for specialized models.
  • Just works – no config files, no setup wizards.
• Advanced MOE (Mixture of Experts) Support:
  • Run 70B+ models on consumer hardware with intelligent CPU/GPU hybrid processing:
  • CPU MOE Offloading: Automatically distribute model layers across CPU and GPU.
  • Intelligent Layer Placement: Optimizes which layers run where for maximum performance.
  • Memory Efficiency: Fit larger models in limited VRAM by using system RAM strategically.
  • Hybrid Acceleration: Get GPU speed where it matters most, CPU reliability everywhere else.
  • Configurable: --cpu-moe and --n-cpu-moe flags for fine control.
• Local Development:
  • Privacy: Your code never leaves your machine.
  • Cost: No API keys, no per-token billing.
  • Speed: Local inference, sub-second responses.
  • Reliability: No rate limits, no downtime.

Website: github.com/Michael-A-Kuykendall/shimmy
Support:
Developer: Michael A. Kuykendall
License: MIT License

Shimmy is written in Rust and C. Learn Rust with our recommended free books and free tutorials. Learn C with our recommended free books and free tutorials.

Explore Linux
	Best Free and Open Source Software - A huge collection of recommended free and open source software covering every major category.
	In-depth Linux Software Reviews - Detailed assessments of Linux software covering features, usability and overall quality.
	The Big List of Active Linux Distros - An extensive collection of actively developed Linux distributions.
	Open Source Alternatives - Alternatives to software and services from Google, Microsoft, Apple, Adobe, IBM, Autodesk, Oracle and others.
	Essential Linux System Tools - Indispensable utilities for monitoring, maintaining and managing Linux systems.
	Linux Productivity Tools - Useful utilities for improving productivity and streamlining everyday tasks.
	Alternatives to Popular CLI Tools - Modern alternatives to familiar command-line utilities supplied with Linux systems.
	Awesome Free Linux Game Tools - Tools, utilities and companion software that improve gaming on Linux.
	Machine Learning on Linux - Machine learning and deep learning software, self-hosted applications and practical tools.
	Saving Money with Linux - Practical ways to reduce electricity consumption and lower computing costs with Linux.
	Linux Candy - Fun, entertaining and sometimes delightfully quirky software from the lighter side of Linux.