Shimmy is a 4.8MB single-binary that provides 100% OpenAI-compatible endpoints for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free.
This is free and open source software.
Features include:
- Compatible with OpenAI SDKs and Tools:
- No code changes needed – just change the API endpoint:
- Any OpenAI client: Python, Node.js, curl, etc.
- Development applications: Compatible with standard SDKs
- VSCode Extensions: Point to http://localhost:11435
- Cursor Editor: Built-in OpenAI compatibility
- Continue.dev: Drop-in model provider.
- Zero Configuration Required:
- Automatically finds models from Hugging Face cache, Ollama, local dirs.
- Auto-allocates ports to avoid conflicts.
- Auto-detects LoRA adapters for specialized models.
- Just works – no config files, no setup wizards.
- Advanced MOE (Mixture of Experts) Support:
- Run 70B+ models on consumer hardware with intelligent CPU/GPU hybrid processing:
- CPU MOE Offloading: Automatically distribute model layers across CPU and GPU.
- Intelligent Layer Placement: Optimizes which layers run where for maximum performance.
- Memory Efficiency: Fit larger models in limited VRAM by using system RAM strategically.
- Hybrid Acceleration: Get GPU speed where it matters most, CPU reliability everywhere else.
- Configurable:
--cpu-moeand--n-cpu-moeflags for fine control.
- Local Development:
- Privacy: Your code never leaves your machine.
- Cost: No API keys, no per-token billing.
- Speed: Local inference, sub-second responses.
- Reliability: No rate limits, no downtime.
Website: github.com/Michael-A-Kuykendall/shimmy
Support:
Developer: Michael A. Kuykendall
License: MIT License
Shimmy is written in Rust and C. Learn Rust with our recommended free books and free tutorials. Learn C with our recommended free books and free tutorials.
| Popular series | |
|---|---|
 | The largest compilation of the best free and open source software in the universe. Each article is supplied with a legendary ratings chart helping you to make informed decisions. |
 | Hundreds of in-depth reviews offering our unbiased and expert opinion on software. We offer helpful and impartial information. |
 | The Big List of Active Linux Distros is a large compilation of actively developed Linux distributions. |
 | Replace proprietary software with open source alternatives: Google, Microsoft, Apple, Adobe, IBM, Autodesk, Oracle, Atlassian, Corel, Cisco, Intuit, SAS, Progress, Salesforce, and Citrix |
 | Awesome Free Linux Games Tools showcases a series of tools that making gaming on Linux a more pleasurable experience. This is a new series. |
 | Machine Learning explores practical applications of machine learning and deep learning from a Linux perspective. We've written reviews of more than 40 self-hosted apps. All are free and open source. |
 | New to Linux? Read our Linux for Starters series. We start right at the basics and teach you everything you need to know to get started with Linux. |
 | Alternatives to popular CLI tools showcases essential tools that are modern replacements for core Linux utilities. |
 | Essential Linux system tools focuses on small, indispensable utilities, useful for system administrators as well as regular users. |
 | Linux utilities to maximise your productivity. Small, indispensable tools, useful for anyone running a Linux machine. |
 | Surveys popular streaming services from a Linux perspective: Amazon Music Unlimited, Myuzi, Spotify, Deezer, Tidal. |
 | Saving Money with Linux looks at how you can reduce your energy bills running Linux. |
 | Home computers became commonplace in the 1980s. Emulate home computers including the Commodore 64, Amiga, Atari ST, ZX81, Amstrad CPC, and ZX Spectrum. |
 | Now and Then examines how promising open source software fared over the years. It can be a bumpy ride. |
 | Linux at Home looks at a range of home activities where Linux can play its part, making the most of our time at home, keeping active and engaged. |
 | Linux Candy reveals the lighter side of Linux. Have some fun and escape from the daily drudgery. |
 | Getting Started with Docker helps you master Docker, a set of platform as a service products that delivers software in packages called containers. |
 | Best Free Android Apps. We showcase free Android apps that are definitely worth downloading. There's a strict eligibility criteria for inclusion in this series. |
 | These best free books accelerate your learning of every programming language. Learn a new language today! |
 | These free tutorials offer the perfect tonic to our free programming books series. |
 | Linux Around The World showcases usergroups that are relevant to Linux enthusiasts. Great ways to meet up with fellow enthusiasts. |
 | Stars and Stripes is an occasional series looking at the impact of Linux in the USA. |