Data Science

llm-visuals – terminal dashboard for local LLM inference

llm-visuals is a terminal-based monitoring dashboard for local large language model inference. It brings together information about inference performance, GPU activity and memory consumption, allowing users to investigate how their hardware behaves when running language models. The dashboard is particularly useful when comparing different models, quantisations or inference settings, and when trying to identify performance bottlenecks.

The application detects supported local inference servers and obtains measurements from their monitoring endpoints, operating-system interfaces and GPU drivers. It provides detailed integration with llama.cpp and also supports monitoring other inference engines. Available measurements depend on the server and hardware being used. The application distinguishes measured values from estimates and identifies unavailable telemetry rather than presenting unsupported figures as real measurements.

This is free and open source software.

Key Features

  • Displays live token generation and prompt processing speeds.
  • Measures time to first token and inference request durations.
  • Provides performance history graphs and sparklines.
  • Monitors GPU utilisation, memory consumption, temperature, power draw and clock speeds.
  • Supports NVIDIA, AMD and Intel GPU telemetry through available system interfaces.
  • Includes Apple silicon GPU monitoring using macmon or powermetrics.
  • Displays model weights and KV cache memory allocation where available.
  • Tracks context utilisation and prompt cache statistics.
  • Shows speculative decoding acceptance rates when exposed by the server.
  • Visualises model layer placement across multiple GPUs.
  • Provides an optional expert-routing view for mixture-of-experts models.
  • Includes a memory pipeline view covering storage, RAM, PCIe and GPU memory.
  • Tracks GPU energy consumption and estimated energy per generated token.
  • Calculates estimated inference energy costs using a configurable electricity price.
  • Supports monitoring multiple inference servers and models simultaneously.
  • Records individual inference requests with timing and token statistics.
  • Provides configurable performance and resource alerts.
  • Includes benchmarking and performance comparison facilities.
  • Stores monitoring sessions in an SQLite database.
  • Supports historical session replay with playback and seeking controls.
  • Includes a demonstration mode using synthetic inference and GPU data.
  • Cross-platform support – runs on Linux and macOS.

Website: github.com/DingoOz/llm-visuals
Support:
Developer: DingoOz
License: MIT License

llm-visuals is written in Rust. Learn Rust with our recommended free books and free tutorials.


Best Free and Open Source Software LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.

We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more.

Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion.

Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README.
Subscribe

Please read our Comment Policy before commenting.

Notify of
guest
0 Comments
Oldest
Newest Most Voted