CPU

Machine Learning in Linux: Text generation web UI

In Operation

To start the software, run the start_linux.sh shell script.

Text generation web UI start script

This is what we see in Firefox.

Text Generation Web UI startup

As explained in the Installation section, you’ll need to install a model. Click the Model menu entry at the top of the screen. In the Download custom model or LoRA field, enter the model. We’ll download facebook/opt-1.3b.

Model downloaded and loaded
Click image for full size

We’ve downloaded the facebook_opt-1.3b model and loaded it.

Here’s an example chat with this model.

Example chat

There are, in fact, 13 eggs in a baker’s dozen. Like many responses from LLMs, accuracy of the responses should never be blindly trusted.

What features are available? An impressive array. Here are the highlights:

  • 3 interface modes: default, notebook, and chat.
  • Multiple model backends: transformers, llama.cpp, ExLlama, AutoGPTQ, GPTQ-for-LLaMa.
  • Dropdown menu for quickly switching between different models.
  • LoRA: load and unload LoRAs on the fly, train a new LoRA. One of the techniques that helps reduce the costs of fine-tuning enormously is “low-rank adaptation” (LoRA). With LoRA, you can fine-tune LLMs at a fraction of the cost it would normally take.
  • Precise instruction templates for chat mode, including Llama 2, Alpaca, Vicuna, WizardLM, StableLM, and many others.
  • Multimodal pipelines, including LLaVA and MiniGPT-4.
  • 8-bit and 4-bit inference through bitsandbytes.
  • CPU mode for transformers models.
  • DeepSpeed ZeRO-3 inference. ZeRO-Inference pins the entire model weights in CPU or NVMe (whichever is sufficient to accommodate the full model) and streams the weights layer-by-layer into the GPU for inference computation.
  • Extensions.
  • Custom chat characters.
  • Very efficient text streaming.
  • Markdown output with LaTeX rendering, to use for instance with GALACTICA.

Summary

Text generation web UI offers a trouble-free way to experiment with a wide range of LLMs. It’s easy-peasy to install.

The software lets you experiment with lots of language models. We are particularly fascinated with GALACTICA, a general-purpose scientific language model. It’s trained on a large corpus of scientific text and data and performs scientific NLP tasks at a high level.

At the time of writing, Text generation web UI has amassed nearly 20k GitHub stars.

Website: github.com/oobabooga/text-generation-webui
Support:
Developer: oobabooga
License: GNU Affero General Public License v3.0

Artificial intelligence icon For other useful open source apps that use machine learning/deep learning, we’ve compiled this roundup.

Text generation web UI is written in Python. Learn Python with our recommended free books and free tutorials.

Pages in this article:
Page 1 – Introduction and Installation
Page 2 – In Operation and Summary

Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

1 Comment
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
PenguinMania
PenguinMania
8 months ago

This software lives up to its hype!