Why A 50 TOPS NPU Is Almost Completely Useless

LinuxLinks has kindly given me the space to publish a semi-regular blog where I share my thoughts on open source, the Linux ecosystem, Windows, and other topics. It’s a place for my opinions and personal perspectives. As you might expect, everything I write here reflects my own views and prejudices.

NPUs are everywhere now. Intel has them. AMD has them. Qualcomm has them. Mini PC manufacturers proudly slap the TOPS figure all over their product pages as though it tells you something terribly important. 40 TOPS. 48 TOPS. 50 TOPS. Wonderful. What are you actually going to do with it?

For most people, particularly Linux users, the answer is remarkably close to nothing.

TOPS means trillions of operations per second, but it is a theoretical measure of certain low-precision AI calculations. It does not make your computer generally faster. Compile software, encode video, render in Blender, compress files, play a game or run a database and your shiny NPU will sit there doing absolutely bugger all. At least RGB lighting has the decency to glow.

Yes, NPUs are meant for AI. Fair enough. So fire up some popular local AI software. With a couple of exceptions, in Linux the NPU does nothing. Most software targets NVIDIA GPUs first, other GPUs if supported, and CPUs as the reliable fallback. NPU support often lives somewhere between experimental, awkward and “perhaps later”.

Linux makes this particularly amusing. You can buy a processor advertised around its 50 TOPS NPU, load a model and watch the CPU or GPU do all the work while the dedicated AI accelerator enjoys another peaceful afternoon.

Actually getting an NPU working can be even more entertaining. You may need a particular runtime, model format, conversion tool, quantisation scheme and exactly the right library versions. Then one unsupported operator ruins the party. GPUs can certainly be awkward, but NPUs sometimes turn running a model into an archaeological dig through GitHub issues and half-finished documentation.

And 50 TOPS itself tells you surprisingly little. What precision? Which operators? What memory bandwidth? Which models and frameworks work? What performance is sustained? Without that information, TOPS is largely a marketing number. It is a bit like advertising a car solely by its engine RPM.

NPUs are not inherently pointless. They can be excellent for low-power, always-on tasks such as noise suppression, speech processing, webcam effects and small vision workloads. The problem is that the hardware has arrived before much of the software capable of using it.

That will probably change. But I look at hardware as it exists today, not as vendors hope it might work two years from now. Right now, a 50 TOPS NPU often feels like a beautifully engineered room in your house with no door. Very impressive on the floor plan. Not much use to anyone living there.