Your next CPU could be the biggest AI upgrade in years, here’s why

AI is fast approaching a fork in the road. The spiraling cost of memory for both personal and cloud platforms, rising compute costs for training next-gen models, and high user demand for cloud platforms have made AI quite expensive to both provide and use. Then there’s the growing privacy concerns — just how much sensitive personal information are we giving away to chatbots?

The solution to all these problems increasingly looks like running models on your own devices. Whether that’s Gemini’s Nano models summarizing your emails on your phone or tweaking your LoRAs to generate images on your PC’s GPU, running AI locally is already a powerful tool that, in theory, can do even more without an internet connection.

However, this is easier said than done. Running your own large language models not only requires a large pool of expensive, fast RAM, but you also need an accelerator to crunch those numbers. While our smartphones have long supported NPUs for exactly this purpose, proprietary APIs and inconsistent software support have limited widespread adoption. Not to mention that flagship and mid-range phones have vastly different capabilities, meaning developers often have to maintain multiple code paths or fall back to slower CPU implementations anyway.

Instead, accelerating basic AI workloads on the CPU is an increasingly popular middle ground. It might not be anywhere near as fast or as capable as running GPT-OSS on your NVIDIA 5090, but targeting the CPU means the tool runs on essentially anything, without the development headaches of proprietary drivers and APIs. With the right building blocks deeper in the CPU, this doesn’t have to be as slow as it might sound at first.

Mobile CPUs began moving toward more capable AI compute capabilities with Armv9 in 2021, which introduced SVE2. Rather than a fixed-width single instruction multiple data (SIMD) design, SVE2 uses a vector-length-agnostic model, allowing implementations to scale SIMD width from 128-bit up to 2048-bit depending on the hardware. In other words, it is a more flexible approach to doing fast math in parallel. It also added support for INT8 dot-product operations and enhanced support for low-precision arithmetic such as FP16, making it well-suited for modern quantized AI workloads.

However, the real shift came with SME and SME2. SME2 extends the CPU with a dedicated matrix execution mode, including a new matrix register and hardware support for GEMM-style operations. Unlike traditional vector execution, SME2 is designed specifically for the dense linear algebra patterns that dominate transformer and LLM workloads, significantly reducing memory traffic and increasing throughput for matrix-multiplication-heavy tasks.

Even if ideas like SME2 can’t close the performance gap on power-hungry dedicated accelerators, they offer vastly improved performance over older CPUs without acceleration support — up to 3x to 5x better in some cases, Arm cites. Importantly, they’re low-power, take up very little extra space, and are easier to target from a development standpoint, as they’re already embedded in common libraries like Arm’s KleidiAi.

SME2 is already available in MediaTek’s Dimensity 9500 processor and will undoubtedly arrive in more next-gen chipsets built from Arm’s C1 Ultra cores or licensed architectures.

With the history lesson out of the way, in recent news, AMD and Intel announced a joint venture to bring AI Compute Extensions (ACE) to future CPUs. Much like Arm has already done, this brings native matrix instructions to the x86 ISA, along with support for tiny INT4 sub-byte types and more conventional BF16 and FP16 data types, all from the familiar CPU instruction set.

This builds on existing AVX instructions, delivering consistent, enhanced functionality and faster parallel math processing across vendors on the x86 platform, making life much easier for developers looking to bring AI workloads to consumer products. Just like modern phone processors, ACE is integrated directly into the CPU pipeline, meaning no external GPU/NPU offloading or external APIs are required.

While implementation details will vary across vendors, SME2 and ACE both represent tightly integrated CPU matrix execution capabilities rather than offload-style accelerator architectures. CPUs are evolving beyond vector SIMD into architectures that natively support tensor and matrix computation. While GPUs’ hugely parallel nature will remain dominant for large-scale training and running the largest models, CPUs are rapidly becoming far more capable for on-device and low-latency AI inference workloads.

Perhaps the most important part is that it will be much easier for developers to target CPU-based AI acceleration across phones, laptops, and PCs, without relying on proprietary GPU or NPU APIs. If there’s one drawback, it’s that existing phones and laptops won’t benefit, but perhaps your next upgrade will.