Apple made some big claims about the M1 Ultra’s GPU performance. Were they being accurate or misleading?
As reviewers start releasing benchmarks of the new Mac Studio with M1 Ultra, people are taking to the ramparts to debate Apple’s performance claims.
Apple’s M1 performance graphs, like the one above, are highly triggering to some people. The X axis is pretty straight forward. It’s showing the M1 Ultra’s power consumption in watts compared against the Nvidia GeForce RTX 3090, a very high-end PC graphics card. But the Y axis lists “relative performance.” This is vague, and I can understand why many see it as Apple using sleight of hand. And to a point, they are.
When Apple hosts a product launch event or writes a press release, they’re speaking to a diverse audience, most of whom aren’t fluent in high-end tech specs and hardware. The main point that Apple wants to get across is that the M1 Ultra can achieve the performance of an RTX 3090 while using less power. But can it? This is where it gets tricky.
Here’s Monica Chin, writing for The Verge:
Apple, in its keynote, claimed that the M1 Ultra would outperform Nvidia’s RTX 3090. I have no idea where Apple’s getting that from. We ran Geekbench Compute, which tests the power of a system’s GPU, on both the Mac Studio and a gaming PC with an RTX 3090, a Core i9-10900, and 64GB of RAM. And the Mac Studio got… destroyed. It got less than half the score that the RTX 3090 did on that test — not only is it not beating Nvidia’s chip, but it’s not even coming close.
The other graphics benchmark they performed was testing the game The Shadow of the Tomb Raider. I wouldn’t say that’s an unfair benchmark to test with, but they should have disclosed to their readers that that game is running under Apple’s Rosetta 2 translation layer because it’s an Intel binary, not an Apple Silicon-native one. That’s a pretty key distinction, and why it’s important to run a diverse array of benchmarks to truly assess a chip’s “relative performance,” to use Apple’s term.
MacRumors also picked up The Verge’s reporting and ran with the headline, “M1 Ultra Doesn't Beat Out Nvidia's RTX 3090 GPU Despite Apple's Charts,” which again leans heavily on that single Geekbench 5 benchmark. I’ve seen people react to these stories by accusing Apple of outright lying about the M1 Ultra’s performance. And to be fair, Apple does open themselves up to criticism when they fail to disclose their testing methodology. We can’t recreate what Apple claims to have seen in their testing because we don’t know what tests they performed.
Having said that though, there’s no way they’re lying because they know it would be easy for people to perform their own benchmarking and see for themselves. So if it’s not lying, and The Verge’s Geekbench scores are legit (and they are) what gives?
What gives is that CPU and GPU performance isn’t trivial to benchmark because different apps and processes take advantage of computer hardware in different ways. I have no doubt that in many workflows, an RTX 3090 is going to beat the M1 Ultra. But I also have no doubt that there will be other workflows where the M1 Ultra comes out on top, and I’ll bet we’re going to see that clearly in the next few days as more in-depth benchmarking is done across a wider variety of workflows and synthetic tests. Meanwhile, to some degree all this sound and fury about GPU comparisons is a distraction. Just look at what else Chin says in her piece:
What I did see was a host of professionals being shocked by how much they could get done on this machine. They were using the same software they use every day, but they were doing things with it they’ve never been able to do before.
Apple’s charts can be frustrating, but Apple’s messaging has never primarily been about “speeds and feeds.” It’s about features and benefits. Take this bit from Apple’s M1 Ultra press release:
M1 Ultra features an extraordinarily powerful 20-core CPU with 16 high-performance cores and four high-efficiency cores. It delivers 90 percent higher multi-threaded performance than the fastest available 16-core PC desktop chip in the same power envelope. Additionally, M1 Ultra reaches the PC chip’s peak performance using 100 fewer watts. That astounding efficiency means less energy is consumed and fans run quietly, even as apps like Logic Pro rip through demanding workflows, such as processing massive amounts of virtual instruments, audio plug-ins, and effects.
In other words “here’s the technical details on what the M1 Ultra has, and here’s what it’s going to mean for the way you use the chip.” The first part of that framing is very important, but the second part is what Apple is most concerned with communicating. That’s why we get charts like the above.