AMD’s hotly anticipated Ryzen 9000 desktop processors launched earlier this month, and well, the gaming performance improvements seen by independent reviewers weren’t as hot as expected – or as hot as AMD attested to in early marketing statements. So what happened?
It’s been a long, winding road as reviewers and AMD alike track down what happened. Last night, AMD released a community post pointing the finger at a variety of issues: A difference in the Windows mode used for testing, VBS security settings, how rival Intel systems were configured, and – as ever – the specific games benchmarked.
Phew. Today, David McAfee (who leads AMD’s client channel segment) joined us for a special edition of The Full Nerd to untangle the mess and explain exactly what happened.
As you read the blog post (and listen to David’s chat), it becomes clear: A big part of this stems from how AMD tested versus how reviewers tested, and not just the games chosen. Here’s a tidbit from AMD’s post:
“The ‘Zen 5’ architecture incorporates a wider branch prediction capacity than prior ‘Zen’ generations. Our automated test methodology was run in ‘Admin’ mode which produced results that reflect branch prediction code optimizations not present in the version of Windows reviewers used to test Ryzen 9000 Series. We have a further update on accessing this performance for users below.”
PC users will be able to tap into those performance improvements when Windows 11’s annual feature update, dubbed “24H2” for now, starts to roll out later this year – its branch prediction optimizations mimic the Super Admin changes AMD tested with. (The always-excellent Wendell of Level1Techs already has a nice and nerdy analysis video about the patch, which is already available to Windows 11 Insider preview testers.)
But nobody should be using Super Admin mode to run games, and McAfee stressed that in the interview. Why does AMD do so, and why were the differences so pronounced this generation? It’s all about AMD’s established, automated testing framework, which is necessary to be able to tests products at scale across multiple hardware configurations. It created a blind spot, McAfee said.
“Historically, when our automation framework was built, the difference between performance in super admin mode and what you’d consider user mode when testing was negligible – there was very little difference there. That has changed over time, and quite honestly, it was a change we were blind to. When we collected this data, we didn’t see that difference because for several generations we were using that same framework to collect data through out automation suite. So clearly an oversight on our part, clearly something where we were a bit disconnected from how reviewers were testing these applications and how users were playing games on their systems. And that’s been corrected in the way that we collect data for our products going forward.”
Those changes that AMD was blind to are what bit its initial marketing numbers for Ryzen 9000.
“What we tested for Computex was running current shipping build of Windows, 23H2, but it was run in the Super Admin mode that bypasses a number of the security layers in Windows and gets closer to the bare metal performance. It’s a case of an oversight of how much a performance delta there was in those two operating modes.”
Willis Lai/Foundry
Willis Lai/Foundry
Willis Lai/Foundry
McAfee – and AMD’s community post – also touches on how game selection and even game scene selection can influence benchmark results, a fact alluded to by Hardware Unboxed’s continued excellent coverage of this saga. Whether you use a built-in benchmark or manually run through in-game scenes for testing matters, as McAfee elaborated.
“Even within an individual game, the balance of the system where portions of a game which lean heavier on the CPU versus lean heavier on the GPU result in massive differences in relative performance between product A and product B. I think you can look at many games where you can look at significant differences between the two products, and move to another scene where you effectively get clarity between the two.”
You can hear McAfee talk much more about how AMD chose its Ryzen 9000 gaming benchmarks, including how the company was surprised by some reviewer tests this generation, around the 30 minute mark of the interview. It’s a fascinating listen – you can tell there’s clear thought and intent going into the process, and it’s simply not a case of “benchmark games that show us in the best light,” as some community members have been implying across the web.
Willis Lai / Foundry
Willis Lai / Foundry
Willis Lai / Foundry
One final bit to touch on before I encourage you to go watch the full, insightful interview. Towards the end, Gordon asks simply: “Is AMD blaming reviewers here? Did hardware reviewers screw this up?”
“Not at all,” McAfee said.
“At the end of the day, there were a series of decisions that AMD made, that differed from how reviewers were testing, that led to different conclusions,” McAfee said. “This is not saying that reviewers did anything wrong, or that there’s anything in the process of how these reviews were done that was incorrect. We were as puzzled by some of the results that reviewers were seeing as the Ryzen fans are out there in the world, and that’s really what led us to this sort of multi-layered conclusion that there’s a number of factors going on here that created that separation.
This is no shade whatsoever on reviewers and how they tested, this is completely on things that we didn’t really understand as a part of the way that we tested and configured our products, and how that differed from both reviewer game suites moving in a slightly different direction, as well as how they tested the product versus how we did. It’s not a criticism of reviewers or how they test, simply significant differences in setup and configuration that got us from point A to point B and I think we understand that now.”
And you’ll understand it too once you watch our full AMD Ryzen interview with David McAfee. This recap is just the tip of the iceberg – check out the entire talk for many more interesting details, such as why AMD configured Intel test systems the way they did, how Windows 11’s VBS affects all this, whether Windows 10 users will also get branch prediction optimizations, why AMD didn’t delay Ryzen 9000’s launch given this confusion (it makes sense!), core parking, and a whole lot more.
Be sure to subscribe to our new, dedicated Full Nerd channel on YouTube for special interviews and nerdy PC podcasts like this week-in and week-out. We’ve split the podcast off from the main PCWorld channel so click that button if you don’t want to miss out!