It's 2017 so what do benchmark numbers mean now?
Numbers can mean a lot of things, but not all the things.
Recently we had a look at some benchmark numbers for the Galaxy S8. Or at least we assume they are for the Galaxy S8 since it's not really here just yet. As expected, just by existing these numbers got plenty of people talking about them.
Some conversation about benchmarks is just idle chatter. "Oh, cool! The Snapdragon calculates the "stuff" in a benchmark application about as well as the Exynos" was the majority of the conversation. People talking about them because it's something to talk about while we wait for the Galaxy S8. Tomorrow, another factoid about the Galaxy S8 will conveniently "leak" and we'll talk about the new thing instead.
But some folks get serious about benchmark numbers and consider them an important part of a buying decision. I pulled a dumbass stunt and Tweeted some raw numbers for the Galaxy S8 processors compared to the A10 in an iPhone 7 plus with zero context. I do stupid stuff like that every now and then because I forget how seriously some folks take all this stuff. As a punishment to myself, I'm going to spend a Saturday afternoon talking about benchmarks and iPhones.
For everyone obsessing over which CPU is better for the Galaxy S8. http://pic.twitter.com/28TTXdIDhW
— Jerry Hildenbrand (@gbhil) March 17, 2017
Benchmarks on mobile phones aren't really benchmarking any hardware, at least not the way we think they are. They don't have access to the hardware itself because they are using the operating system's application layer. They have a laundry list of things they have the phone do through the APIs exposed by the operating system, then they calculate how well it did them. There is an intermediary layer through the "brains" behind the operating system that controls the hardware directly. So a benchmark app is benchmarking the hardware through some software. You might have heard iOS people talking about Metal or Android people talking about an NDK. This is the software used to build benchmark apps so they can best use the hardware.
If you buy a phone because you like to run benchmarks, you should probably buy an iPhone.
Apple's intermediary layer is better. Let's throw that out there right where we all can see it. Apple builds their own processor and their own software so that they work really well together. Google has to build software that can be adapted to work with anything. They've done an amazing job and the software that powers an Android phone is a beautiful thing that's incredibly complicated. Something like a benchmarking app using Apple's interface to the hardware automatically has an advantage over Android, no matter who built it, because the interface itself is more streamlined and "faster" with iOS.
You're benchmarking the phone as a whole, not just the processor. When it comes to crunching numbers on each CPU core the iPhone 7 plus does it a lot better.
Let's look at those cores in Apple's A10 processor. That thing is undeniably the best consumer ARM chip ever designed when it comes to raw performance per core. That's because it was designed to do just that. We've talked about ARM architecture before, and the A10 is a great example of how you can scale ARM to do just about anything you want. So are the Qualcomm 835 and the Exynos 8895, they just were designed with different criteria in mind.
We compare them because they all are inside a phone, but Apple is thirsty to build one ARM processor that can power an iPhone, an iPad, and a MacBook. Qualcomm and Samsung build processors to sell to other companies for small mobile devices. They aren't spending money and time to squeeze more from a processor that no other company wants to buy for their phone because it's too expensive.
When you take a tool designed to only do certain things in a certain order and see how "fast" they can be done, the A10 will always win. It should always win, and we should want it to always win. A CPU designed for a 13-inch MacBook needs to perform single core calculations faster than an Exynos 8895. The A10 isn't that CPU, but it is a step in that direction. And Apple is a tech company that we should want to do really cool things to drive tech forward just like we want Samsung or Google or Microsoft to do.
Qualcomm or Samsung could build an ARM processor that is as powerful as the A10, but they have no reason to do it.
My little snip of a benchmark scoreboard that had no context was intended to show that these numbers have little bearing on how great something like a phone is to use. The user experience has little to do with the hardware because the hardware has been good enough for a while now. The innards of a Galaxy S5 or Nexus 7 or Note 4 are more than enough to do the things we expect a phone to do as long as the software is up to snuff. You don't have to take my word on that, just stumble over to XDA where people who don't want or can't afford to buy something newer have built custom software for each.
I'm convinced even mobile VR would be fine if companies cared enough to support Vulkan correctly on their older processors. We'll never know because the companies involved exist to make new things and sell them to us.
What we really see from these benchmarks
What we can take away from these benchmark scores is that the way a CPU core calculates things and works with GPU cores isn't broken. Numbers can be crunched a little faster with newer hardware. The way the CPU cores crunch numbers hasn't been the bottleneck for a long time, so these small differences and increases won't be noticed when you're not running a benchmark application. Newer hardware might be better than last years, and one processor might be better than another, but not in any way that benefits you while you are using it.
The Galaxy S8 will have a new operating system that should provide a better experience than last year's Galaxy S7 did. Many of us here will consider it a better experience than Apple offers with the iPhone 7, while many will feel the opposite. None of this is because of a benchmark score.
No comments: