The popular Vellamo benchmark suite got a pretty big update today, bringing new features and a really nice UI to the popular software from Qualcomm. We talked with Sy Choudhury, senior director of product management at Qualcomm about it earlier in the week, and while they haven't convinced me that benchmarks are the perfect way to measure the performance of our Android phones and tablets just yet, they did answer a couple questions I've had and have me looking at the issue again. That's no small feat -- I'm pretty stubborn.
First, let's talk about the app itself. A benchmark tool doesn't have to look ugly, and Vellamo really does things up nicely. The main screen of the app lets you chose between HTML 5 based browser benchmarks, "Metal", as in hardware measuring, benchmarks, or a set of extras like tools to measure touch screen response, the Octane benchmark, and "advanced" video benchmarking tools.
Benchmarking is easy, just tap one of the circular icons and let it run. When finished, you can compare your scores with others, or look a bit deeper to evaluate each test individually. You can also long press on the main benchmark icon and select individual tests. The settings allow you to run multiple tests to get the best average (recommended), email the comprehensive set of results to you when finished, and turn the tutorial on and off.
Most importantly, a link to a full description of each test is provided right in the About section. It's no use to run these things if you have no clue what it all means, and Qualcomm does a great job explaining each test in both the HTML 5 and the Metal category, as well as the extras. This is what makes Vellamo my recommendation for folks who are fiddling with software and want to test the result on their phone. Sure, Qualcomm can send me a well written pdf file (they did, and I thank them for it) about each test but putting it out there for everyone using the app to read is pretty damn smart. Excellent move, Qualcomm.
As mentioned, Mr. Choudhury from Qualcomm took a few minutes to talk with us about Vellamo. He walked me through each of the tests and tools, but more importantly they shared a little about why they made the app and how they use it. Vellamo started as an internal testing suite for engineers and developers at Qualcomm. Some of their customers (folks like Samsung and HTC) requested the tools for themselves. Qualcomm obliged, and then decided since they had to build a stand alone app anyway that they would release it to the public. While it's not exactly the same app the fellows designing Qualcomm chips is using, it does provide the same exact tools they use to test performance. Learning about the how and why is always cool, and talking with Mr. Choudhury let me know that they take performance of our mobile devices pretty seriously at Qualcomm -- at least as seriously as we do.
Of course, I asked about any weighting or bias towards Qualcomm-powered devices -- a popular theory around the Internet. The tests used in Vellamo are industry standards, and are unmodified. The reason Qualcomm devices often score better on Vellamo is because, as mentioned, it's the tool they are using in-house to improve their chips. The engineers address issues they may see in Vellamo, and that means that the end product will score well in Vellamo. There are no sneaky software edits to make Snapdragons outperform other chips.
I'm still not convinced that software based benchmark tools are a good way to compare the way one device runs versus another. There are too many variables, and nobody worries about the raw data given and instead uses the big number at the end as an Internet manhood measuring tool. But if you want to see how flashing different ROMs or making other software edits affects the phone or tablet in your hands, Vellamo is a pretty damn good way to do it. It's free in Google Play, grab it from the link above ad check out the product video after the break.
