Adding Vellamo to our Mobile Benchmark Suite - Six Android Phones Testedby Brian Klug on November 16, 2011 10:43 AM EST
Real web browsing performance is a unique combination of system performance, the Android browser itself, and what contributions or customizations (if any) the OEM has made in the shipping browser build. This is to say nothing of the numerous third party browsers out there. Qualcomm's Innovation Center recently made public a tool for gauging overall browser performance that it has used for a while to checkpoint and gauge performance in house and in collaboration with OEMs. We've used a subtest or two from it in a few reviews already, and it's named Vellamo.
Left to right: T-Mobile SGS2, Motorola Atrix, AT&T SGS2, HTC Sensation, LG Optimus 3D, Xiaomi Mi-One
First off, Qualcomm sent us five dual-core smartphones, almost all of which we've seen and tested before. Just to be sure, we verified that nothing has been tampered with and that all were running the latest shipping OEM ROM and started fresh with a complete reset. We also added another, the T-Mobile SGS2 bringing that total to six devices.
|T-Mobile SGS2||Motorola Atrix||AT&T SGS2||HTC Sensation||LG Optimus 3D||Xiaomi Mi-One|
|SoC||1.5 GHz APQ8060||1.0 GHz Tegra 2||1.2 GHz Exynos 4210||1.2 GHz MSM8260||1.0 GHz OMAP4430||1.5 GHz MSM8260|
|RAM||1 GB LPDDR2||1 GB LPDDR2||1 GB LPDDR2||768 MB LPDDR2||512 MB LPDDR2||1 GB LPDDR2|
|OS||Android 2.3||Android 2.3||Android 2.3||Android 2.3||Android 2.2||Android 2.3|
The selection we've made isn't meant to be totally comprehensive, but gives an idea for just how much performance varies from one OEM to another, largely as a function of changes made to the browser on their respective versions of Android. These devices are all dual core and what I'd personally consider of the latest generation (dual core, Android 2.3, e.t.c.). We're going to be updating our Smartphone Bench category soon (something I've been behind on, as of late) with Vellamo scores for devices as well.
If you've been reading our reviews, basically every device except for the T-Mobile SGS2 (which we're going to review shortly) and the Xiaomi Mi-One should be relatively familiar. The Mi-One is from a Chinese OEM we haven't seen devices from before called Xiaomi, but is a surprisingly decent performer and has one of the most unique (and surprisngly speedy) skins I've ever seen on Android.
Vellamo Test Breakdown
First among these tests is See the Sun Canvas, which renders increasingly demanding stages of a 2D rotating butterfly in an HTML5 <canvas> element. The first stage starts out with cubes, then moves to circles, then circles of different color, and finally alpha-blended for the maximum load.
The Xiaomi Mi-One is a new device for us, but delivers strikingly fast performance thanks to the work that Xiaomi has done to the stock Android browser. I found the SGS2's backing store pretty decent as well, but you can see how much difference there is in performance between the two despite packing the same 1.5 GHz dual core snapdragon (APQ8060 and MSM8260 differ solely in baseband - APQ lacks one).
The next test is Deep Sea Canvas which tests almost the same parameters as See the Sun, except tests are non-sequential here and are an attempt to fully load the platform.
Interestingly enough, Exynos 4210 comes out on top of the pack.
Next up is Aquarium canvas which tests the number of HTML5 canvas objects that can be rendered simultaneously. In addition, this test runs in a native Android Java implementation for comparison with the HTML5 <canvas> implementation in the browser.
Vellamo calculates a score for each subtest alongside FPS and dropped frames, and we've chosen to report FPS wherever possible, but this test makes more sense being reported in base score. Interestingly enough the Sensation 4G edges out the faster T-Mobile SGS2.
Pixel Blender is next which draws multiple images over a static background and applies alpha blending. Vellamo calculates the graphics throughput of the browser in pixels per second to arrive at memory bandwidth for blitting and blending.
Again there's a pretty shocking disparity between Samsung's browser and the Xiaomi, we're reporting the base score here as well.
Where I feel Vellamo really shines however is this next suite, which we've shown scores from before, and this is the User Experience Test category. This is the area that until now has proven challenging to qualitate without using video or other subjective measures - things like scrolling performance and responsiveness.
I have to admit that Vellamo's numbers here closely mirror my own subjective scrolling performance perceptions when using the respective devices. The three scrolling tests represent a way to finally codify one of the more elusive (and yet easily perceptible) browser performance aspects.
The final test is a networking loader suite which loads a page stored locally and times how long it takes in milliseconds. This is done in both hot and cold mode (with the caches cleared and not cleared) and is designed to test local performance.
Finally Vellamo also reports a total score which is a summation of all the base scores from each subtest.
The total score subjectively lines up with where I would mentally rank the devices - we've seen how competent Samsung's browser optimizations are (especially their browser's backing store), and how much sometimes HTC's Sense tends to weigh the platform down. Motorola's browser on the Atrix and LG's on the Optimus 3D take a more stock approach, and the Xiaomi surprises us with some serious optimizations from a company I doubt many in the US have heard of.
We're going to be adding (and simultaneously giving the same treatment) a few more benchmarks in preparation for a revamped testing suite for 2012 and an updated smartphone Bench as well.
Post Your CommentPlease log in or sign up to comment.
View All Comments
weiln12 - Wednesday, November 16, 2011 - linkI'm really curious how the performance changes when these phones are tested running stock (which in my experience is horrificly crap) and then also with MIUI and CM? In my experience with multiple devices, performance improved TREMENDOUSLY with CM7, to the point it was almost like an upgrade to a new device. Take the difference between the MI-One and the Sensation...they're the same chip, but with VASTLY different scores. Kinda makes you go "hmmm....".
I think by having a reputable site like AnandTech do testing for these different devices it can help highlight the affect stock ROM skins and changes are having on Android. The skins and inconsistencies are the single reason I went back to an iPhone. It's interface is consistent, and performance is consistent (if not near as fast as CM). The whole experience is just consistent...which to me makes a big difference.
I think having numbers behind the decision to differentiate their devices would be a huge benefit to consumers, and would be very eye-opening to a lot of people.
Just my thoughts.
cbutters - Wednesday, November 16, 2011 - linkI have heard that a lot of the chinese phones are using CM7 base code right off the bat in many of their android phones, which may even explain the performance difference of the xiaomi. The good news is that most of these phones can be rooted and have custom ROMs which will improve performance. But like you are saying... wouldn't it be nice if we didn't have to rely on third parties to utilize the performance of the device.
I think that the fracturing of android via carrier and manufactuer customization will come to an end in the next few years. I get the feeling that even with android 4.0 (ice cream sandwich) google is attempting to bring the interface together in a coherent and beautiful way that most carrier will choose to adopt instead of customizing their own frontend applications (sense, touchwiz, etc...)
weiln12 - Thursday, November 17, 2011 - linkI certainly hope you're right. I do wish that Google was able to exert more influence over the devices, if only for consistency's sake. I don't want all the random apps that I can't uninstall, I don't want Sense or some other resource hog. If the user wants to add that, fine, but it certainly shouldn't come from the factory that way.
ThisWasATriumph - Wednesday, November 16, 2011 - linkJust ran this on my Captivate with CM7.2 and got 846! Only single core and hanging with the big boys. See what a little optimization can do for a 1.5 year old phone?
mortyg99 - Wednesday, November 16, 2011 - linkjust tried it on my t-mobile g2 running cm7.1 and got 636
nnitko - Wednesday, November 16, 2011 - linkI have HTC Desire with MIUI (2.3.7) @ 1GHz and my overall score is 680 which is better than dual core Atrix 4G. I would say that's a damn good score for such an old phone. Good job MIUI!
s44 - Wednesday, November 16, 2011 - linkLots of people are running out of time to change the original AT&T SGS2 for the Skyrocket... any quick pre-review summary on if the Snapdragon in the Hercules gets in the way in any non-game-use?
TooYellow - Wednesday, November 16, 2011 - linkThis is a good comparison of the stock phone experience as a whole but it does not give good data on the true power of the platforms.
I realize that only a small portion of the population uses a custom ROM for their phones. But standardizing the OS/skin across the phones will make the performance number more relevant to hardware benchmarks. Of course build optimization varies across devices running CM (or any AOSP ROM).
I have a Sensation with Cyanongenmod 7 and it's leagues faster than Sense.
Exodite - Wednesday, November 16, 2011 - linkMy stock, orginal Xperia arc using a 1.0GHz MSM8255 and firmware 4.0.2.A.0.42 got a result of 845.
Not too bad.
georgekn3mp - Wednesday, November 16, 2011 - linkMy HTC Rezound which I just got on release day from VZW on 11/14/2011. Dual core...MSM 86xx 1.5ghz with 1GB RAM running Sense 3.5... but the 1280x720 S-LCD resolution MAY slow the graphics a bit on the Adreno 220...I love this phone! And the iBeats do sound great.
See the Sun Canvas: 56.66
Deep Sea Canvas: 87.68
Aquarium Canvas: 83.59
Pixel Blender: 104
Surf Wax Binder: 127.02
Sun Spider: 2316
V8 Benchmark: 696
Ocean Flinger: 49.46
Image Flinger: 18.1
Text Flinger: 36.54
Networking Loader: 82.72
Overall Score: 883
6th overall of 31 tested running 2.3.4