Introducing the 2012 Mobile Benchmark Matrix
by Jarred Walton on January 6, 2012 8:02 PM EST2012: Meet Our New Mobile Benchmark Suite
Testing computer hardware can be a difficult process. On the one hand there’s a desire for more information and benchmarks, and on the other hand there’s a desire for timely reviews. Our goal at AnandTech has always been to deliver the most comprehensive reviews possible, and while we strive to timeliness there are occasions where additional testing or questions may delay a review. Ultimately, there’s a balancing act that needs to be maintained, and over time we periodically refresh our review suite and testing methodologies.
With 2012 now here, we’re launching a new suite of benchmarks for our laptop reviews. We'll also have the results from our first laptop using the new tests, courtesy of ASUS' G74SX. Some of the tests have already been in use for a while and others are brand new. In order to provide a single location with a list of our benchmarks and testing procedures, we have put together this short overview. We plan on using the following test suite throughout 2012, and while it’s possible we will add some benchmarks, we don’t have any plans to stop using any of the following at least for the next year.
General Performance Tests
Starting with our general tests, all of these have been in use for several months at least, with many tests dating back to 2010 and earlier. We’ll continue to use the full PCMark 7 suite, PCMark Vantage (x64), Graysky’s x264 HD encoding test, Cinebench 11.5, 3DMark06, 3DMark Vantage (Entry-Level and Performance defaults), and 3DMark 11. We’ll also continue with our battery life tests (now with Internet Explorer 9 in place of IE8) and LCD tests. So for most areas, our test suite remains largely unchanged—we’re finally dropping Cinebench 10, but that’s about it.
As we’ll mention in the conclusion, we’re willing to add some additional general performance benchmarks if there are any specific requests. One of the difficult things to quantify with modern PCs is how fast they are in the things most people do on a regular basis. Part of the problem is that most PCs from the past three or four years are all “fast enough” for generic tasks like surfing the web—if you’re actually reading the content of web pages rather than just repeatedly loading a complex page, I’m not sure most users would notice the difference between a 2GHz Core 2 Duo or Athlon X2 laptop and a quad-core i7-2760QM. This is why battery life is such an important element, as where many wouldn’t notice the difference between a web page loading in two seconds and a web page loading in one second, they’re far more likely to notice two hours of battery life versus four or eight hours. Anyway, let us know if you have other mobile benchmarks you’d like us to consider.
With that out of the way, we’ll save the next page for the major changes: our updated gaming suite.
Facebook
Twitter
RSS
48 Comments
View All Comments
losonn - Saturday, January 7, 2012 - link
A few thoughts for someone who went out and bought an XPS 15 L502x immediately after reading your L501x / L502X reviews.I've got to agree with an earlier poster that I always skim the mobile reviews here at AnandTech, but frequently find myself referring to laptop reviews elsewhere (including notebookcheck.net) for better formatted specs, more helpful product pics, more dense information, specific noise / heat info, power adapter weight / pics, build quality opinions, etc.. Your current article format is perfect for pro hardware and new tech reviews (7970 review, vertex 3 review, etc.) but your consumer / mobile reviews could be a lot more dense ( http://www.notebookcheck.net/Review-Samsung-305U1A... ) or at least a little bit less ugly (see http://www.theverge.com/2012/1/4/2677801/hp-envy-1... ).
This may have more to do with your page layout formatting than anything else though, even on my 1080p screen I have to do a -lot- of scrolling on your site and the dropdown navigation for your articles is a bit clunky. This site has a solid reputation and I trust the recommendations of Anand more than any other tech site, but you could really use a facelift that was a little bit less all-business-all-the-time a la the asus transformer prime :)
As for benchmarks a sleep, hibernate, wake from sleep, wake from hibernate could be really helpful.
coolhardware - Sunday, January 8, 2012 - link
Nice points here.+Notebookcheck definitely does a great job. :-)
-To me though, theverge layout doesn't read/flow nearly as nicely as Anandtech does right now... :-(
colonelclaw - Saturday, January 7, 2012 - link
Ok, it's very niche, but if you would like to try a real ball-busting CPU test I would recommend a combination of 3DSMax 2012 and VRay. It's the most popular combination in the 3D world at the moment and nothing sucks the life out of a PC better than a VRay render.Plus these days VRay is available for Max, Maya, Sketchup, Rhino, Softimage and probably more. It also runs on OSX under the apps available for that platform. Something to think about.
RoninX - Sunday, January 8, 2012 - link
I'd like to request a benchmark for battery life when gaming. This wouldn't need to be the full set of gaming tests, but just one or two games run at value, mainstream, and enthusiast settings.I own a high-end gaming desktop for gaming at home, but I also do a lot of business travel, and I like to having a laptop that can run games during long waits at the airport. Right now, I have a Dell XPS 15, which does pretty well, but in the future, I'd be interested to know the tradeoffs in battery life and performance for dedicated gaming laptops (e.g. Alienware) as well as high-end mainstream laptops (e.g. XPS).
I realize that laptop battery life is pretty poor across the board when running demanding 3D games, but for me, there's a big difference between 90 minutes (which I get from my XPS 15), and say, 20 minutes. With a spare battery, the former gives you 3 hours of gaming time -- more than enough for typical flight delays.
JarredWalton - Sunday, January 8, 2012 - link
Besides tripling the number of battery life tests (ouch!), there are other factors to consider. First, you need a "game test" that's repeatable. If you play Batman for a couple hours on a laptop, and then play it again for a couple hours, I'm not sure the load will be the same. Then again, I doubt that the difference between Value/Mainstream/Enthusiast and even different games will be much -- they all activate the GPU and put a decent load on the CPU, so they're pretty much the worst-case scenario.How about this one: start battery life timer, open Skyrim save, and let the laptop just sit there until it runs dead. (Note that Skyrim will have the camera start circling the player in third person mode after a minute or two of inactivity.) That's about as consistently repeatable as I can come up with, though not necessarily a realistic test case. I would also be willing to check battery life on one laptop at our three detail settings to verify whether there's a difference in battery life or not. I might also check out some other games to see if they show variance in battery life (and if I can find a good looping test). Let me see if we can come up with something reasonable -- and we'd also want to test performance on battery power, as most higher end GPUs really clamp down on maximum clocks when on DC power.
RoninX - Sunday, January 8, 2012 - link
That Skyrim test sounds good to me, and testing performance on battery power would also be very useful.Thanks!
joelypolly - Monday, January 9, 2012 - link
Would be nice to have a web based comparison table so you can compare things like viewing angles, general specs and performance.JamesAnthony - Thursday, January 12, 2012 - link
I was wondering if you would be able to add to the video tests for games more notebooks with Nvidia Quadro and ATI FireGL video cards, as Dell specifically & several other vendors tend to use those cards as the only option available in business level or professional level laptops & it would be nice to know how well you could expect your $1500 Business laptop $3,000 CAD laptop to play some games when you get home with it.For Example
AMD FirePro M8900 Mobility Pro
NVIDIA Quadro 3000M
NVIDIA Quadro 4000M
NVIDIA Quadro 5010M
AMD FirePro M5950 Mobility Pro
NVIDIA Quadro 1000M
NVIDIA Quadro 2000M
NVIDIA NVS 4200M