Printed from http://www.electronista.com

HTC, Asus join Samsung in group of companies cheating on benchmarks

updated 10:04 pm EDT, Wed October 2, 2013

Custom Android branches by each company responsible for boosts

Following reports of Samsung rigging benchmarking scores on its Galaxy S 4 flagship device and Galaxy Note 3 phablet, the entire speed-testing process for Android devices has come under some fire by enthusiast websites. It has been discovered that most, if not all, of the Android device manufacturers -- including Samsung, HTC, and Asus -- are guilty of artificially boosting benchmarking results by identifying the benchmark application by name, with the phone OS branch customized to then have the processor to run, and stay, at maximum speed possible for the duration of the tests.

Devices tested by Anandtech that artificially boost some benchmark apps span multiple processors. Specific models are the Asus Padfone Infinity, the HTC One, the HTC One mini, and the LG G2. Samsung inflates results from the Galaxy S4, the Galaxy Note 3, the Galaxy Tab 3 10.1, and the Galaxy Note 10.1 (2014 edition). Devices proven to not boost speed during identified testing are the Google Nexus 4 and 7, Motorola Mobile devices, the Nvidia Shield gaming device, and Apple iPhones and iPads.

Anand Lal Shimpi of Anandtech notes that "all of this is wrong" and "really isn't worth the minimal effort the OEMs put into even playing these games." Shimpi notes that in most cases, CPU gains are less than five percent, and less than 10 percent on GPU performance benchmarks. "it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors. Whether the OEMs choose to change or not however, we've seen how this story ends. We're very much in the mid-1990s PC era in terms of mobile benchmarks."

In addition, benchmarks are artificial statistics representing maximum speeds of a processor, which bears little relevancy to actual use but see heavy use in marketing materials and apparently matter to some small percentage of buyers. At no point are the processors in the Galaxy S4, for example, pushed greater than rated speeds by the hardcoded exception list -- but the fudging causes the unit to heat up, drain battery and work significantly harder than it would for normal use.

The benchmarks could be viewed as theoretically accurate for the advertised limits of the processor, but with limits placed in real-world applications. As it stands, smartphone processor speeds vary wildly depending on the environment, with both thermal conditions and battery life being primary determiners of clock speed.

Phone manufacturers Broadcom, Huawei, Oppo, and Samsung have ironically recently formed a MobileBench consortium, allegedly to consolidate benchmarking applications and utilities for a more uniform assessment of how devices perform. Being that Samsung has been caught gaming existing benchmarks, it is uncertain what Samsung's contribution to the coalition may be. Perhaps the Korean company will "reform" its code base before it works with the coalition, or the new benchmarking suite may be "contaminated" by its code branch of Android.



By Electronista Staff
toggle

Comments

Login Here

Not a member of the MacNN forums? Register now for free.

toggle

Network Headlines

toggle

Most Popular

Sponsor

Recent Reviews

ActvContent Sync Smartband

Smartbands of all sorts are hitting the market. Some build on the buzz around fitness trackers, while others offer simpler features fo ...

RocketStor 6324L Thunderbolt 2 eSATA bridge

Like it or not, the shift to Thunderbolt is underway. The connection is extremely flexible, allowing for video and data to co-habitate ...

Patriot Stellar Boost XT 64GB USB 3.0 drive

A vast selection of USB memory sticks means that consumers can often find exactly the size drive they need in a configuration that can ...

Sponsor

toggle

Most Commented

 
toggle

Popular News