updated 10:04 pm EDT, Wed October 2, 2013
Custom Android branches by each company responsible for boosts
Following reports of Samsung rigging benchmarking scores on its Galaxy S 4 flagship device and Galaxy Note 3 phablet, the entire speed-testing process for Android devices has come under some fire by enthusiast websites. It has been discovered that most, if not all, of the Android device manufacturers -- including Samsung, HTC, and Asus -- are guilty of artificially boosting benchmarking results by identifying the benchmark application by name, with the phone OS branch customized to then have the processor to run, and stay, at maximum speed possible for the duration of the tests.
Devices tested by Anandtech that artificially boost some benchmark apps span multiple processors. Specific models are the Asus Padfone Infinity, the HTC One, the HTC One mini, and the LG G2. Samsung inflates results from the Galaxy S4, the Galaxy Note 3, the Galaxy Tab 3 10.1, and the Galaxy Note 10.1 (2014 edition). Devices proven to not boost speed during identified testing are the Google Nexus 4 and 7, Motorola Mobile devices, the Nvidia Shield gaming device, and Apple iPhones and iPads.
Anand Lal Shimpi of Anandtech notes that "all of this is wrong" and "really isn't worth the minimal effort the OEMs put into even playing these games." Shimpi notes that in most cases, CPU gains are less than five percent, and less than 10 percent on GPU performance benchmarks. "it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors. Whether the OEMs choose to change or not however, we've seen how this story ends. We're very much in the mid-1990s PC era in terms of mobile benchmarks."
In addition, benchmarks are artificial statistics representing maximum speeds of a processor, which bears little relevancy to actual use but see heavy use in marketing materials and apparently matter to some small percentage of buyers. At no point are the processors in the Galaxy S4, for example, pushed greater than rated speeds by the hardcoded exception list -- but the fudging causes the unit to heat up, drain battery and work significantly harder than it would for normal use.
The benchmarks could be viewed as theoretically accurate for the advertised limits of the processor, but with limits placed in real-world applications. As it stands, smartphone processor speeds vary wildly depending on the environment, with both thermal conditions and battery life being primary determiners of clock speed.
Phone manufacturers Broadcom, Huawei, Oppo, and Samsung have ironically recently formed a MobileBench consortium, allegedly to consolidate benchmarking applications and utilities for a more uniform assessment of how devices perform. Being that Samsung has been caught gaming existing benchmarks, it is uncertain what Samsung's contribution to the coalition may be. Perhaps the Korean company will "reform" its code base before it works with the coalition, or the new benchmarking suite may be "contaminated" by its code branch of Android.