updated 03:05 pm EDT, Wed June 18, 2014
Dedicated phone button allows for product search, keeps history for future purchases
During the unveiling of Amazon's most recent piece of tech, the Fire Phone, Jeff Bezos introduced a feature called Firefly. The technology will allow phone users to search for items through the camera by holding a button on the phone. The accompanying app will pull up an Amazon listing for a product based on what it sees, rather than relying on just a barcode.
On stage, Bezos went through a table full of items to show off the new feature. Books, CDs, video games and food were detected, bringing up a small Amazon listing at the bottom of the screen with its rating. Bezos was able to find Nutella, FIFA 14 for Xbox 360, soup and a film listing by holding the dedicated button on the left side of the Fire Phone. The Amazon shopping app for iPhone has had a similar feature (locating items by use of the camera or scanning a barcode) for some time.
However, Firefly can pick up more than physical products. If a CD cover is seen, songs from it can be played through Amazon or other applications. Firefly can also determine the specific scene of a television show that is being viewed just by listening. Demoing Game of Thrones, Firefly was able to determine the episode being viewed, listing who was in the scene. Extended information was returned, giving users the option to purchase the episode or look at an IMDB listing.
The listening feature extends to music as well. Taking a little over a second, Firefly listens to the song and brings up a page for it. Users are then given an option on how to access it. Amazon's music store and iHeartRadio are included in the listing.
Bezos said that Firefly can recognize 100 million items during the presentation. Non-consumer items can make use of the app, returning different information than purchase listings. Scanning art will access a Wikipedia entry on the piece. Firefly can also pick out phone numbers, and check to see if they are valid. It can also be used for URLs and QR codes.
Each Firefly scan, as demonstrated on stage, is quick. Most items are recognized in less than a second. As it locates the object, little glowing dots appear on the screen of Fire Phone to depict that the item is being searched for. The phone uses cloud computing and "semantic boosting" to do the work. Images are cleared of irrelevant information in order to submit the smallest amount of data possible for detection.
Firefly will keep a history of all of the items that a user has scanned. By looking through the history, consumers will be able to tap on the item and purchase it. Should the technology become widespread, store owners may object to customers using the device in their stores, as the practice of "show rooming" is already driving merchants out of business, but the practice is generally less blatant than Amazon's new approach.
An SDK is available for Firefly from Amazon now. MyFitnessPal and Vivino have already used the technology in their apps. With the SDK, app developers will be able to use the same text, audio, image recognition and content databases. Third-party developers will be able to add in custom actions.