Method and apparatus for enabling real-time product and vendor identification

• US 9,262,780 B2
• Filed: 01/09/2012
• Issued: 02/16/2016
• Est. Priority Date: 01/09/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Petrou, David, Adam, Hartwig, Garcia-Barrio, Laura, Neven, Hartmut
• Primary Examiner(s): PALAVECINO, KATHLEEN GAGE
• Application Number:
  US13/346,549
• Publication Number:
  US 20130179303A1
• Time in Patent Office: 1,499 Days
• Field of Search: 705 261- 272
• US Class Current: 1/1
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G06Q 30/0623 Item investigation

Abstract

A method and apparatus for enabling dynamic product and vendor identification and the display of relevant purchase information are described herein. According to embodiments of the invention, a recognition process is executed on sensor data captured via a mobile computing device to identify one or more items, and to identify at least one product associated with the one or more items. Product and vendor information for the at least one product is retrieved and displayed via the mobile computing device. In the event a user gesture is detected in response to displaying the product and vendor information data, processing logic may submit a purchase order for the product (e.g., for an online vendor) or contact the vendor (e.g., for an in-store vendor).

Claims

1. A method comprising:

in response to an event defined by a user of the mobile computing device, capturing sensor data via a sensor with a mobile computing device, the sensor data comprising a video data stream of a live view display of the mobile computing device, the video data stream including image data and audio data;

transmitting at least one of the image data or the audio data of the sensor data to a server device;

augmenting the live view display with a product identification received from the server device that identifies at least one product identified from the sensor data within the live view display;

detecting a first user gesture directed towards the product identification augmented in the live view display;

augmenting the live view display with purchase information data, received from the server device for the at least one product, in response to the first user gesture directed towards the product identification;

detecting a second user gesture directed towards the purchase information data; and

transmitting a purchase order to the server device in response to detecting the second user gesture.

2. The method of claim 1, wherein the mobile computing device comprises a user wearable computing device including a head mounted display.

3. The method of claim 2, further comprising:

detecting a user head motion while the purchase information data for the at least one product is displayed on the head mounted display.

4. The method of claim 2, wherein the user wearable computing device comprises a head mounted gaze tracking device, the method further comprising:

detecting the user gazing at the displayed purchase information data for the at least one product past a threshold time value.

5. The method of claim 4, wherein the sensor data includes image data identifying a plurality of items, and the method further comprises:

transmitting gaze direction information to the server device along with the sensor data for identifying the at least one product from the sensor data based on the gaze direction information.

6. The method of claim 1, wherein the purchase information includes review information associated with the at least one product.

7. The method of claim 1, further comprising:

transmitting the location data identifying the physical location of the mobile computing device to the server device along with the sensor data;

wherein the received purchase information includes data identifying vendors proximate to the physical location of the mobile computing device.

15. The method of claim 1, further comprising:

capturing location data identifying a physical location of the mobile computing device, wherein the user defined event is triggered when the physical location of the mobile computing device matches a location identified by the user.

8. A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform a method comprising:

in response to an event defined by a user of the mobile computing device, capturing sensor data via a sensor with a mobile computing device, the sensor data comprising a video data stream of a live view display of the mobile computing device, the video data stream including image data and audio data;

transmitting at least one of the image data or the audio data of the sensor data to a server device;

augmenting the live view display with a product identification received from the server device that identifies at least one product identified from the sensor data within the live view display;

detecting a first user gesture directed towards the product identification augmented in the live view display;

augmenting the live view display with purchase information data, received from the server device for the at least one product, in response to the first user gesture directed towards the product identification;

detecting a second user gesture directed towards the purchase information data; and

transmitting a purchase order to the server device in response to detecting the second user gesture.

9. The non-transitory computer readable storage medium of claim 8, wherein the mobile computing device comprises a user wearable computing device including a head mounted display.

10. The non-transitory computer readable storage medium of claim 9, the method further comprising:

detecting a user head motion while the purchase information data for the at least one product is displayed on the head mounted display.

11. The non-transitory computer readable storage medium of claim 9, wherein the user wearable computing device comprises a head mounted gaze tracking device, the method further comprising:

detecting the user gesture comprises detecting the user gazing at the displayed purchase information data for the at least one product past a threshold time value.

12. The non-transitory computer readable storage medium of claim 11, wherein the sensor data includes image data identifying a plurality of items, the method further comprising:

transmitting gaze direction information to the server device along with the sensor data for identifying the at least one product from the sensor data based on the gaze direction information.

13. The non-transitory computer readable storage medium of claim 8, wherein the purchase information includes review information associated with the at least one product.

14. The non-transitory computer readable storage medium of claim 8, the method further comprising:

transmitting the location data identifying the physical location of the mobile computing device to the server device along with the sensor data;

wherein the received purchase information includes data identifying vendors proximate to the physical location of the mobile computing device.