Augmented reality information system

• US 9,424,472 B2
• Filed: 11/26/2012
• Issued: 08/23/2016
• Est. Priority Date: 11/26/2012
• Status: Expired due to Fees

• Current Assignee: eBay Inc.
• Original Assignee: eBay Inc.
• Inventors: So, Wai Kei
• Primary Examiner(s): BEUTEL, WILLIAM A
• Application Number:
  US13/685,479
• Publication Number:
  US 20140146082A1
• Time in Patent Office: 1,366 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal

Abstract

In various example embodiments, a system and method for providing information in an augmented reality display are provided. In example embodiments, a continuous stream of image data captured by a client device is received. An object within the continuous stream of image data is identified. Based on an identification of the object from the continuous stream of image data, a search for information related to the object is performed. A result is determined by filtering the information related to the object. The result is formatted to be displayed over a real-time image of the object on the client device.

Claims

1. A method comprising:

initially receiving a continuous stream of image data captured by a client device;

deriving an initial identification of an object within the continuous stream of image data, the initial identification of the object derived from the continuous stream of image data comprising a category of the object;

based on the initial identification of the object derived from the continuous stream of image data, searching for information related to the category of the object;

determining, by a processor of a machine, a result to return to the client device by filtering the information related to the category of the object, the result comprising generic information corresponding to the category of the object;

formatting the result to be displayed with a real-time image display of the object on the client device;

further receiving the continuous stream of image data after an event, the continuous stream of image data after the event comprising additional image data regarding the object, the additional image data providing more visual details and features of the object than the image data initially received;

using the additional visual data, automatically refining the identification of the object, the refined identification indicating a specific type within the category of the object;

automatically, without human intervention, performing a further search for more information in response to the automatically refined identification of the object, the more information comprising information more specific than the generic information and pertaining to the specific type within the category; and

refining the result returned to the client device based on the further search, the refined result comprising the more specific information.

2. The method of claim 1, further comprising:

receiving location data captured by the client device, the location data including an orientation of the client device relative to the object and direction of the client device; and

refining the identification of the object using the location data.

3. The method of claim 1, further comprising:

receiving time data from the client device, the time data indicating a time that the continuous stream of image data is captured,

wherein the result comprises local store information based on the time that the image data is captured being during business hours and online merchant information based on the time that the image data is captured being after business hours.

4. The method of claim 1, further comprising receiving location data from the client device, wherein the filtering of the information comprises providing local store information near a location of the client device that carries the object.

5. The method of claim 1, wherein determining the result comprises:

accessing a user profile of a user of the client device; and

using attributes of the user profile to filter the information.

6. The method of claim 5, wherein the determining comprises providing more or fewer details for the object based on the attributes of the user profile.

7. The method of claim 1, wherein

the refined result comprising the more specific information comprises a product listing for the specific type within the category.

8. The method of claim 1, further comprising:

tracking session data associated with the object from a plurality of users; and

determining whether to cache information for the object based on the tracked session data, information being cached for objects that require more information due to being less popular.

9. The method of claim 1, wherein a plurality of objects are identified within the continuous stream of image data, the method further comprising:

evaluating relevance of each of the plurality of objects using at least one of user preference data, location data, or tracked data; and

determining for which one or more of the plurality of objects to return the result based on the relevance.

10. A machine-readable storage medium having no transitory signals and storing instructions which, when executed by the at least one processor of a machine, cause the machine to perform operations comprising:

initially receiving a continuous stream of image data captured by a client device;

deriving an initial identification of an object within the continuous stream of image data, the initial identification of the object derived from the continuous stream of image data comprising a category of the object;

based on the initial identification of the object derived from the continuous stream of image data, searching for information related to the category of the object;

determining a result to return to the client device by filtering the information related to the category of the object, the result comprising generic information corresponding to the category of the object;

formatting the result to be displayed with a real-time image display of the object on the client device;

further receiving the continuous stream of image data after an event, the continuous stream of image data after the event comprising additional image data regarding the object, the additional image data providing more visual details and features of the object than the image data initially received;

using the additional visual data, automatically refining the identification of the object, the refined identification indicating a specific type within the category of the object;

automatically, without human intervention, performing a further search for more information in response to the automatically refined identification of the object, the more information comprising information more specific than the generic information and pertaining to the specific type within the category; and

refining the result returned to the client device based on the further search, the refined result comprising the more specific information.

11. The machine-readable storage medium of claim 10, wherein the operations further comprise:

receiving location data captured by the client device, the location data including an orientation of the client device relative to the object and direction of the client device; and

refining the identification of the object using the location data.

12. The machine-readable storage medium of claim 10, wherein the operations further comprise:

receiving time data from the client device, the time data indicating a time that the continuous stream of image data is captured,

wherein the result comprises local store information based on the time that the image data is captured being during business hours and online merchant information based on the time that the image data is captured being after business hours.

13. The machine-readable storage medium of claim 10, wherein the operations further comprise receiving location data from the client device, wherein the filtering of the information comprises providing local store information near a location of the client device that carries the object.

14. The machine-readable storage medium of claim 10, wherein

the refined result comprising the more specific information comprises a product listing for the specific type of the object in the category.

15. A system comprising:

one or more processors configured to perform operations comprising;

initially receiving a continuous stream of image data captured by a client device;

deriving an initial identification of an object within the continuous stream of image data, the initial identification of the object derived from the continuous stream of image data comprising a category of the object;

based on the initial identification of the object derived from the continuous stream of image data, searching for information related to the category of the object;

determining a result to return to the client device by filtering the information related to the category of the object, the result comprising generic information corresponding to the category of the object;

formatting the result to be displayed with a real-time image display of the object on the client device;

receiving the continuous stream of image data after an event, the continuous stream of image data after the event comprising additional image data regarding the object, the additional image data providing more visual details and features of the object than the image data initially received;

using the additional visual data, automatically refining the identification of the object, the refined identification indicating a specific type within the category of the object;

automatically, without human intervention, performing a further search for more information in response to the automatically refined identification of the object, the more information comprising information more specific than the generic information and pertaining to the specific type within the category; and

refining the result returned to the client device based on the further search, the refined result comprising the more specific information.

16. The system of claim 15, wherein the operations further comprise receiving location data captured by the client device, the location data including an orientation of the client device relative to the object and direction of the client device and to refine the identification of the object using the location data.

17. The system of claim 15, wherein the operations further comprise filtering information based on a time that the image data is captured.

18. The system of claim 15, wherein the operations further comprise filtering the information based on a location of the client device.

19. The system of claim 15, wherein the operations further comprise accessing a user profile of a user of the client device and use attributes of the user profile to filter the information.

20. The system of claim 15, wherein the operations further comprise:

tracking session data associated with the object from a plurality of users; and

determining whether to cache information for the object based on the tracked session data.

21. The system of claim 15, wherein the operations further comprise evaluating relevance of a plurality of objects that are identified within the continuous stream of image data using at least one of user preference data, location data, or tracked data, and determine for which one or more of the plurality of objects to return the result based on the relevance.