Augmented reality information system

• US 10,216,997 B2
• Filed: 07/26/2016
• Issued: 02/26/2019
• Est. Priority Date: 11/26/2012
• Status: Active Grant

• Current Assignee: eBay Inc.
• Original Assignee: eBay Inc.
• Inventors: So, Wai Kei
• Primary Examiner(s): Beutel, William A
• Application Number:
  US15/219,790
• Publication Number:
  US 20160335498A1
• Time in Patent Office: 945 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal

Abstract

In various example embodiments, a system and method for providing information in an augmented reality display are provided. In example embodiments, an initial continuous stream of image data captured by a client device is received. An object within the continuous stream of image data is initially identified. The continuous stream of image data is further received and comprises additional image data regarding the object. The additional image data provides more visual details and features of the object than the initially received image data. Using the additional visual data, the identification of the object is refined to indicate a specific type of object. Based on the refined identification of the object, a search for information pertaining to the specific type of object is performed. A result is formatted to be displayed with a real-time image of the object on the client device.

Claims

1. A method comprising:

capturing, by a client device, an object from a first perspective in a continuous stream of image data;

transmitting, by the client device, the continuous stream of image data to an augmentation system;

receiving, by the client device, an initial identification of the object including information describing the object based on the first perspective;

displaying, by the client device, the object with the information describing the object from the initial identification;

capturing, by the client device, the object from a second perspective in the continuous stream of image data, the second perspective including a visual feature of the object not visible from the first perspective, and the second perspective differing from the first perspective based on movement of the client device relative to the object;

transmitting, by the client device, the continuous stream of image data including the object captured from the second perspective to the augmentation system;

receiving, by the client device, a refined identification of the object including additional information describing the object based on the second perspective; and

updating the display of the object with the information describing the object from the initial identification with the additional information describing the object based on the refined identification.

2. The method of claim 1, the initial identification of the object including information describing a type of the object and the refined identification of the object including additional information describing a manufacturer of the object.

3. The method of claim 1, the initial identification of the object including information describing a manufacturer of the object and the refined identification of the object including additional information describing a model number for the object.

4. The method of claim 1, wherein capturing the object from the second perspective in the continuous stream of image data is performed independent of a prompt for a user to capture the object from a different perspective.

5. The method of claim 1, further comprising transmitting location data for the client device to the augmentation system with the continuous stream of image data, the location data for the client device describing an orientation of the client device relative to the object.

6. The method of claim 1, further comprising:

transmitting time data to the augmentation system with the continuous stream of data, the time data describing a time at which the continuous stream of data is captured; and

receiving, by the client device, information describing one or more options for purchasing the object at the time at which the continuous stream of data is captured.

7. The method of claim 1, further comprising:

transmitting location information to the augmentation system with the continuous stream of data, the location information describing a location of the client device; and

receiving, by the client device, information describing one or more locations for purchasing the object within a threshold distance to the location of the client device.

8. A system comprising:

one or more processors; and

a computer-readable storage medium having instructions stored thereon that are executable by the one or more processors to perform operations comprising;

capturing, by a client device, an object from a first perspective in a continuous stream of image data;

transmitting, by the client device, the continuous stream of image data to an augmentation system;

receiving, by the client device, an initial identification of the object including information describing the object based on the first perspective;

displaying, by the client device, the object with the information describing the object from the initial identification;

capturing, by the client device, the object from a second perspective in the continuous stream of image data, the second perspective including a visual feature of the object not visible from the first perspective, and the second perspective differing from the first perspective based on movement of the client device relative to the object;

transmitting, by the client device, the continuous stream of image data including the object captured from the second perspective to the augmentation system;

receiving, by the client device, a refined identification of the object including additional information describing the object based on the second perspective; and

updating the display of the object with the information describing the object from the initial identification with the additional information describing the object based on the refined identification.

9. The system of claim 8, wherein the initial identification of the object includes information describing a type of the object and the refined identification of the object includes additional information describing a manufacturer of the object.

10. The system of claim 8, wherein the initial identification of the object includes information describing a manufacturer of the object and the refined identification of the object includes additional information describing a model number for the object.

11. The system of claim 8, wherein capturing the object from the second perspective in the continuous stream of image data is performed independent of a prompt for a user to capture the object from a different perspective.

12. The system of claim 8, the operations further comprising transmitting location data for the client device to the augmentation system with the continuous stream of image data, the location data for the client device describing an orientation of the client device relative to the object.

13. The system of claim 8, the operations further comprising:

transmitting time data to the augmentation system with the continuous stream of data, the time data describing a time at which the continuous stream of data is captured; and

receiving, by the client device, information describing one or more options for purchasing the object at the time at which the continuous stream of data is captured.

14. The system of claim 8, the operations further comprising:

transmitting location information to the augmentation system with the continuous stream of data, the location information describing a location of the client device; and

receiving, by the client device, information describing one or more locations for purchasing the object within a threshold distance to the location of the client device.

15. A computer-readable storage medium having instructions stored thereon that are executable by one or more hardware processors to perform operations comprising:

capturing, by a client device, an object from a first perspective in a continuous stream of image data;

transmitting, by the client device, the continuous stream of image data to an augmentation system;

receiving, by the client device, an initial identification of the object including information describing the object based on the first perspective;

displaying, by the client device, the object with the information describing the object from the initial identification;

capturing, by the client device, the object from a second perspective in the continuous stream of image data, the second perspective including a visual feature of the object not visible from the first perspective, and the second perspective differing from the first perspective based on movement of the client device relative to the object;

transmitting, by the client device, the continuous stream of image data including the object captured from the second perspective to the augmentation system;

receiving, by the client device, a refined identification of the object including additional information describing the object based on the second perspective; and

updating the display of the object with the information describing the object from the initial identification with the additional information describing the object based on the refined identification.

16. The computer-readable storage medium of claim 15, the operations further comprising:

transmitting time data to the augmentation system with the continuous stream of data, the time data describing a time at which the continuous stream of data is captured; and

receiving, by the client device, information describing one or more options for purchasing the object at the time at which the continuous stream of data is captured.

17. The computer-readable storage medium of claim 15, the operations further comprising:

transmitting location information to the augmentation system with the continuous stream of data, the location information describing a location of the client device; and

receiving, by the client device, information describing one or more locations for purchasing the object within a threshold distance to the location of the client device.

18. The computer-readable storage medium of claim 15, wherein the initial identification of the object includes information describing a type of the object and the refined identification of the object includes additional information describing a manufacturer of the object.

19. The computer-readable storage medium of claim 15, wherein the initial identification of the object includes information describing a manufacturer of the object and the refined identification of the object includes additional information describing a model number for the object.

20. The computer-readable storage medium of claim 15, wherein capturing the object from the second perspective in the continuous stream of image data is performed independent of receiving a prompt for a user to capture the object from a different perspective.