Object datastore in an augmented reality environment

• US 8,953,889 B1
• Filed: 09/14/2011
• Issued: 02/10/2015
• Est. Priority Date: 09/14/2011
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Rawles, LLC (Amazon.com, Inc.)
• Inventors: Worley, William Spencer III, Crump, Edward Dietz
• Primary Examiner(s): Shah, Paras D
• Assistant Examiner(s): Thomas-Homescu, Anne
• Application Number:
  US13/232,814
• Time in Patent Office: 1,245 Days
• Field of Search: 704/257, 704/235, 704/9, 704/255, 704/226, 704/8, 704/275, 704/270.1, 704/274, 704/270, 704/233, 704/251, 704/242, 704/231, 704/244, 382/103, 382/141, 382/118, 382/154, 382/159, 382/153, 382/124, 382/128, 382/190, 382/131, 382/291, 382/305, 34/140.8, 34/46, 34/36, 348/36, 348/46, 455/405, 455/404.1, 455/457, 379/52, 379/93.03, 379/283, 379/361
• US Class Current: 382/190
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/289 Object oriented databases
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

An augmented reality environment allows interaction between virtual and real objects and enhances an unstructured real-world environment. An object datastore comprising attributes of an object within the environment may be built and/or maintained from sources including manufacturers, retailers, shippers, and users. This object datastore may be local, cloud based, or a combination thereof. Applications may interrogate the object datastore to provide user functionality.

Claims

1. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

detecting an object in an environment from an image of the environment;

determining one or more attributes about the object via one or more sensors within the environment;

generating a preliminary identification of the object based at least in part on a comparison of the one or more attributes to attributes maintained in an object datastore;

calculating a confidence value for the preliminary identification;

comparing the confidence value for the preliminary identification to a confidence threshold;

in response to the confidence value for the preliminary identification reaching or exceeding the confidence threshold, identifying the object using the preliminary identification; and

in response to the confidence value for the preliminary identification being below the confidence threshold;

providing a visual indicator to identify the object, wherein the visual indicator is displayed proximate to the object or overlapping with the object;

querying a user for identification information associated with the object identified by the visual indicator; and

storing the identification information associated with the object in the object datastore.

2. The one or more non-transitory computer-readable storage media of claim 1, wherein the object datastore comprises physical attributes, functional attributes, and location attributes associated with a variety of different objects in the environment, in other environments, or both.

3. The one or more non-transitory computer-readable storage media of claim 1, the acts further comprising:

recognizing speech of the user at least partly in response to the querying; and

wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the speech.

4. The one or more non-transitory computer-readable storage media of claim 1, the acts further comprising:

recognizing a gesture made by the user at least partly in response to the querying; and

wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the gesture.

5. The one or more non-transitory computer-readable storage media of claim 1, wherein the object datastore is stored at least partially at a remote server accessible via a network interface.

6. A computer-implemented method comprising:

detecting an object in an environment from an image of the environment;

determining, via at least one computing device, one or more attributes about the object via one or more sensors within the environment;

generating, via the one or more computing devices, a preliminary identification of the object based at least in part on a comparison of the one or more attributes to attributes maintained in an object datastore;

calculating, via the at least one computing device, a confidence value for the preliminary identification;

comparing, via the at least one computing device, the confidence value for the preliminary identification to a confidence threshold;

in response to the confidence value for the preliminary identification reaching or exceeding the confidence threshold, identifying, via the at least one computing device, the object using the preliminary identification; and

in response to the confidence value for the preliminary identification being below the confidence threshold and via the at least one computing device;

providing a visual indicator to identify the object, wherein the visual indicator is displayed proximate to the object or overlapping with the object;

querying a user for identification information associated with the object identified by the visual indicator; and

storing the identification information associated with the object in the object datastore.

7. The computer-implemented method of claim 6, wherein the object datastore comprises physical attributes, functional attributes, and location attributes associated with a variety of different objects in the environment, in other environments, or both.

8. The computer-implemented method of claim 6, further comprising:

recognizing speech of the user at least partly in response to the querying; and

wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the speech.

9. The computer-implemented method of claim 6, further comprising:

recognizing a gesture made by the user at least partly in response to the querying; and

wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the gesture.

10. The computer-implemented method of claim 6, wherein the object datastore is stored at least partially at a remote server accessible via a network interface.

11. A system comprising:

one or more processors; and

memory storing computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising;

detecting an object in an environment from an image of the environment;

determining one or more attributes about the object via one or more sensors within the environment;

generating a preliminary identification of the object based at least in part on a comparison of the one or more attributes to attributes maintained in an object datastore;

calculating a confidence value for the preliminary identification;

comparing the confidence value for the preliminary identification to a confidence threshold;

in response to the confidence value for the preliminary identification reaching or exceeding the confidence threshold, identifying the object using the preliminary identification; and

in response to the confidence value for the preliminary identification being below the confidence threshold;

providing a visual indicator to identify the object, wherein the visual indicator is displayed proximate to the object or overlapping with the object;

querying a user for identification information associated with the object identified by the visual indicator; and

storing the identification information associated with the object in the object datastore.

12. The system of claim 11, wherein the object datastore comprises physical attributes, functional attributes, and location attributes associated with a variety of different objects in the environment, in other environments, or both.

13. The system of claim 11, the acts further comprising:

recognizing speech of the user at least partly in response to the querying; and

wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the speech.

14. The system of claim 11, the acts further comprising:

recognizing a gesture made by the user at least partly in response to the querying; and

wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the gesture.

15. The system of claim 11, wherein the object datastore is stored at least partially at a remote server accessible via a network interface.