Integrated augmented reality environment

• US 9,721,386 B1
• Filed: 12/27/2010
• Issued: 08/01/2017
• Est. Priority Date: 12/27/2010
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Worley, III, William Spencer, Betts-LaCroix, Jonathan, Crump, Edward Dietz
• Primary Examiner(s): Chauhan, Ulka
• Assistant Examiner(s): Yoon, Sae Won
• Application Number:
  US12/978,800
• Time in Patent Office: 2,409 Days
• Field of Search: 345505, 345633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Augmented reality environments allow users in their physical environment to interact with virtual objects and information. Augmented reality applications are developed and configured to utilize local as well as cloud resources. Application management allows control over distribution of applications to select groups or all users. An application programming interface allows simplified control and distribution of tasks between local and cloud resources during development and post-development operation. This integration between local and cloud resources along with the control afforded by application management allows rapid development, testing, deployment, and updating of augmented reality applications.

Claims

1. A device coupled to an augmented reality environment, the device comprising:

a processor;

a memory coupled to the processor;

a network adapter;

an internal task resource capable of processing tasks without receiving data via the network adapter; and

an augmented reality application stored in the memory and comprising;

an application module configured to identify one or more tasks based at least in part on input detected by an augmented reality node based on verbal or physical interaction within an augmented reality environment;

a task analysis module configured to determine one or more security parameters that govern which resources are eligible for processing of the one or more tasks;

a task distribution module configured to;

identify, from a collection of resources, task resources capable of processing at least some of the one or more tasks, the task resources including (i) the internal task resource and (ii) external task resources in communication with the device via the network adapter;

access resource availability data for at least some of the task resources;

determine available task resources from the task resources based at least in part on comparing the one or more security parameters of the one or more tasks with the resource availability data associated with the task resources;

assign a task of the one or more tasks to a first task resource of the available task resources, the first task resource being the internal task resource and the first task resource processing the task using a first algorithm and first reference data to generate a first textual result; and

assign the task to a second task resource of the available task resources, the second task resource being accessible via the network adapter of the device, the second task resource processing the task using a second algorithm and second reference data while using same input data as the first task resource to generate a second textual result; and

a task disbursement interface module configured to disburse the task to the first task resource and the second task resource and receive the first textual result and the second textual result from the first task resource and the second task resource, at least one of the first textual result or the second textual result being used to modify the augmented reality environment, wherein the task of the one or more tasks is processed in parallel by the first task resource and the second task resource.

2. The device of claim 1, further comprising a structured light projector and a camera coupled to the processor and configured to recover the structured light and wherein the augmented reality application is configured to accept data from the recovered structured light to identify at least gestures provided by a user from the physical interaction within the augmented reality environment, the gestures used by the application module to identify the one or more tasks.

3. The device of claim 1, wherein the one or more security parameters restrict at least one task from being sent by the device via the network adapter.

4. The device of claim 1, wherein the task disbursement interface module is further configured to:

implement the first textual result to modify the augmented reality environment;

determine that the second textual result is different than the first textual result, the second textual result received after the first textual result; and

implement, without user input, the second textual result to replace a portion of the first textual result.

5. The device of claim 1, wherein the task distribution module assigns the task of the one or more tasks to the two or more resources to minimize overall response time.

6. The device of claim 1, wherein the task distribution module assigns the task of the one or more tasks to the two or more resources according to at least one of a pre-determined preferential resource hierarchy or a pre-determined mapping.

7. The device of claim 1, wherein the first task resource includes a first time budget for processing the task and the second task resource includes a second time budget for processing the task, wherein a difference between the first time budget and the second time budget is based at least in part on latency caused by use of the network adapter.

8. The device of claim 1, wherein the task disbursement interface module is configured to disburse another task of the one or more tasks to each of (i) the local devices accessible via the internal communication bus, the local area network devices accessible via the local area network, and the cloud resources accessible via the wide area network.

9. The device of claim 1, further comprising an augmented reality development application stored in the memory and comprising a development application module configured to build or modify the augmented reality application.

10. The device of claim 9, further comprising a resource simulator configured to simulate resources available on local devices, local area network devices, or both.

11. The device of claim 1, wherein the augmented reality module further comprises a pre-processing module configured to pre-process data associated with the task prior to disbursement of the data to the assigned resource.

12. The device of claim 1, wherein the second textual result is received after the first textual result, and wherein the task interface module is further configured to:

implement the first textual result to modify the augmented reality environment; and

implement at least a portion of the second textual result to further modify the augmented reality environment.

13. The device of claim 1, wherein the application module is further operable to sense one or more physical objects in the augmented reality environment based at least in part on an appearance of the one or more physical objects, and wherein the input detected by the augmented reality node is based at least in part on the physical interaction with the one or more physical objects within the augmented reality environment.

14. The device of claim 1, wherein the task includes processing speech, and wherein the first textual result includes first text generated by the first resource resulting from processing the task, and wherein the second textual result includes second, different text generated by the second resource resulting from processing the task, the second textual result being an authoritative result that is prioritized over the first textual result.

15. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:

receiving, at an augmented reality device, a task request to complete a task associated with modifying an augmented reality environment;

determining one or more task parameters of the task;

accessing resource availability data for task resources;

comparing the one or more task parameters with the resource availability data of the task resources to determine available task resources located throughout or coupled to the augmented reality environment;

assigning the task to an internal task resource of the available task resources located internal to the augmented reality device, the internal task resource using a first algorithm to generate a first textual result using input data;

assigning the task to an external task resource of the available task resources in communication with the augmented reality environment via a network adapter, the external task resource using a second algorithm to generate a second textual result using the input data, the task to be processed separately by the first task resource and by the second task resource;

distributing the task to the first task resource;

distributing the task to the second task resource;

receiving the first textual result from the internal task resource; and

receiving the second textual result from the external task resource, wherein the first textual result is different than the second textual result.

16. The one or more non-transitory computer-readable storage media of claim 15, wherein the computer-executable instructions cause one or more processors to perform acts comprising:

presenting a first output based at least in part on the first textual result; and

presenting a second output to replace the first output, the second output based at least in part on the second textual result.

17. The one or more non-transitory computer-readable storage media of claim 15, wherein the augmented reality environment comprises an interactive scene of physical objects and virtual objects.

18. The one or more non-transitory computer-readable storage media of claim 15, wherein at least some of the task resources comprise shared resources accessible to a plurality of augmented reality environments and accessible via the network adapter.

19. The one or more non-transitory computer-readable storage media of claim 15, wherein the assigning comprises a comparison of the one or more tasks with a pre-determined preferential resource hierarchy.

20. The one or more non-transitory computer-readable storage media of claim 15, wherein the assigning comprises a pre-determined mapping between specified tasks and specified resources.

21. A method comprising:

receiving a task request to complete a task associated with modifying an augmented reality environment;

determining one or more task parameters of the task;

accessing resource availability data for task resources;

comparing, by one or more computing devices, the one or more task parameters with the resource availability data of the task resources to determine available task resources;

distributing the task to an internal task resource of the available task resources, the internal task resource processing the task using a first algorithm and first reference data to generate a first textual result, the task distributed to the internal task resource without use of a network adapter;

distributing the task to an external task resource of the available task resources, the external task resource processing the task using a second algorithm and second reference data to generate a second textual result, the task distributed to the external task resource using the network adapter, wherein the internal task resource and the external task resource use same input data for the task and process the task independently; and

receiving the first textual result from the internal task resource and the second textual result from the external task resource, the second textual result being different than the first textual result.

22. The method as recited in claim 21, further comprising:

implementing the first textual result from the internal task resource; and

implementing the second textual result from the external task resource to replace the first textual result, wherein the first textual result is received before the second textual result.

23. The method as recited in claim 21, wherein the task includes processing speech to determine textual content, and further comprising:

receiving first text as the first textual result from the internal task resource; and

receiving second text as the second textual result from the external task resource, wherein the second text is different than the first text.

24. The method as recited in claim 21, wherein the one or more task parameters includes a security parameter, and further comprising distributing another task to the internal task resource based at least in part on the security parameter.