MIXED REALITY SYSTEM LEARNED INPUT AND FUNCTIONS

• US 20130342571A1
• Filed: 06/25/2012
• Published: 12/26/2013
• Est. Priority Date: 06/25/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kinnebrew, Peter Tobias, Kamuda, Nicholas Ferianc
• Granted Patent: US 9,696,547 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels

Abstract

A see-through, near-eye, mixed reality display apparatus providing a mixed reality environment wherein one or more virtual objects and one or more real objects exist within the view of the device. Each of the real and virtual have a commonly defined set of attributes understood by the mixed reality system allowing the system to manage relationships and interaction between virtual objects and other virtual objects, and virtual and real objects.

Claims

1. A method of interpreting commands to a mixed reality environment, comprising;

rendering one or more virtual objects within a field of view;

receiving input actions with the virtual objects in the field of view;

monitoring input actions for

known input actions enabling known functions of virtual objects;

unknown input actions for which known functions of virtual objects be enabled,

known input actions for which unknown functions of a virtual object may be enabled;

unknown input actions for which unknown function of a virtual object may be enabled; and

for each unknown input action detected, determining input data resulting in an input action to link to a function; and

for each unknown function, creating a function for the virtual object.

2. The method of claim 1 wherein the method further includes linking known functions to new functions.

3. The method of claim 1 wherein the method further includes linking new input actions to known functions.

4. The method of claim 1 wherein the method further includes linking new input actions to new functions.

5. The method of claim 1 wherein receiving input actions comprises receiving input data from the field of view comprising a combination of data from sensors detecting a natural human interaction with a virtual object.

6. The method of claim 5 wherein the input data comprises input from the field of view as one or more of:

audio data;

depth data;

image data;

motion data; and

focus data.

7. The method of claim 1 wherein the steps of determining input data and creating a function are enabled during a learning mode.

8. The method of claim 1 wherein the steps of determining input data and creating a function are enabled after a plurality of repetitions of input actions related to a same result are monitored in the input data.

9. The method of claim 8 wherein the repetitions of input actions relative to a result are not identical.

10. A see-through head mounted display apparatus, comprising:

a see-through, near-eye, augmented reality display;

one or more processing devices in wireless communication with apparatus, the one or more processing devices determine an environment, one or more real objects in the environment and one or more virtual objects in the environment, the one or more processing devices receive input actions with the one or more virtual objects in a field of view of the display, and monitor received input actions for known input actions enabling known functions of virtual objects, unknown input actions for which known functions of virtual objects may be enabled, known input actions for which unknown functions of a virtual object may be enabled; and

unknown input actions for which unknown function of a virtual object may be enabled; and

the one or more processing devices determining input data resulting in a new input action for a function, and creating a new function for one or more virtual objects.

11. The apparatus of claim 10 further including a plurality of input data sensors providing a combination of input comprising:

audio data;

depth data;

image data;

motion data; and

focus data.

12. The apparatus of claim 10 wherein the one or more processing devices render virtual objects in the environment in the display, at least one of said virtual objects linked to a new input action, including rendering a function in response to input data for the new input action.

13. The apparatus of claim 10 wherein the one or more processing devices render virtual objects in the environment in the display, at least one of said virtual objects linked to a new function, including rendering a the new function in response to input data for the new input action.

14. The apparatus of claim 10 wherein the one or more processing devices determine input data resulting in the new input action and creating a new function after a plurality of repetitions of input actions related to a same result are received in the input data.

15. A method generating new input actions and new functions for virtual objects in a see through head mounted display system, comprising:

rendering virtual objects in an environment, each object having at least a viewable physical representation and behavior, the virtual object responsive to input actions;

receiving input data from a plurality of sensors including data representing input actions with virtual and real objects in the environment;

determining whether the input data represents;

unknown input actions for which known functions of virtual objects may be enabled,

known input actions for which unknown functions of a virtual object may be enabled; and

unknown input actions for which unknown function of a virtual object may be enabled;

for each unknown input action detected, determining input data resulting in a new input action to link to a function; and

for each unknown function, creating a function to link to a virtual object; and

linking new input actions and new functions to virtual objects in the environment.

16. The method of claim 15 wherein the step of rendering includes rendering an object to perform a function when a new input action is received.

17. The method of claim 16 wherein the step of rendering includes rendering an object to perform a function when input data enabling a new function is received.

18. The method of claim 15 further including the step of rendering shared virtual objects created by other see though head mounted display systems.

19. The method of claim 15 wherein the new function are enabled after a plurality of repetitions of input actions related to a same result are monitored in the input data.

20. The method of claim 15 wherein the new input action comprises input data from one or more combinations of:

audio data;

depth data;

image data;

motion data; and

focus data.