MIXED REALITY INTERACTIONS

• US 20140320389A1
• Filed: 04/29/2013
• Published: 10/30/2014
• Est. Priority Date: 04/29/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Scavezze, Michael, Steed, Jonathan, Latta, Stephen, Geisner, Kevin, McCulloch, Daniel, Mount, Brian, Hastings, Ryan, Heckinger, Phillip Charles
• Granted Patent: US 9,443,354 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts

Abstract

Embodiments that relate to interacting with a physical object in a mixed reality environment via a head-mounted display are disclosed. In one embodiment a mixed reality interaction program identifies an object based on an image from captured by the display. An interaction context for the object is determined based on an aspect of the mixed reality environment. A profile for the physical object is queried to determine interaction modes for the object. A selected interaction mode is programmatically selected based on the interaction context. A user input directed at the object is received via the display and interpreted to correspond to a virtual action based on the selected interaction mode. The virtual action is executed with respect to a virtual object associated with the physical object to modify an appearance of the virtual object. The modified virtual object is then displayed via the display.

Claims

1. A mixed reality interaction system for interacting with a physical object in a mixed reality environment, the mixed reality interaction system comprising:

a head-mounted display device operatively connected to a computing device, the head-mounted display device including a display system for presenting the mixed reality environment and a plurality of input sensors including a camera for capturing an image of the physical object; and

a mixed reality interaction program executed by a processor of the computing device, the mixed reality interaction program configured to;

identify the physical object based on the captured image;

determine an interaction context for the identified physical object based on one or more aspects of the mixed reality environment;

query a stored profile for the physical object to determine a plurality of interaction modes for the physical object;

programmatically select a selected interaction mode from the plurality of interaction modes based on the interaction context;

receive a user input directed at the physical object via one of the input sensors of the head-mounted display device;

interpret the user input to correspond to a virtual action based on the selected interaction mode;

execute the virtual action with respect to a virtual object associated with the physical object to thereby modify an appearance of the virtual object; and

display the virtual object via the head-mounted display device with the modified appearance.

2. The mixed reality interaction system of claim 1, wherein the mixed reality interaction program is further configured to:

present a first query to confirm an accuracy of an identity of the physical object; and

in response to the query, either (1) receive a confirmation of the accuracy of the identity of the physical object or (2) where the confirmation of the accuracy of the identity is not received, present a second query requesting the identity of the physical object.

3. The mixed reality interaction system of claim 1, wherein the mixed reality interaction program is further configured to geo-locate the virtual object in physical proximity to the physical object.

4. The mixed reality interaction system of claim 1, wherein the selected interaction mode is a first selected interaction mode and the virtual action is a first virtual action, and the mixed reality interaction program is further configured to:

determine a change in the interaction context;

based on the change in the interaction context, programmatically select a second selected interaction mode from the plurality of interaction modes;

interpret the user input to correspond to a second virtual action based on the second selected interaction mode;

execute the second virtual action with respect to the virtual object associated with the physical object to thereby modify an appearance of the virtual object; and

display the virtual object via the head-mounted display device with the modified appearance.

5. The mixed reality interaction system of claim 4, wherein the virtual object is a first virtual object, and the mixed reality interaction program is further configured to:

execute the second virtual action with respect to a second virtual object associated with the physical object to thereby modify an appearance of the second virtual object; and

display the second virtual object via the head-mounted display device with the modified appearance.

6. The mixed reality interaction system of claim 5, wherein the mixed reality interaction program is further configured to:

interpret the user input to correspond to a third virtual action applicable to the first virtual object and the second virtual object;

execute the third virtual action with respect to the first virtual object and the second virtual object to thereby modify the appearance of the first virtual object and the second virtual object; and

display the first virtual object and the second virtual object via the head-mounted display device with the modified appearances.

7. The mixed reality interaction system of claim 1, wherein the selected interaction mode is a first selected interaction mode, the user input is a first user input, and the virtual action is a first virtual action, the mixed reality interaction program is further configured to:

receive a second user input via one of the plurality of input sensors of the head-mounted display device;

interpret the second user input to correspond to a request to modify the first selected interaction mode;

based on the second user input, change from the first selected interaction mode to a second selected interaction mode from the plurality of interaction modes;

receive a third user input via one of the plurality of input sensors of the head-mounted display device;

interpret the third user input to correspond to a second virtual action based on the second selected interaction mode;

execute the second virtual action with respect to the virtual object associated with the physical object to thereby modify an appearance of the virtual object; and

display the virtual object via the head-mounted display device with the modified appearance.

8. The mixed reality interaction system of claim 1, wherein the user input is a first user input and the virtual action is a first virtual action, wherein the mixed reality interaction program is further configured to:

display a geo-located target virtual object in the mixed reality environment;

determine an interaction context for the geo-located target virtual object based on the one or more aspects of the mixed reality environment;

query a stored profile for the geo-located target virtual object to determine a plurality of interaction modes for the geo-located target virtual object;

programmatically select a selected interaction mode from the plurality of interaction modes for the geo-located target virtual object based on the interaction context;

receive a second user input directed at the geo-located target virtual object via one of the input sensors of the head-mounted display device;

interpret the second user input to correspond to a second virtual action based on the selected interaction mode for the geo-located target virtual object;

execute the second virtual action with respect to the geo-located target virtual object to thereby modify an appearance of the geo-located target virtual object; and

display the geo-located target virtual object with the modified appearance.

9. The mixed reality interaction system of claim 1, wherein the mixed reality interaction program is further configured to create a new profile for the physical object that associates one or more interaction modes with the physical object.

10. The mixed reality interaction system of claim 1, wherein the mixed reality interaction program is further configured to:

identify an updated physical state of the physical object; and

based on the updated physical state, further modify the appearance of the virtual object associated with the physical object.

11. A method for interacting with a physical object in a mixed reality environment, comprising:

providing a head-mounted display device operatively connected to a computing device, the head-mounted display device including a display system for presenting the mixed reality environment and a plurality of input sensors including a camera for capturing an image of the physical object;

identifying the physical object based on the captured image;

determining an interaction context for the identified physical object based on one or more aspects of the mixed reality environment;

querying a stored profile for the physical object to determine a plurality of interaction modes for the physical object;

programmatically selecting a selected interaction mode from the plurality of interaction modes based on the interaction context;

receiving a user input directed at the physical object via one of the input sensors of the head-mounted display device;

interpreting the user input to correspond to a virtual action based on the selected interaction mode;

executing the virtual action with respect to a virtual object associated with the physical object to thereby modify an appearance of the virtual object; and

displaying the virtual object via the head-mounted display device with the modified appearance.

12. The method of claim 11, further comprising:

presenting a query to confirm an accuracy of an identity of the physical object; and

in response to the query, receiving a confirmation of the accuracy of the identity of the physical object.

13. The method of claim 11, further comprising geo-locating the virtual object in physical proximity to the physical object.

14. The method of claim 11, wherein the selected interaction mode is a first selected interaction mode and the virtual action is a first virtual action, and the method further comprises:

determining a change in the interaction context;

based on the change in the interaction context, programmatically selecting a second selected interaction mode from the plurality of interaction modes;

interpreting the user input to correspond to a second virtual action based on the second selected interaction mode;

executing the second virtual action with respect to the virtual object associated with the physical object to thereby modify an appearance of the virtual object; and

displaying the virtual object via the head-mounted display device with the modified appearance.

15. The method of claim 11, wherein the virtual object is a first virtual object, and the method further comprises:

executing the second virtual action with respect to a second virtual object associated with the physical object to thereby modify an appearance of the second virtual object; and

displaying the second virtual object via the head-mounted display device with the modified appearance.

16. The method of claim 11, wherein the selected interaction mode is a first selected interaction mode, the user input is a first user input, and the virtual action is a first virtual action, and the method further comprises:

receiving a second user input via one of the plurality of input sensors of the head-mounted display device;

interpreting the second user input to correspond to a request to modify the first selected interaction mode;

based on the second user input, changing from the first selected interaction mode to a second selected interaction mode from the plurality of interaction modes;

receiving a third user input via one of the plurality of input sensors of the head-mounted display device;

interpreting the third user input to correspond to the second virtual action based on the second selected interaction mode;

executing the second virtual action with respect to the virtual object associated with the physical object to thereby modify an appearance of the virtual object; and

displaying the virtual object via the head-mounted display device with the modified appearance.

17. The method of claim 11, wherein the user input is a first user input and the virtual action is a first virtual action, and the method further comprises:

displaying a geo-located target virtual object in the mixed reality environment;

determining an interaction context for the geo-located target virtual object based on the one or more aspects of the mixed reality environment;

querying a stored profile for the geo-located target virtual object to determine a plurality of interaction modes for the geo-located target virtual object;

programmatically selecting a selected interaction mode from the plurality of interaction modes for the geo-located target virtual object based on the interaction context;

receiving a second user input directed at the geo-located target virtual object via one of the input sensors of the head-mounted display device;

interpreting the second user input to correspond to a second virtual action based on the selected interaction mode for the geo-located target virtual object;

executing the second virtual action with respect to the geo-located target virtual object to thereby modify an appearance of the geo-located target virtual object; and

displaying the geo-located target virtual object with the modified appearance.

18. The method of claim 11, wherein the user input is a first user input, and the method further comprises using a second user input to associate the physical object with the selected interaction mode.

19. A method for interacting with a physical object in a mixed reality environment, comprising:

providing a head-mounted display device operatively connected to a computing device, the head-mounted display device including a display system for presenting the mixed reality environment and a plurality of input sensors including a camera for capturing an image of the physical object;

identifying the physical object based on the captured image;

determining an interaction context for the identified physical object based on one or more aspects of the mixed reality environment;

querying a stored profile for the physical object to determine a plurality of interaction modes for the physical object;

programmatically selecting a first selected interaction mode from the plurality of interaction modes based on the interaction context;

receiving a first user input directed at the physical object via one of the input sensors of the head-mounted display device;

interpreting the first user input to correspond to a first virtual action based on the first selected interaction mode;

executing the first virtual action with respect to a virtual object associated with the physical object to thereby modify an appearance of the virtual object to a first modified appearance; and

displaying the virtual object via the head-mounted display device with the first modified appearance;

determining a change in the interaction context;

based on the change in the interaction context, programmatically selecting a second selected interaction mode from the plurality of interaction modes;

receiving a second user input directed at the physical object via one of the input sensors of the head-mounted display device;

interpreting the second user input to correspond to a second virtual action based on the second selected interaction mode;

executing the second virtual action with respect to the virtual object associated with the physical object to thereby modify an appearance of the virtual object to a second modified appearance; and

displaying the virtual object via the head-mounted display device with the second modified appearance.

20. The method of claim 19, wherein the first user input and the second user input are each selected from the group consisting of eye-tracking input, voice input, gesture input, and head movement input.