Tactile interaction in virtual environments

• US 10,445,939 B2
• Filed: 01/10/2018
• Issued: 10/15/2019
• Est. Priority Date: 09/09/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shapira, Lior, Fernandez, Judith Amores, Palos, Xavier Benavides
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US15/867,494
• Publication Number:
  US 20180190034A1
• Time in Patent Office: 643 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Tactile virtual reality (VR) and/or mixed reality (MR) experiences are described. Techniques described herein include receiving data from a sensor and accessing a position and an orientation of a real object that is physically present in a real scene. Furthermore, techniques described herein include identifying the real object based at least in part on the position and the orientation of the real object and causing a graphical element corresponding to the real object to be rendered on a display of a VR and/or MR display device. The graphical element can be determined based at least in part on a VR and/or MR application. The techniques described herein include determining an interaction with the real object and causing a functionality associated with the graphical element to be performed in the VR or MR environment rendered via the VR and/or MR display device, respectively.

Claims

1. A method comprising:

accessing, by one or more processors in a head-mounted display (HMD), video data captured by a camera in the HMD;

identifying, by the one or more processors, based at least in part on the video data, a real object that is physically present in a real scene, the real object being held by a user wearing the HMD;

mapping, by the one or more processors, a virtual object to the real object, the virtual object having a same three-dimensional geometry as the real object, the virtual object being different from the real object;

generating, by the one or more processors, a virtual reality scene based on the video data, the virtual object replacing the real object in the virtual reality scene;

presenting the virtual reality scene in a display of the HMD;

detecting, by the one or more processors, a motion of the real object being held by the user;

determining, by the one or more processors, a functionality of the virtual object based on the motion;

changing, by the one or more processors, a view of the virtual object based on the determined functionality; and

presenting, after changing the view of the virtual object, the virtual reality scene in the display of the HMD.

2. The method as recited in claim 1, wherein changing the view of the virtual object further includes:

mapping a graphical element to the determined functionality, wherein the virtual object is presented in the virtual reality scene based on the graphical element.

3. The method as recited in claim 1, wherein presenting the virtual reality scene further includes rendering a graphical element on top of the virtual object.

4. The method as recited in claim 1, wherein detecting the motion further includes:

detecting a shaking of the real object.

5. The method as recited in claim 4, further comprising:

causing presentation of virtual rain or virtual snow to fall around the virtual object in the virtual reality scene.

6. The method as recited in claim 1, wherein detecting the motion further includes:

detecting a throw of the real object.

7. The method as recited in claim 6, further comprising:

creating a virtual explosion around the virtual object in the virtual reality scene.

8. The method as recited in claim 1, wherein detecting the motion further includes:

detecting a tap interaction.

9. The method as recited in claim 8, further comprising:

changing a representation of the virtual object in the virtual reality scene.

10. The method as recited in claim 1, wherein detecting the motion further includes:

detecting that the real object is placed on a floor.

11. The method as recited in claim 10, further comprising:

causing additional virtual content to be placed around the virtual object in the virtual reality scene.

12. The method as recited in claim 1, wherein the real object is a block and the virtual object has a same dimensions and shape as the block.

13. The method as recited in claim 1, wherein the functionality is associated with a binocular, wherein the binocular enables a user to see Previously Presented virtual content in the virtual reality scene.

14. The method as recited in claim 1, wherein the functionality is associated with sound, wherein a sound is generated when the real object is moved within a predetermined threshold distance from a ear of the user wearing the HMD.

15. The method as recited in claim 1, wherein the functionality is associated with a portal.

16. The method as recited in claim 1, wherein identifying the real object further comprises:

determining a geometry of the real object based on combinations of volumetric data, skeletal data, and perspective data; and

mapping the virtual object to the real object based on the geometry of the real object.

17. A head-mounted display (HMD) comprising:

a display;

a camera for capturing video data in front of the HMD;

a memory comprising instructions; and

one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising;

accessing video data captured by the camera;

identifying, based at least in part on the video data, a real object that is physically present in a real scene, the real object being held by a user wearing the HMD;

mapping a virtual object to the real object, the virtual object having a same three-dimensional geometry as the real object, the virtual object being different from the real object;

generating a virtual reality scene based on the video data, the virtual object replacing the real object in the virtual reality scene;

presenting the virtual reality scene in the display of the HMD;

detecting a motion of the real object being held by the user;

determining a functionality of the virtual object based on the motion;

changing a view of the virtual object based on the determined functionality; and

presenting, after changing the view of the virtual object, the virtual reality scene in the display of the HMD.

18. The HMD as recited in claim 17, wherein changing the view of the virtual object further includes:

mapping a graphical element to the determined functionality, wherein the virtual object is presented in the virtual reality scene based on the graphical element.

19. The HMD as recited in claim 17, wherein detecting the motion further includes:

detecting one or more of a shaking of the real object, a throw of the real object, a tap interaction, or placing the real object on a floor.

20. A non-transitory machine-readable storage medium including instructions that, when executed by a processor, cause the processor to perform operations comprising:

accessing video data captured by a camera in in a head-mounted display (HMD);

identifying, based at least in part on the video data, a real object that is physically present in a real scene, the real object being held by a user wearing the HMD;

mapping a virtual object to the real object, the virtual object having a same three-dimensional geometry as the real object, the virtual object being different from the real object;

generating a virtual reality scene based on the video data, the virtual object replacing the real object in the virtual reality scene;

presenting the virtual reality scene in a display of the HMD;

detecting a motion of the real object being held by the user;

determining a functionality of the virtual object based on the motion;

changing a view of the virtual object based on the determined functionality; and

presenting, after changing the view of the virtual object, the virtual reality scene in the display of the HMD.