Tactile interaction in virtual environments

• US 9,898,869 B2
• Filed: 11/04/2015
• Issued: 02/20/2018
• Est. Priority Date: 09/09/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shapira, Lior, Palos, Xavier Benavides, Fernandez, Judith Amores
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US14/932,282
• Publication Number:
  US 20170069134A1
• Time in Patent Office: 839 Days
• Field of Search: 345633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Tactile virtual reality (VR) and/or mixed reality (MR) experiences are described. Techniques described herein include receiving data from a sensor and accessing a position and an orientation of a real object that is physically present in a real scene. Furthermore, techniques described herein include identifying the real object based at least in part on the position and the orientation of the real object and causing a graphical element corresponding to the real object to be rendered on a display of a VR and/or MR display device. The graphical element can be determined based at least in part on a VR and/or MR application. The techniques described herein include determining an interaction with the real object and causing a functionality associated with the graphical element to be performed in the VR or MR environment rendered via the VR and/or MR display device, respectively.

Claims

1. A system comprising:

a sensor;

one or more processors;

memory; and

one or more modules stored in the memory and executable by the one or more processors to perform operations comprising;

receiving first data from the sensor;

identifying, based at least in part on the first data, a first real object and a second real object that are physically present in a real scene;

determining a position of the first real object and a position of the second real object in the real scene;

accessing a first virtual element and a second virtual element representing at least one of a geometry or surface details of the first real object and the second real object;

causing a first virtual object and a second virtual object, corresponding to the first virtual element and the second virtual element respectively, to be rendered in a virtual reality scene based on a virtual reality application;

receiving second data from the sensor;

detecting, based on the second data, an interaction where the second real object is placed within a threshold distance of the first real object in the real scene; and

causing, in response to the interaction, presentation of a third virtual object to be rendered on the virtual reality scene to replace the first virtual object and the second virtual object.

2. A system as claim 1 recites, wherein the first virtual element is different for different virtual reality applications.

3. A system as claim 1 recites, further comprising:

receiving third data from the sensor;

detecting, based on the third data, a second interaction with that the first real object;

determining a functionality associated with the first virtual object and the interaction; and

causing the functionality to be performed in the virtual reality scene, wherein the functionality is different for different virtual reality applications.

4. A system as claim 3 recites, wherein causing the functionality to be performed comprises replacing the first virtual object with a new virtual object based at least in part on the interaction.

5. A system as claim 3 recites, wherein the interaction comprises shaking the first real object.

6. A system as claim 3 recites, wherein the interaction comprises rotating the first real object.

7. A system as claim 3 recites, wherein the interaction comprises tapping the first real object.

8. A system as claim 3 recites, wherein the interaction comprises stacking the first real object on top of or on bottom of one or more other real objects.

9. A system as claim 3 recites, wherein the interaction comprises placing the first real object on a floor of the real scene.

10. A system as claim 3 recites, wherein the interaction comprises moving the first real object within a threshold distance of a predetermined position on a body of a user.

11. A system as claim 1 recites, wherein receiving the first data from the sensor further comprises:

receiving position data and orientation data for the first real object and the second real object.

12. A system as claim 1 recites, wherein the sensor is integrated into at least one of a device associated with a user or an external device associated with the real scene.

13. A system as claim 1 recites, wherein detecting the interaction further comprises:

detecting that the first real object and the second real object are stacked or aligned.

14. A computer-implemented method, the method comprising:

receiving first data from a sensor;

accessing, based at least in part on the first data received from the sensor, a position and an orientation of a first real object that is physically present in a real scene;

identifying the first real object based at least in part on the position and the orientation of the first real object;

causing a first graphical element corresponding to the first real object to be rendered on a display of a virtual reality display device, the first graphical element being determined based at least in part on a first virtual reality application;

receiving second data from the sensor;

determining, based at least in part on the second data, a first interaction with the first real object;

causing a first functionality associated with the first graphical element to be performed;

determining that the first virtual reality application is terminated;

determining that a second virtual reality application is activated; and

causing a second graphical element corresponding to the first real object to be rendered on the display of the virtual reality display device, the second graphical element being determined based at least in part on a second virtual reality application and being different from the first graphical element.

15. A computer-implemented method as claim 14 recites, wherein the first functionality comprises causing a second graphical element to be rendered on the display instead of the first graphical element, the second graphical element being determined based at least in part on the first virtual reality application and being different from the first graphical element.

16. A computer-implemented method as claim 14 recites, further comprising determining the first functionality based at least in part on the first interaction.

17. A computer-implemented method as claim 16 recites, further comprising:

receiving third data from the sensor;

determining, based at least in part on the third data, a second interaction with the first real object;

determining, based at least in part on the second interaction, a second functionality associated with the first graphical element; and

causing a performance of the second functionality associated with the first graphical element.

18. One or more computer storage media having computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform operations comprising:

receiving first data from a sensor;

identifying a real object that is physically located in a real environment based at least in part on the first data;

determining, based at least in part on the first data, a geometry of the real object;

assigning a unique identifier to the real object;

generating a virtual element that maps to the geometry of the real object;

assigning the unique identifier to the virtual element;

performing semantic recognition on the real object to determine an identity of the real object;

based at least in part on results of the semantic recognition, associating one or more graphical elements with the unique identifier;

based at least in part on identifying the real object, determining the unique identifier corresponding to the real object; and

accessing the virtual element and the one or more graphical elements;

rendering a virtual object that corresponds to the real object in a virtual reality environment, the virtual object being associated with;

the virtual element that represents the real object;

a graphical element corresponding to the virtual element, the graphical element changing based at least in part on different virtual reality applications; and

one or more functionalities, the one or more functionalities changing based at least in part on at least one of the different virtual reality applications or different interactions with the real object;

receiving second data from the sensor;

determining, based at least in part on the second data, an interaction with the real object; and

causing, based at least in part on the interaction, a performance of the one or more functionalities in at least one of a mixed reality environment or a virtual reality environment.