Tactile Interaction In Virtual Environments

• US 20170069134A1
• Filed: 11/04/2015
• Published: 03/09/2017
• Est. Priority Date: 09/09/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Palos, Xavier Benavides, Fernandez, Judith Amores, Shapira, Lior
• Granted Patent: US 9,898,869 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Tactile virtual reality (VR) and/or mixed reality (MR) experiences are described. Techniques described herein include receiving data from a sensor and accessing a position and an orientation of a real object that is physically present in a real scene. Furthermore, techniques described herein include identifying the real object based at least in part on the position and the orientation of the real object and causing a graphical element corresponding to the real object to be rendered on a display of a VR and/or MR display device. The graphical element can be determined based at least in part on a VR and/or MR application. The techniques described herein include determining an interaction with the real object and causing a functionality associated with the graphical element to be performed in the VR or MR environment rendered via the VR and/or MR display device, respectively.

Claims

1. A system comprising:

a sensor;

one or more processors;

memory; and

one or more modules stored in the memory and executable by the one or more processors to perform operations comprising;

receiving first data from the sensor;

identifying, based at least in part on the first data, a real object that is physically present in a real scene, wherein the real object is associated with a unique identifier;

accessing a virtual element that corresponds to the unique identifier, the virtual element representing at least one of a geometry or surface details of the real object;

accessing a graphical element that corresponds to the unique identifier, the graphical element being determined based at least in part on a virtual reality application;

causing a virtual object that corresponds to the virtual element and the graphical element to be rendered in a virtual reality scene;

receiving second data from the sensor;

determining, based at least in part on the second data, that the real object is interacted with via an interaction;

determining a functionality associated with the graphical element and the interaction; and

causing the functionality to be performed in the virtual reality scene.

2. A system as claim 1 recites, wherein the graphical element is different for different virtual reality applications.

3. A system as claim 1 recites, wherein the functionality is different for different virtual reality applications.

4. A system as claim 1 recites, wherein causing the functionality to be performed comprises replacing the graphical element with a new graphical element based at least in part on the interaction.

5. A system as claim 1 recites, wherein the interaction comprises shaking the real object.

6. A system as claim 1 recites, wherein the interaction comprises rotating the real object.

7. A system as claim 1 recites, wherein the interaction comprises tapping the real object.

8. A system as claim 1 recites, wherein the interaction comprises stacking the real object on top of or on bottom of one or more other real objects.

9. A system as claim 1 recites, wherein the interaction comprises placing the real object on a floor of the real scene.

10. A system as claim 1 recites, wherein the interaction comprises moving the real object within a threshold distance of a predetermined position on a body of a user.

11. A system as claim 1 recites, wherein:

the sensor is associated with the real object;

receiving the first data from the sensor comprises receiving position data and orientation data indicating a position and an orientation, respectively, of the real object from the sensor associated with the real object; and

identifying the real object based at least in part on the position data and the orientation data.

12. A system as claim 1 recites, wherein:

the sensor is integrated into at least one of a device associated with a user or an external device associated with the real scene;

the operations further comprise determining position data and orientation data indicating a position and an orientation, respectively, of the real object based at least in part on the first data received from the sensor; and

identifying the real object based at least in part on the position data and the orientation data.

13. A computer-implemented method, the method comprising:

receiving first data from a sensor;

accessing, based at least in part on the first data received from the sensor, a position and an orientation of a first real object that is physically present in a real scene;

identifying the first real object based at least in part on the position and the orientation of the first real object;

causing a first graphical element corresponding to the first real object to be rendered on a display of a virtual reality display device, the first graphical element being determined based at least in part on a first virtual reality application;

receiving second data from the sensor;

determining, based at least in part on the second data, a first interaction with the first real object; and

causing a first functionality associated with the first graphical element to be performed.

14. A computer-implemented method as claim 13 recites, wherein the first functionality comprises causing a second graphical element to be rendered on the display instead of the first graphical element, the second graphical element being determined based at least in part on the first virtual reality application and being different from the first graphical element.

15. A computer-implemented method as claim 13 recites, further comprising determining the first functionality based at least in part on the first interaction.

16. A computer-implemented method as claim 15 recites, further comprising:

receiving third data from the sensor;

determining, based at least in part on the third data, a second interaction with the first real object;

determining, based at least in part on the second interaction, a second functionality associated with the first graphical element; and

causing a performance of the second functionality associated with the first graphical element.

17. A computer-implemented method as claim 13 recites, further comprising:

determining that the first virtual reality application is terminated;

determining that a second virtual reality application is activated; and

causing a second graphical element corresponding to the first real object to be rendered on the display of the virtual reality display device, the second graphical element being determined based at least in part on a second virtual reality application and being different from the first graphical element.

18. A computer-implemented method as claim 13 recites, further comprising:

determining, based at least in part on the first data received from the sensor, a position and an orientation of a second real object that is physically present in the real scene;

identifying the second real object based at least in part on the position and the orientation of the second real object;

causing a second graphical element corresponding to the second real object to be rendered on the display of the virtual reality display device, the second graphical element being determined based at least in part on the first virtual reality application;

determining that the second real object is placed within a threshold distance of the first real object in a stacked configuration or an aligned configuration; and

replacing the first graphical element and the second graphical element with a third graphical element corresponding to the first real object and the second real object.

19. One or more computer storage media having computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform operations comprising:

receiving first data from a sensor;

identifying a real object that is physically located in a real environment based at least in part on the data;

rendering a virtual object that corresponds to the real object in a virtual reality environment, the virtual object being associated with;

a virtual element that represents the real object;

a graphical element corresponding to the virtual element, the graphical element changing based at least in part on different virtual reality applications; and

one or more functionalities, the one or more functionalities changing based at least in part on at least one of the different virtual reality applications or different interactions with the real object;

receiving second data from the sensor;

determining, based at least in part on the second data, an interaction with the real object; and

causing, based at least in part on the interaction, a performance of the one or more functionalities in at least one of a mixed reality environment or a virtual reality environment.

20. One or more computer storage media as claim 19 recites, wherein, prior to rendering the virtual object that corresponds to the real object in the virtual reality environment, the operations further comprise:

determining, based at least in part on the first data, a geometry of the real object;

assigning a unique identifier to the real object;

generating the virtual element that maps to the geometry of the real object;

assigning the unique identifier to the virtual element;

performing semantic recognition on the real object to determine an identity of the real object;

based at least in part on results of the semantic recognition, associating one or more graphical elements with the unique identifier;

based at least in part on identifying the real object, determining the unique identifier corresponding to the real object; and

accessing the virtual element and the one or more graphical elements.