Mixed Reality Social Interactions

• US 20170039986A1
• Filed: 08/07/2015
• Published: 02/09/2017
• Est. Priority Date: 08/07/2015
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Porras Luraschi, Javier A., Lanier, Jaron, Won, Andrea, Chang, Wayne
• Application Number:
  US14/821,505
• Publication Number:
  US 20170039986A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092

Abstract

Social interactions between two or more users in a mixed reality environment are described. Techniques describe receiving data from a sensor. Based at least in part on receiving the data, the techniques describe determining that an object associated with a first user that is physically present in a real scene interacts with a second user that is present in the real scene. Based at least in part on determining that the object interacts with the second user, causing virtual content corresponding to the interaction and at least one of the first user or the second user to be presented on a user interface corresponding to a mixed reality device associated with the first user. The user interface can present a view of the real scene as viewed by the first user that is enhanced with the virtual content.

Claims

1. A system comprising:

a sensor;

one or more processors;

memory; and

one or more modules stored in the memory and executable by the one or more processors to perform operations comprising;

receiving data from the sensor;

determining, based at least in part on receiving the data, that an object associated with a first user that is physically present in a real scene interacts with a second user that is present in the real scene via an interaction; and

based at least in part on determining that the object interacts with the second user, causing virtual content corresponding to the interaction and at least one of the first user or the second user to be presented on a user interface corresponding to a mixed reality device associated with the first user, wherein the user interface presents a view of the real scene as viewed by the first user that is enhanced with the virtual content.

2. The system as claim 1 recites, wherein the second user is physically present in the real scene.

3. The system as claim 1 recites, wherein:

the second user is physically present in a different real scene than the real scene; and

the operations further comprise causing the second user to be virtually present in the real scene by causing a graphic representation of the second user to be presented via the user interface.

4. The system as claim 1 recites, wherein the object comprises a virtual object associated with the first user.

5. The system as claim 1 recites, wherein the object comprises a body part of the first user.

6. The system as claim 5 recites, wherein:

receiving the data comprises;

receiving, from the sensor, at least one of first volumetric data or first skeletal data associated with the first user; and

receiving, from the sensor, at least one of second volumetric data or second skeletal data associated with the second user; and

the operations further comprise;

determining a first body representation associated with the first user based at least in part on the at least one of the first volumetric data or the first skeletal data;

determining a second body representation associated with the second user, based at least in part on the at least one of the second volumetric data or the second skeletal data; and

determining that the body part of the first user interacts with the second user based at least in part on determining that the first body representation is within a threshold distance of the second body representation.

7. The system as claim 1 recites, wherein the virtual content corresponding to the interaction is defined by the first user.

8. The system as claim 1 recites, wherein the sensor comprises an inside-out sensing sensor.

9. The system as claim 1 recites, wherein the sensor comprises an outside-in sensing sensor.

10. A method for causing virtual content to be presented in a mixed reality environment, the method comprising:

receiving, from a sensor, first data associated with a first user that is physically present in a real scene of the mixed reality environment;

determining, based at least in part on the first data, a first body representation that corresponds to the first user;

receiving, from the sensor, second data associated with a second user that is present in the real scene of the mixed reality environment;

determining, based at least in part on the second data, a second body representation that corresponds to the second user;

determining, based at least in part on the first data and the second data, an interaction between the first user and the second user; and

based at least in part on determining the interaction, causing virtual content to be presented in association with at least one of the first body representation or the second body representation on at least one of a first display associated with the first user or on a second display associated with the second user.

11. The method as claim 10 recites, further comprising receiving streaming data for causing the second user to be virtually present in the real scene of the mixed reality environment.

12. The method as claim 10 recites, wherein:

the first data comprises at least one of volumetric data associated with the first user, skeletal data associated with the first user, or perspective data associated with the first user; and

the second data comprises at least one of volumetric data associated with the second user, skeletal data associated with the second user, or perspective data associated with the second user.

13. The method as claim 10 recites, wherein the virtual content comprises a graphical representation of physiological data associated with at least the first user or the second user.

14. The method as claim 10 recites, wherein the virtual content comprises a graphical representation corresponding to a sticker, a tattoo, or an accessory that conforms to at least the first body representation or the second body representation at a position on at least the first body representation or the second body representation corresponding to the interaction.

15. The method as claim 14 recites, further comprising causing the graphical representation corresponding to the sticker, the tattoo, or the accessory to be presented to the first user and the second user each time the first user and the second user are present at a same time in the mixed reality environment.

16. The method as claim 10 recites, further comprising:

determining permissions associated with at least one of the first user or the second user; and

causing the virtual content to be presented in association with at least one of the first body representation or the second body representation based at least in part on the permissions.

17. A device configured to communicate with at least a first mixed reality device and a second mixed reality device in a mixed reality environment, the device comprising:

one or more processors;

memory; and

one or more modules stored in the memory and executable by the one or more processors to perform operations comprising;

receiving, from a sensor communicatively coupled to the device, first data associated with a first user that is physically present in a real scene of the mixed reality environment;

determining, based at least in part on the first data, a first body representation that corresponds to the first user;

receiving, from the sensor, second data associated with a second user that is physically present in the real scene of the mixed reality environment;

determining, based at least in part on the second data, a second body representation that corresponds to the second user;

determining, based at least in part on the first data and the second data, that the second user causes contact with the first user; and

based at least in part on determining that the second user causes contact with the first user, causing virtual content to be presented in association with the first body representation on a first display associated with the first mixed reality device and a second display associated with the second mixed reality device, wherein the first mixed reality device corresponds to the first user and the second mixed reality device corresponds to the second user.

18. A device as claim 17 recites, the operations further comprising:

determining, based at least in part on the first data, at least one of a volume outline or a skeleton that corresponds to the first body representation; and

causing the virtual content to be presented so that it conforms to the at least one of the volume outline or the skeleton.

19. A device as claim 17 recites, the operations further comprising:

segmenting the first body representation to generate a segmented first body representation; and

causing the virtual content to be presented on a segment of the segmented first body representation corresponding to a position on the first user where the second user causes contact with the first user.

20. A device as claim 17 recites, the operations further comprising causing the virtual content to be presented to visually indicate a position on the first user where the second user causes contact with the first user.