Shared tactile interaction and user safety in shared space multi-person immersive virtual reality

• US 9,898,864 B2
• Filed: 05/28/2015
• Issued: 02/20/2018
• Est. Priority Date: 05/28/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shapira, Lior, Gal, Ran, Burger, Douglas Christopher
• Primary Examiner(s): Zhai, Kyle
• Application Number:
  US14/724,503
• Publication Number:
  US 20160350973A1
• Time in Patent Office: 999 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A “Shared Tactile Immersive Virtual Environment Generator” (STIVE Generator) constructs fully immersive shared virtual reality (VR) environments wherein multiple users share tactile interactions via virtual elements that are mapped and rendered to real objects that can be touched and manipulated by multiple users. Generation of real-time environmental models of shared real-world spaces enables mapping of virtual interactive elements to real objects combined with multi-viewpoint presentation of the immersive VR environment to multiple users. Real-time environmental models classify geometry, positions, and motions of real-world surfaces and objects. Further, a unified real-time tracking model comprising position, orientation, skeleton models and hand models is generated for each user. The STIVE Generator then renders frames of the shared immersive virtual reality corresponding to a real-time field of view of each particular user. Each of these frames is jointly constrained by both the real-time environmental model and the unified real-time tracking model.

Claims

1. A computer-implemented process for constructing shared virtual environments, comprising:

applying a computer to perform process actions for;

generating a real-time environmental model of a real-world environment in which a shared immersive virtual environment is being presented to two or more users;

generating a unified real-time tracking model for each of the two or more users;

for each particular user, applying the real-time environmental model and the real-time tracking model to generate frames of the shared immersive virtual environment corresponding to a real-time field of view of the particular user;

rendering virtual representations of one or more non-participant persons into the shared immersive virtual environment in positions corresponding to a real-world position of each non-participant person; and

presenting the virtual representation of one or more of the non-participant persons as a translucent rendering that increases in solidity whenever it is determined that the non-participant person is speaking to any user.

2. The computer-implemented process of claim 1 wherein each unified real-time tracking model comprises one or more of user position, user orientation, user skeleton models and user hand models.

3. The computer-implemented process of claim 1 wherein each unified real-time tracking model includes user facial expression modeling.

4. The computer-implemented process of claim 1 further comprising process actions for rendering virtual representations of each user into the shared immersive virtual environment in positions corresponding to a real-world position of each user.

5. The computer-implemented process of claim 1 further comprising process actions for rendering one or more indicia of safety into the shared immersive virtual environment.

6. The computer-implemented process of claim 5 wherein one or more of the indicia of safety is represented by a rendering of an indication of a safe path within the shared immersive virtual environment that is clear of real-world obstacles.

7. The computer-implemented process of claim 5 wherein one or more of the indicia of safety is represented by a translucent rendering of a non-participant person within the real-world environment that increases in solidity as any user approaches that translucent rendering.

8. The computer-implemented process of claim 1 further comprising process actions for mapping and rendering one or more virtual elements of the shared immersive virtual environment to one or more real objects in the real-world environment and interacting with one or more of those virtual elements via one or more gesture-based commands performed by one or more of the users.

9. The computer-implemented process of claim 1 further comprising process actions presenting the shared immersive virtual environment to each user via a virtual reality display device that completely replaces any view of the real-world environment with the shared immersive virtual environment.

10. A system for constructing interactive virtual reality environments, comprising:

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to;

apply one or more sensors to capture real-time tracking information relating to real motions and positions in a shared real-world environment of two or more users participating in a real-time rendering of a shared immersive virtual environment, and further relating to motions and positions of one or more non-participant persons in the shared real-world environment;

apply the tracking information to generate a real-time environmental model of the real-world environment;

apply the tracking information to generate a real-time position and skeleton model for each of the two or more users and one or more of the non-participant persons;

apply the real-time environmental model and the real-time position and skeleton models to generate frames of the shared immersive virtual environment that are tailored to a corresponding real-time field of view of each user;

render virtual representations of one or more of the non-participant persons into the shared immersive virtual environment in positions corresponding to a real-world position of each non-participant person; and

present the virtual representation of one or more of the non-participant persons as a translucent rendering that increases in solidity whenever it is determined that the non-participant person is speaking to any user.

11. The system of claim 10 further comprising applying the tracking information to generate a facial expression model for each user.

12. The system of claim 11 further comprising rendering virtual representations of each user into the shared immersive virtual environment in positions corresponding to a real-world position of each user, each virtual representation including real-time facial expressions of each corresponding user.

13. The system of claim 10 further comprising rendering one or more indicia of safety into the shared immersive virtual environment to alert users as to potential unintended contact with real objects in the shared real-world environment.

14. The system of claim 10 further comprising detecting one or more gesture-based commands performed by one or more of the users for use in interacting with one or more virtual elements mapped and rendered into the shared immersive virtual environment.

15. A computer-readable storage device having computer executable instructions stored therein, said instructions causing a computing device to execute a method comprising:

applying one or more sensors to capture real-time tracking information of users, non-participant persons, and objects in a shared real-world environment;

applying the tracking information to generate a real-time environmental model of the real-world environment;

applying the tracking information to generate a real-time position and skeleton model for each of two or more users and one or more of the non-participant persons;

generating frames of a shared immersive virtual environment that are tailored to corresponding real-time field of view of each user;

applying the real-time environmental model and the real-time position and skeleton models to render virtual representations of one or more non-participant persons into the shared immersive virtual environment in positions corresponding to a real-world position of each non-participant person; and

presenting one or more of the virtual representations of one or more non-participant persons as a translucent rendering that increases in solidity whenever it is determined that the non-participant person is speaking to any user.

16. The computer-readable storage device of claim 15 further comprising rendering one or more indicia of safety into the shared immersive virtual environment to alert users as to potential unintended contact with real objects in the shared real-world environment.

17. The computer-readable storage device of claim 15 further comprising detecting one or more gesture-based commands performed by one or more of the users for use in interacting with one or more virtual elements mapped and rendered into the shared immersive virtual environment.