Molding and anchoring physically constrained virtual environments to real-world environments

• US 9,911,232 B2
• Filed: 02/27/2015
• Issued: 03/06/2018
• Est. Priority Date: 02/27/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shapira, Lior, Freedman, Daniel
• Primary Examiner(s): Wu, Sing-Wai
• Application Number:
  US14/633,661
• Publication Number:
  US 20160253842A1
• Time in Patent Office: 1,103 Days
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/013 Force feedback applied to a game
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2021 Shape modification

Abstract

An “Anchored Environment Generator” generates a physically constrained virtual environment that is molded and anchored to a real-world environment around a user (or multiple users). This molding and anchoring of the physically constrained virtual environment ensures that at least a portion of the physically constrained virtual environment matches tactile truth for one or more surfaces and objects within the real-world environment. Real objects and surfaces in the real-world environment may appear as different virtual objects, and may have different functionality, in the physically constrained virtual environment. Consequently, users may move around within the physically constrained virtual environment while touching and interacting with virtual objects in the physically constrained virtual environment. In some implementations, the physically constrained virtual environment is constructed from virtual building blocks that are consistent with a theme-based specification (e.g., French bistro, space station, starship, alternate real-world environments, alien world, apocalyptic wasteland, medieval village, hospital, castle, etc.).

Claims

1. A computer-implemented process for constructing virtual environments, comprising:

using a computer to perform process actions for;

generating environmental information by scanning a real-world environment to classify geometry of surfaces and objects within that environment;

receiving a master set of virtual building blocks, each virtual building block including a plurality of characteristics and including semantic and geometric constraints relating to the virtual building blocks;

procedurally molding a physically constrained virtual environment by performing two or more iterations of a joint optimization of a layout of an automatically selected variable subset of at least two of the virtual building blocks selected for each iteration relative to a scene-specific cost function;

each iteration of the physically constrained virtual environment comprising a different trial solution of a joint layout of all the virtual building blocks in the automatically selected variable subset;

each iteration being consistent with the characteristics and constraints of the variable subset of virtual building blocks and the environmental information of the real-world environment;

the scene-specific cost function being derived for each iteration from the characteristics and constraints of the subset of virtual building blocks in combination with the environmental information;

displaying the physically constrained virtual environment via a display device, such that at least a portion of the physically constrained virtual environment is anchored to the real-world environment in a way that at least partially matches tactile truth for one or more surfaces and objects within the real-world environment.

2. The computer-implemented process of claim 1 further comprising process actions for rendering virtual representations of one or more real-world objects into the physically constrained virtual environment to enable one or more users to interact with those real-world objects.

3. The computer-implemented process of claim 1 further comprising process actions for enabling user interaction with the physically constrained virtual environment via speech-based commands.

4. The computer-implemented process of claim 1 further comprising process actions for enabling user interaction with the physically constrained virtual environment via gesture-based commands.

5. The computer-implemented process of claim 1 further comprising process actions for rendering a virtual object into a user'"'"'s hand within the physically constrained virtual environment.

6. The computer-implemented process of claim 1 further comprising process actions for rendering a virtual object to at least partially match tactile truth of a real object held in a user'"'"'s hand within the physically constrained virtual environment.

7. The computer-implemented process of claim 1 further comprising process actions for dynamically updating the physically constrained virtual environment in response to interaction by one or more users with one or more of the virtual building blocks of the physically constrained virtual environment.

8. The computer-implemented process of claim 1 further comprising process actions for dynamically updating the physically constrained virtual environment in response to movement of one or more users within the real-world environment.

9. The computer-implemented process of claim 1 further comprising process actions for dynamically updating the physically constrained virtual environment in response to movement of one or more real objects within the real-world environment.

10. The computer-implemented process of claim 1 further comprising process actions for applying a marked Markov point process that evaluates the scene-specific cost function to score each iteration of the jointly optimized layout of the physically constrained virtual environment.

11. The computer-implemented process of claim 1 further comprising process actions for limiting the automatically selected variable subset of the virtual building blocks to a theme-based specification.

12. The computer-implemented process of claim 1 wherein each iteration of the jointly optimized layout of the physically constrained virtual environment is also consistent with one or more objective constraints.

13. A method for generating an immersive virtual environment, comprising:

receiving a classification of geometry of surfaces and objects within a real-world environment;

receiving a set of virtual objects comprising virtual object characteristics and including semantic, geometric, and objective constraints relating to the virtual objects;

iteratively jointly optimizing, via two or more iterations, a layout of an automatically selected variable subset of at least two of the virtual objects selected for each iteration relative to a scene-specific cost function;

each iteration of the subset of virtual objects comprising a different trial solution of a joint layout of all of the virtual objects in the automatically selected variable subset of virtual objects;

each iteration being consistent with the geometry of surfaces and objects within the real-world environment;

wherein the scene-specific cost function scores each iteration of the jointly optimized layout based on the characteristics and constraints of the subset of virtual objects relative to the geometry of surfaces and objects within the real-world environment;

rendering the iteratively jointly optimized layout of virtual objects to generate a tactile virtual reality (TVR);

replacing a view of the real-world environment around a plurality of users by displaying the TVR to those users via head worn display devices, such that at least a portion of the TVR is anchored to the real-world environment in a way that at least partially matches tactile truth for one or more surfaces and objects within the real-world environment; and

dynamically performing real-time updates to the TVR in response to user interaction with the TVR.

14. The method of claim 13 further comprising rendering virtual representations of one or more real-world objects that exist within the real-world environment into corresponding locations within the TVR so that the virtual representations at least partially match tactile truth of the corresponding real-world objects.

15. The method of claim 13 further comprising applying one or more natural user interface (NUI) processes to enable user interaction with the TVR via any combination of speech-based commands, gestures, and facial expressions.

16. The method of claim 13 further comprising rendering a virtual object to at least partially match tactile truth of a real object held in a user'"'"'s hand within the TVR.

17. The method of claim 13 further comprising dynamically updating the TVR in real-time in response to movement of one or more real objects within the real-world environment corresponding to the TVR.

18. The method of claim 13 further comprising automatically selecting the variable subset of the virtual objects to be consistent with a theme-based specification.

19. A computer-readable storage device having computer executable instructions stored therein, said instructions causing a computing device to:

automatically classify geometry of surfaces and objects within a real-world environment in response to environmental information received from one or more scanning devices;

receive a set of virtual objects comprising virtual object characteristics and including semantic and geometric constraints relating to the virtual objects;

perform two or more iterations to iteratively jointly optimize a layout of an automatically selected variable subset of at least two of the virtual objects selected for each iteration relative to a scene-specific cost function that scores each iteration of the jointly optimized layout based on the characteristics and constraints of the subset of virtual objects relative to the geometry of surfaces and objects within the real-world environment;

wherein each iteration of jointly optimized layout of the subset of virtual objects comprises a different trial solution of a joint layout of all of the virtual objects of the automatically selected variable subset of virtual objects;

wherein each iteration is consistent with the geometry of surfaces and objects within the real-world environment;

render the iteratively jointly optimized layout of virtual objects to generate a tactile virtual reality (TVR) that is anchored to the real-world environment in a way that at least partially matches tactile truth for one or more surfaces and objects within the real-world environment; and

replace a view of the real-world environment around one or more users by displaying the TVR via an immersive display device.

20. The computer-readable storage device of claim 19 further comprising dynamically performing real-time updates to the TVR in response to any user interactions with any virtual objects in the TVR and any real objects within the real-world environment corresponding to the TVR.