Motion control of a virtual environment

• US 9,317,112 B2
• Filed: 11/19/2013
• Issued: 04/19/2016
• Est. Priority Date: 11/19/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cao, Xiang, Shiratori, Takaaki, Tong, Xin, Xu, Feng, Gersten, Thomas, Leyvand, Tommer
• Primary Examiner(s): SHERMAN, STEPHEN G
• Application Number:
  US14/084,481
• Publication Number:
  US 20150138063A1
• Time in Patent Office: 882 Days
• Field of Search: 345/156
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

An optical flow of depth video of a depth camera imaging a human subject is recognized. An energy field created by motion of the human subject is generated as a function of the optical flow and specified rules of a physical simulation of the virtual environment. The energy field is mapped to a virtual position in the virtual environment. A property of a virtual object in the virtual environment is adjusted based on a plurality of energy elements of the energy field in response to the virtual object interacting with the virtual position of the energy field.

Claims

1. On a computing device, a computer-implemented method for controlling a virtual environment, comprising:

visually presenting, via a display, a representation of the virtual environment;

receiving, from a depth camera, depth video imaging a human subject;

recognizing a virtual skeleton that models the human subject with a plurality of virtual joints characterized by three-dimensional positions, the virtual skeleton derived from the depth video;

using the virtual skeleton to identify a gesture performed by the human subject;

recognizing an optical flow of the depth video, the optical flow derived from a plurality of cells mapped to the depth video, each cell having a motion value indicating a magnitude of motion in that cell recognized between consecutive image frames of the depth video;

using the optical flow to assign a magnitude to the gesture performed by the human subject; and

visually presenting, via the display, an updated representation of the virtual environment, the updated representation being modified based on the gesture, and the updated representation being modified in proportion to the magnitude.

2. The method of claim 1, wherein the plurality of cells is arranged in a grid of cells mapped to image frames of the depth video.

3. The method of claim 1, further comprising:

generating an energy field created by motion of the human subject as a function of the optical flow and specified rules of a physical simulation of the virtual environment, the energy field including a plurality of energy elements derived from the plurality of cells of the optical flow, each energy element including a plurality of energy properties derived from the motion value of a corresponding cell and the specified rules of the physical simulation, and wherein a property of the virtual environment is further controlled as a function of the energy field.

4. The method of claim 3, wherein the plurality of energy elements of the energy field is arranged in a three-dimensional array.

5. The method of claim 3, wherein generating the energy field includes filtering out motion values from cells indicative of a designated motion.

6. The method of claim 3, wherein the plurality of energy properties of the energy element include a position, magnitude, shape, velocity, and lifespan.

7. The method of claim 3, wherein the gesture is indicative of an attack and the property of the virtual environment is an amount of damage caused by the attack on a virtual object in the virtual environment.

8. The method of claim 3, wherein the property includes a virtual force applied to a virtual object in the virtual environment.

9. The method of claim 1, wherein the virtual skeleton includes a plurality of parts, each part including a corresponding weight such that motion of different parts are weighted differently in terms of magnitude of motion, and the magnitude is further based on a weight of the part associated with the gesture.

10. On a computing device, a computer-implemented method for controlling a virtual environment, comprising:

visually presenting, via a display, a representation of the virtual environment including a virtual object;

receiving, from a depth camera, depth video imaging a human subject;

recognizing an optical flow of the depth video, the optical flow derived from a plurality of cells mapped to the depth video, each cell having a motion value indicating a magnitude of motion recognized between consecutive image frames of the depth video;

generating an energy field created by motion of the human subject as a function of the optical flow and specified rules of a physical simulation of the virtual environment, the energy field including a plurality of energy elements derived from the plurality of cells of the optical flow, each energy element including a plurality of energy properties derived from the motion value of a corresponding cell and the specified rules of the physical simulation;

mapping the energy field to a virtual position in the virtual environment;

adjusting a property of the virtual object in the virtual environment based on one or more energy properties of the plurality of energy elements in response to the virtual object interacting with the virtual position of the energy field; and

visually presenting, via the display, an updated representation of the virtual environment, the updated representation including the virtual object having the adjusted property.

11. The method of claim 10, wherein the plurality of cells is arranged in a grid of cells mapped to image frames of the depth video.

12. The method of claim 10, wherein the plurality of energy elements of the energy field is arranged in a three-dimensional array, and wherein for any given image frame of the depth video the energy field includes an accumulation of energy elements actively contributing energy of motion in that image frame.

13. The method of claim 10, further comprising:

determining a region of interest of the energy field that interacts with the virtual object in the virtual environment, the region of interest being smaller than a volume of the energy field, and wherein the property of the virtual object is adjusted based on an accumulation of values of one or more energy properties of energy elements in the region of interest.

14. The method of claim 13, wherein the one or more energy properties includes a magnitude of motion of the energy elements in the region of interest, the property of the virtual object includes a damage tolerance level, and the magnitudes of the energy elements are accumulated to generate an amount of damage dealt to the virtual object to reduce the damage tolerance level.

15. The method of claim 13, wherein the one or more energy properties includes a magnitude of motion and a velocity of the energy elements in the region of interest, and the magnitudes and the velocities of the energy elements are accumulated to generate a momentum applied to the virtual object.

16. The method of claim 10, wherein generating the energy field includes filtering out motion values from cells representative of a designated type of motion.

17. The method of claim 10, wherein the plurality of energy properties of each energy element include a position, magnitude, shape, velocity, and lifespan.

18. The method of claim 10, wherein the specified rules of the physical simulation specify initialization values of the plurality of energy properties of each energy element and adjust the plurality of energy properties of energy elements in the energy field for image frames of the depth video that images the human subject.

19. On a computing device, a computer-implemented method for controlling a virtual environment, comprising:

visually presenting, via a display, a representation of the virtual environment including a virtual object;

receiving, from a depth camera, depth video imaging a human subject;

recognizing an optical flow of the depth video, the optical flow derived from a plurality of cells mapped to the depth video, each cell having a motion value indicating a magnitude of motion recognized between consecutive image frames of the depth video;

generating an energy field created by motion of the human subject as a function of the optical flow and specified rules of a physical simulation of the virtual environment, the energy field including a plurality of energy elements derived from the plurality of cells of the optical flow, each energy element including a plurality of energy properties derived from the motion value of a corresponding cell and the specified rules of the physical simulation;

mapping the energy field to a virtual position in the virtual environment;

determining a region of interest of the energy field that interacts with a virtual object in the virtual environment, the region of interest being smaller than a volume of the energy field;

adjusting a property of the virtual object in the virtual environment based on an accumulation of values of one or more energy properties of energy elements in the region of interest; and

visually presenting, via the display, an updated representation of the virtual environment, the updated representation including the virtual object having the adjusted property.

20. The method of claim 19, wherein the virtual object is a virtual character, the one or more energy properties includes a magnitude of motion of the energy elements in the region of interest, the property of the virtual character includes a damage tolerance level, and the magnitudes of the energy elements are accumulated to generate an amount of damage dealt to the virtual character to reduce the damage tolerance level.