Using a Three-Dimensional Environment Model in Gameplay

• US 20120194517A1
• Filed: 01/31/2011
• Published: 08/02/2012
• Est. Priority Date: 01/31/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Izadi, Shahram, Molyneaux, David, Hilliges, Otmar, Kim, David, Shotton, Jamie Daniel Joseph, Kohli, Pushmeet, Fitzgibbon, Andrew, Hodges, Stephen Edward, Butler, David Alexander
• Granted Patent: US 8,570,320 B2
• Time in Patent Office: Days
• US Class Current: 345/420
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6009 for importing or creating game content, e.g. authoring tools during game development, adapting content to different platforms, use of a scripting language to create content
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Use of a 3D environment model in gameplay is described. In an embodiment, a mobile depth camera is used to capture a series of depth images as it is moved around and a dense 3D model of the environment is generated from this series of depth images. This dense 3D model is incorporated within an interactive application, such as a game. The mobile depth camera is then placed in a static position for an interactive phase, which in some examples is gameplay, and the system detects motion of a user within a part of the environment from a second series of depth images captured by the camera. This motion provides a user input to the interactive application, such as a game. In further embodiments, automatic recognition and identification of objects within the 3D model may be performed and these identified objects then change the way that the interactive application operates.

Claims

1. An interactive computing system comprising:

an input for receiving a first series of depth images from a mobile depth camera when in motion and a second series of depth images from the mobile depth camera when static;

a real-time camera tracking engine for tracking a position of the mobile depth camera when in motion;

a dense 3D environment modeling system arranged to generate a dense 3D model of the environment from any of the series of depth images and position data from the real-time camera tracking engine; and

an integration engine arranged to integrate at least a part of the dense 3D model into an application and to provide user input to the application by tracking real-time movement in a part of the environment using the second series of depth images.

2. An interactive computing system according to claim 1, further comprising:

an object recognition module arranged to analyze the dense 3D model of the environment and identify objects within the model.

3. An interactive computing system according to claim 2, wherein the integration engine is further arranged to adapt the application based on an object identified within the dense 3D model.

4. An interactive computing system according to claim 3, wherein adapting the application comprises automatically determining position calibration information for an object based on a detected relative location of a display and the mobile depth camera.

5. An interactive computing system according to claim 4, wherein the object comprises a pointing device or a user of the system.

6. An interactive computing system according to claim 1, further comprising the mobile depth camera.

7. An interactive computing system according to claim 6, further comprising a plurality of additional depth cameras.

8. An interactive computing system according to claim 3, wherein the identified object is a projector and the integration engine is arranged to use the 3D model to influence output of the projector.

9. An interactive computing system according to claim 1, wherein the system is a gaming system and the application is a game.

10. A method of operating an interactive computing system comprising:

receiving a series of depth images from a mobile depth camera in motion around an environment;

generating a dense 3D model of the environment from the series of depth images;

integrating at least a part of the dense 3D model into an application;

receiving a second series of depth images from the mobile depth camera when not in motion; and

tracking real-time movement in 3D in a region of the environment using the second series of depth images.

11. A method according to claim 10, further comprising:

generating the dense 3D model using the second series of depth images.

12. A method according to claim 11, further comprising:

updating the dense 3D model using the second series of depth images.

13. A method according to claim 10, wherein integrating at least a part of the dense 3D model into an application comprises:

analyzing the dense 3D model to automatically identify objects in the model;

where an object is identified, inputting object type and object location information into the application; and

adjusting operation of the application based on the input object type and location information.

14. A method according to claim 13, wherein the object comprises an active object and adjusting performance of the application comprises adjusting an output of the active object.

15. A method according to claim 13, wherein the object comprises a display and adjusting performance of the application comprises automatically calibrating a position of an object relative to the display based on a detected relative position of the display and the mobile depth camera.

16. A method according to claim 13, wherein adjusting performance of the application comprises adjusting the output of another detected object.

17. A method according to claim 13, wherein the object comprises another depth camera and adjusting performance of the application comprises calibrating the system based on a relative position of each depth camera.

18. A method according to claim 10, further comprising:

receiving a series of depth images from each of one or more additional depth cameras;

generating a dense 3D model of the environment from each series of depth images;

identifying common features in more than one dense 3D model; and

computing relative locations of cameras based on the identified common features.

19. A method according to claim 10, wherein the interactive computing system is a gaming system and the application is a game.

20. A gaming system having a first and second phase of operation, the gaming system comprising:

an input for receiving a first series of depth images from a mobile depth camera in the first phase of operation and a second series of depth images from the mobile depth camera in the second phase of operation, wherein the mobile depth camera is in motion in the first phase of operation and is static in the second phase of operation;

a real-time camera tracking engine arranged to track a position of the mobile depth camera in the first phase of operation;

a dense 3D modeling system arranged to generate a 3D model of a room using at least the first series of depth images and the position of the mobile depth camera when each depth image was captured; and

a gaming engine arranged to integrate at least a part of the dense 3D model into a game and to track real-time movement in a part of the room using the second series of depth images as a user input to the game.