Video game play using panoramically-composited depth-mapped cube mapping

• US 7,256,779 B2
• Filed: 08/08/2003
• Issued: 08/14/2007
• Est. Priority Date: 05/08/2003
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Donnelly, Paul
• Primary Examiner(s): Chauhan; Ulka J.
• Assistant Examiner(s): Pappas; Peter-Anthony
• Application Number:
  US10/636,978
• Publication Number:
  US 20040222988A1
• Time in Patent Office: 1,467 Days
• Field of Search: 345/621, 345/625, 463/33
• US Class Current: 345/421
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/405 using Z-buffer

Abstract

Video game play rendered using a panoramic view of a cube map style rendering uses an associated depth map to supply three-dimensionality to the pre-rendered scene. The resulting panoramic rendering may be indistinguishable from rendering the original scene in real-time except that the background is of pre-rendered quality.

Claims

1. A video game playing method comprising:

loading a pre-determined environment-cube-mapped image having multiple planar projected images and associated depth maps;

receiving real-time interactive inputs from a user;

at least in part in response to said real-time interactive user inputs, compositing at least one additional three-dimensional object, having plural non-coplanar surfaces, into a scene comprising said environment-cube-mapped images;

in response to said received user input, moving said three-dimensional object having said plural non-coplanar surfaces within said environment-cube-mapped images, said compositing using said depth maps to selectively render at least portions of said three-dimensional object to provide a composite image;

said compositing performing depth comparisons between depth values associated with said plural non-coplanar surfaces of said three-dimensional object and said depth maps associated with said environment-cube-mapped multiple planar projected images so as to selectively allow said three-dimensional object surfaces to be partially occluded by portions of said environment-cube-mapped images as said user inputs cause said three-dimensional object to move and to thereby interact with said environment-cube-mapped images; and

using a three-dimensional graphics engine to panoramically render said composite image using a desired viewing angle to provide interactive 3D video game play.

2. The method of claim 1 wherein said environment-cube-mapped image is pre-rendered.

3. The method of claim 1 including performing said compositing with a home video game system, and further including displaying rendering results on a home color television set.

4. The method of claim 1 wherein said receiving comprises receiving interactive real-time user inputs via at least one handheld controller, and said method further includes defining animation of said three-dimensional object at least in part in response to said inputs.

5. The method of claim 1 wherein said rendering comprises applying said composite image to a mesh, and rendering said mesh using a projection matrix to apply desired viewing angle and frustum parameters.

6. The method of claim 1 further including testing whether said three-dimensional object intersects any of multiple faces of said environment-cube-mapped image in at least two dimensions.

7. The method of claim 1 further including performing Z comparison and removing hidden surfaces in response to said Z comparison.

8. The method of claim 1 further including performing said rendering using a frame buffer.

9. The method of claim 1 further including detecting collisions between the three-dimensional object surfaces and the environment cube mapped images.

10. A computer readable storage medium storing instructions that, when executed by a home video game system or personal computer, provide interactive real-time video game play on a display at least in part in response to user input, said storage medium storing:

at least one pre-rendered cubic environment map;

at least one pre-rendered depth map corresponding to said cubic environment map;

instructions that, when executed, composite at least portions of said pre-rendered cubic environment map with at least one dynamically-generated three-dimensional object having plural non-coplanar surfaces, said compositing based at least in part on said pre-rendered depth map;

instructions that, in response to received user inputs, interactively move said three-dimensional object within said cubic environment map;

said compositing performing depth comparisons between said plural non-coplanar surfaces of said three-dimensional object and said cubic environment map so as to selectively allow said three-dimensional object surfaces to be partially occluded by portions of said cubic environment map as said user input causes said three-dimensional object to move within and interact with said cubic environment map; and

instructions that control a three-dimensional graphics engine to generate a rendered panoramic image based at least in part on said depth comparisons.

11. The storage medium of claim 10 wherein said pre-rendered cubic environment map comprises six images as if looking through faces of a cube with the viewpoint at the center of the cube.

12. The storage medium of claim 10 wherein said compositing includes rendering a real-time object into a frame buffer storing said pre-rendered cubic environment map using dynamically defined view port and/or frustum parameters.

13. The storage medium of claim 10 further storing instructions that detect a collision between the three-dimensional object and the cubic environment map.

14. A video game playing system comprising:

means for loading a pre-determined environment-cube-mapped image having multiple planar projected images and associated multiple depth maps;

means for, at least in part in response to real-time interactive user input, compositing into said mapped images at least one additional three dimensional object having plural non-coplanar surfaces;

said compositing means using said depth maps to selectively render at least portions of said three-dimensional object within the cube mapped image to provide a composited mapped image;

means for receiving interactive inputs from a user;

means for moving said three-dimensional object within said mapped image at least in part in response to said received user inputs,

said compositing means dynamically performing depth comparisons between said plural non-coplanar surfaces of said three-dimensional object and said mapped image so as to selectively allow said three-dimensional object surfaces to be partially hidden by portions of said mapped image as said user input moves said three-dimensional object within said mapped image; and

means for using a three-dimensional graphics engine to panoramically render said composited mapped image using a desired viewing angle to provide interactive 3D video game play.

15. The system of claim 14 wherein said pre-determined environment cube mapped image is created at authoring time.

16. The system of claim 14 wherein said pre-determined environment cube mapped image is created before game play.

17. The system of claim 14 wherein said compositing means includes means for compositing said composited mapped image in pieces less than the full size of a cube face.

18. The system of claim 14 wherein said compositing means includes means for representing said depth maps using Z textures.

19. The system of claim 14 wherein said compositing means composites and outputs only those cube face images which overlap both the final frustum and real-time objects to be imaged.

20. The system of claim 14 further including means for using at least one movie texture to provide said environment cube mapped image, and means for specifying portions of said environment map to be animated.

21. The system of claim 14 further including means for detecting a collision between the three-dimensional object surfaces and the cube mapped image.

22. A video game playing system comprising:

means for loading a pre-determined environment-mapped image having multiple planar projected images and associated multiple depth map images;

means for at least in part in response to real-time interactive user input, compositing at least one additional object into said mapped images, said compositing using said depth map to selectively render at least portions of said object into the cube mapped image to provide a composited mapped image; and

means for panoramically rendering said composited mapped image using a desired viewing angle and frustum to provide interactive video game play,

wherein said multiple depth map images are biased towards using background Z value as the representative Z value for pixels whose RGB is a mixture of foreground and background colors.