Apparatus and method for three-dimensional image production and presenting real objects in virtual three-dimensional space

• US 20020024517A1
• Filed: 07/12/2001
• Published: 02/28/2002
• Est. Priority Date: 07/14/2000
• Status: Abandoned Application

• Current Assignee: Komatsu Ltd.
• Original Assignee: Komatsu Ltd.
• Inventors: Yamaguchi, Hiroyoshi, Hamachi, Hisashi, Shinbo, Tetsuya
• Application Number:
  US09/902,644
• Publication Number:
  US 20020024517A1
• Time in Patent Office: Days
• US Class Current: 345/424
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/50 characterized by details of game servers
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/279 : the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/289 : Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 19/597 : specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals

Abstract

Using a stereo viewing method, three-dimensional model data are produced that completely express an object in a three-dimensional shape, or moving images of the object as seen from any viewpoint are produced.

The object 10 is photographed by a plurality of multi-eyes stereo cameras 11, 12, and 13 deployed at different locations, and, for each of the multi-eyes stereo cameras 11, 12, and 13, a brightness image of the object 10 and a distance image indicating the distance to the outer surface of the object 10 are obtained. Based on these brightness images and distances, voxels in which the outer surface of the object 10 exists are determined, out of a multiplicity of voxels 30 virtually defined by finely dividing a space 20 into which the object 10 has entered, and the brightness of the object 10 in each of those voxels is determined. Based on these results, a three-dimensional model of the object 10 is produced, and, using that three-dimensional model, images looking at the object 10 from any viewpoint 40 are rendered. As a modification, the production of the three-dimensional model of the object 10 can be omitted, and images looking at the object 10 from any viewpoint 40 made directly on the basis of the brightness images and distance images described above.

Claims

1. A three-dimensional modeling apparatus comprising:

a stereo processing unit that receives images from a plural number of stereo cameras deployed at different locations so as to photograph a same object, and produces a plurality of distance images of said object from images from said plural number of stereo cameras;

a voxel processing unit that receives said plurality of distance images from said stereo processing unit, and, from among a multiplicity of voxels established beforehand in a prescribed space into which said object enters, selects voxels in which the surface of said object exists; and

a modeling unit for producing a three-dimensional model of said object, based on coordinates of the voxels selected by said voxel processing unit.

2. The three-dimensional modeling apparatus according to claim 1, wherein said stereo cameras output moving images respectively;

and, for each frame of said moving images from said stereo cameras, said stereo processing unit, said voxel processing unit, and said modeling unit are configured so as to perform, respectively, a process for producing said distance images, a process for selecting voxels in which the surface of said object exists, and a process for producing a three-dimensional model of said object.

3. A three-dimensional modeling method comprising the steps of:

receiving images from a plural number of stereo cameras deployed at different locations so as to photograph a same object, and producing a plurality of distance images of said object from images from said plural number of stereo cameras;

receiving a plurality of said distance images and selecting, from among a multiplicity of voxels established beforehand inside a prescribed space into which said object enters, voxels in which the surface of said object exists; and

producing a three-dimensional model of said object based on coordinates of said selected voxels.

4. A three-dimensional image production apparatus comprising:

a stereo processing unit that receives images from a plural number of stereo cameras deployed at different locations so as to photograph a same object, and produces a plurality of distance images of said object from images from said plural number of stereo cameras;

an object detection unit that receives said plurality of distance images from said stereo processing unit, and determines coordinates in which the surface of said object exists, in a viewpoint coordinate system referenced to a viewpoint established at any location; and

a target image production unit for producing an image of said object as seen from said viewpoint, based on said coordinates determined by said object detection unit.

5. The three-dimensional image production apparatus according to claim 4, wherein said stereo cameras are output moving images, respectively;

and, for each frame of said moving images from said stereo cameras, said stereo processing unit, said object detection unit, and said target image production unit are configured so as to perform, respectively, a process for producing said distance images, a process for determining coordinates wherein surface of said object exists, and a process for producing images of said object.

6. A three-dimensional image production method comprising the steps of:

receiving images from a plural number of stereo cameras deployed at different locations so as to photograph a same object, and producing a plurality of distance images of said object from images from said plural number of stereo cameras;

receiving plurality of said distance images and determining coordinates in which the surface of said object exists, in a viewpoint coordinate system referenced to a viewpoint established at any location; and

producing an image of said object as seen from said viewpoint, based on said determined coordinates.

7. A system for enabling a real physical object to appear in virtual three-dimensional space in a computer application used by a user, comprising:

photographed data reception means for receiving photographed data produced by stereo photographing a real physical object, from a stereo photographing apparatus usable by said user, capable of communicating with said stereo photographing apparatus;

modeling means for producing a three-dimensional model of said physical object, based on said received photographed data, in a prescribed data format that can be imported into virtual three-dimensional space by said computer application; and

three-dimensional model output means for outputting three-dimensional model data of said physical object, by a method wherewith those data can be presented to said user or to a computer application used by said user.

8. The system according to claim 7, wherein said photographed data received from said stereo photographing apparatus comprise photographed data for a plurality of poses photographed, respectively, when said real physical object assumed different poses;

and said modeling means produce said three-dimensional model data for said physical object, based on said photographed data for said plurality of poses, in such a configuration that different poses can be assumed or different motions reproduced.

9. The system according to claim 7, wherein said photographed data received from said stereo photographing apparatus comprise photographed data for moving images photographed when said real physical object performed some motion;

and said modeling means produce said three-dimensional model data for said physical object, based on said photographed data for said moving images, in such a configuration that same motion as that performed by said physical object is reproduced.

10. The system according to claim 9, wherein said modeling means produce said three-dimensional model data so that same motion is reproduced as the motion performed by said real physical object, following said latter motion substantially in real time during the photographing by said stereo photographing apparatus.

11. A method for enabling a real physical object to appear in virtual three-dimensional space in a computer application used by a user, comprising the steps of:

receiving photographed data produced by stereo photographing a real physical object from a stereo photographing apparatus that can be used by said user;

producing a three-dimensional model of said physical object, based on said photographed data received, in a prescribed data format capable of being imported into virtual three-dimensional space by said computer application; and

outputting three-dimensional model data for said physical object by a method that enables the date to be provided to said user or to said computer application used by said user.

12. A system for enabling a real physical object to appear in virtual three-dimensional space in a computer application used by a user, comprising:

a stereo photographing apparatus that can be used by said user; and

a modeling apparatus capable of communicating with said stereo photographing apparatus, and also capable of communicating with a computer apparatus that can be used by said user;

wherein said modeling apparatus has;

photographed data receiving means for receiving photographed data produced by stereo photographing a real physical object from said stereo photographing apparatus;

modeling means for producing a three-dimensional model of said physical object, based on said photographed data received, in a prescribed data format capable of being imported into virtual three-dimensional space by said computer application; and

three-dimensional model transmission means for transmitting three-dimensional model data for said physical object to said computer apparatus that can be used by said user.

13. A system for enabling a real physical object to appear in virtual three-dimensional space in a computer application used by a user, comprising:

a computer apparatus for execution of said computer application by said user;

a stereo photographing apparatus that can be used by said user; and

a modeling apparatus capable of communicating with said stereo photographing apparatus and said computer apparatus;

wherein said modeling apparatus has;

photographed data receiving means for receiving photographed data produced by stereo photographing a real physical object from said stereo photographing apparatus;

modeling means for producing a three-dimensional model of said physical object, based on said photographed data received, in a prescribed data format capable of being imported into virtual three-dimensional space by said computer application; and

three-dimensional model transmission means for transmitting three-dimensional model data for said physical object to said computer apparatus.

14. A method for enabling a real physical object to appear in virtual three-dimensional space in a computer application used by a user, comprising the steps of:

stereo-photographing a real physical object;

producing a three-dimensional model of said physical object, based on photographed data of said physical object obtained by stereo photographing, in a prescribed data format capable of being imported into virtual three-dimensional space by said computer application; and

inputting three-dimensional model data for said physical object into a computer apparatus capable of executing said computer application.