Storage medium having stored therein stereoscopic image display program, stereoscopic image display device, stereoscopic image display system, and stereoscopic image display method

• US 8,842,166 B2
• Filed: 06/07/2011
• Issued: 09/23/2014
• Est. Priority Date: 06/14/2010
• Status: Active Grant

• Current Assignee: HAL Laboratory, Inc., Nintendo Company Limited
• Original Assignee: HAL Laboratory, Inc., Nintendo Company Limited
• Inventors: Ito, Yuichiro, Nishimura, Yuki
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): WALKER, JARED T
• Application Number:
  US13/154,964
• Publication Number:
  US 20110304700A1
• Time in Patent Office: 1,204 Days
• Field of Search: 348/47
• US Class Current: 348/47
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/5252 using two or more virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character changes room or displaying a rear-mirror view in a car-driving game
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/30 characterized by output arrangements for receiving control signals generated by the game device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/302 : for viewing without the aid of special glasses, i.e. using autostereoscopic displays

Abstract

A game device 10 calculates, for each of two real cameras, position/orientation information representing a relative position and orientation with respect to a predetermined image-capture object, and based on this calculates a camera interval in the predetermined coordinate system. Then, it calculates correspondence information between a unit length in the predetermined coordinate system and a unit length in a real space by using the calculated distance in the predetermined coordinate system and the camera interval in the real world.

Claims

1. A non-transitory computer-readable storage medium having stored therein an image display program for stereoscopically displaying a real world image synthesized with a three-dimensional virtual object on a screen of a stereoscopically-viewable stereoscopic display member by using outputs from a right-eye real camera and a left-eye real camera, wherein:

the image display program comprising instructions that instructs at least one processor to;

recognize a predetermined image-capture object in right captured image data outputted from the right-eye real camera so as to calculate first position/orientation information representing a relative position and orientation between the right-eye real camera and the predetermined image-capture object in a predetermined coordinate system;

recognize a predetermined image-capture object in left captured image data outputted from the left-eye real camera so as to calculate second position/orientation information representing a relative position and orientation between the left-eye real camera and the predetermined image-capture object in a predetermined coordinate system;

determine a position and an orientation for both of a right virtual camera for producing a right-eye image and a left virtual camera for producing a left-eye image in a predetermined virtual space by using at least one of the first position/orientation information and the second position/orientation information;

producing a right virtual image showing the virtual space as seen from the right virtual camera;

producing a left virtual image showing the virtual space as seen from the left virtual camera;

synthesizing the right virtual image with the right captured image and synthesizing the left virtual image with the left captured image so as to output an image for stereoscopic viewing to the stereoscopic display member;

calculating a distance in the predetermined coordinate system between the right-eye real camera and the left-eye real camera by using the calculated first position/orientation information and the calculated second position/orientation information; and

calculating a ratio between a unit length in the predetermined coordinate system and a unit length in a real space by using the calculated distance in the predetermined coordinate system and a distance in a real world between the right-eye real camera and the left-eye real camera.

2. The non-transitory computer-readable storage medium according to claim 1, wherein the image display program instructs the computer to:

perform a process of matching the unit length in the predetermined coordinate system with the unit length in the real space based on the calculated ratio.

3. The non-transitory computer-readable storage medium according to claim 1, wherein the image display program instructs the computer to:

place a predetermined virtual object in the predetermined coordinate system at a position that is away from a predetermined position in the coordinate system by a predetermined distance in the real world.

4. The non-transitory computer-readable storage medium according to claim 1, wherein the image display program instructs the computer to:

calculate a virtual distance in the predetermined coordinate system to the predetermined image-capture object in an image-capturing direction of the right-eye real camera or the left-eye real camera by using the calculated first position/orientation information and the calculated second position/orientation information; and

calculate a distance in the real space to the predetermined image-capture object based on the calculated virtual distance and the calculated ratio.

5. The non-transitory computer-readable storage medium according to claim 1, wherein the image display program instructs the computer to:

set a reference distance for a distance from the real camera in an image-capturing direction using the calculated distance as a reference,

wherein the display control displays the right captured image in the display area so that a reference point in the right captured image is displayed at a center of the display area and displays the left captured image in the display area so that a reference point in the left captured image is displayed at the center of the display area, wherein the reference point is a point that is away from the real cameras by the reference distance in the image-capturing direction of the two real cameras on a straight line passing through a median point between the two real cameras and extending in the image-capturing direction.

6. The non-transitory computer-readable storage medium according to claim 1, wherein the display control sets a projection matrix of the right virtual camera so that a virtual reference point in the right virtual image is displayed at a center of the display area and sets a projection matrix of the left virtual camera so that a virtual reference point in the left virtual image is displayed at a center of the display area, wherein the virtual reference point is a point that is away from the virtual cameras by a distance corresponding to the reference distance in an image-capturing direction of the two virtual cameras on a straight line passing through a median point between the two virtual cameras and extending in the image-capturing direction.

7. An image display device for stereoscopically displaying a real world image synthesized with a three-dimensional virtual object on a screen of a stereoscopically-viewable stereoscopic display member by using outputs from a right-eye real camera and a left-eye real camera, comprising at least one processor configured to:

recognize a predetermined image-capture object in right captured image data outputted from the right-eye real camera so as to calculate first position/orientation information representing a relative position and orientation between the right-eye real camera and the predetermined image-capture object in a predetermined coordinate system;

recognize a predetermined image-capture object in left captured image data outputted from the left-eye real camera so as to calculate second position/orientation information representing a relative position and orientation between the left-eye real camera and the predetermined image-capture object in a predetermined coordinate system;

determine a position and an orientation for both of a right virtual camera for producing a right-eye image and a left virtual camera for producing a left-eye image in a predetermined virtual space by using at least one of the calculated first position/orientation information and the calculated second position/orientation information;

produce a right virtual image showing the virtual space as seen from the right virtual camera;

produce a left virtual image showing the virtual space as seen from the left virtual camera;

synthesize the right virtual image with the right captured image and synthesize the left virtual image with the left captured image so as to output an image for stereoscopic viewing to the stereoscopic display member;

calculate a distance in the predetermined coordinate system between the right-eye real camera and the left-eye real camera by using the calculated first position/orientation information and the calculated second position/orientation information; and

calculate a ratio between a unit length in the predetermined coordinate system and a unit length in a real space by using the calculated distance in the predetermined coordinate system and a distance in a real world between the right-eye real camera and the left-eye real camera.

8. An image display system for stereoscopically displaying a real world image synthesized with a three-dimensional virtual object on a screen of a stereoscopically-viewable stereoscopic display member by using outputs from a right-eye real camera and a left-eye real camera, comprising:

a predetermined image-capture object;

at least one processor configured to recognize the predetermined image-capture object in right captured image data outputted from the right-eye real camera so as to calculate first position/orientation information representing a relative position and orientation between the right-eye real camera and the predetermined image-capture object in a predetermined coordinate system;

the at least one processor further configured to recognize the predetermined image-capture object in left captured image data outputted from the left-eye real camera so as to calculate second position/orientation information representing a relative position and orientation between the left-eye real camera and the predetermined image-capture object in a predetermined coordinate system;

the at least one processor being further configured to determine a position and an orientation for both of a right virtual camera for producing a right-eye image and a left virtual camera for producing a left-eye image in a predetermined virtual space by using at least one of the calculated first position/orientation information and the calculated second position/orientation information;

the at least one processor being further configured to produce a right virtual image showing the virtual space as seen from the right virtual camera and to produce a left virtual image showing the virtual space as seen from the left virtual camera;

the at least one processor being further configured to synthesize the right virtual image with the right captured image and synthesize the left virtual image with the left captured image so as to output an image for stereoscopic viewing to the stereoscopic display member;

the at least one processor being further configured to calculate a distance in the predetermined coordinate system between the right-eye real camera and the left-eye real camera by using the calculated first position/orientation information and the calculated second position/orientation information; and

the at least one processor being further configured to calculate a ratio between a unit length in the predetermined coordinate system and a unit length in a real space by using the calculated distance in the predetermined coordinate system and a distance in a real world between the right-eye real camera and the left-eye real camera.

9. An image display method carried out by a computer for stereoscopically displaying a real world image synthesized with a three-dimensional virtual object on a screen of a stereoscopically-viewable stereoscopic display member by using outputs from a right-eye real camera and a left-eye real camera, comprising:

recognizing a predetermined image-capture object in right captured image data outputted from the right-eye real camera so as to calculate first position/orientation information representing a relative position and orientation between the right-eye real camera and the predetermined image-capture object in a predetermined coordinate system;

recognizing a predetermined image-capture object in left captured image data outputted from the left-eye real camera so as to calculate second position/orientation information representing a relative position and orientation between the left-eye real camera and the predetermined image-capture object in a predetermined coordinate system;

determining a position and an orientation for both of a right virtual camera for producing a right-eye image and a left virtual camera for producing a left-eye image in a predetermined virtual space by using at least one of the calculated first position/orientation information and the calculated second position/orientation information;

producing a right virtual image showing the virtual space as seen from the right virtual camera;

producing a left virtual image showing the virtual space as seen from the left virtual camera;

synthesizing the right virtual image with the right captured image and synthesizing the left virtual image with the left captured image so as to output an image for stereoscopic viewing to the stereoscopic display member;

calculating a distance in the predetermined coordinate system between the right-eye real camera and the left-eye real camera by using the calculated position/orientation information and the calculated second position/orientation information; and

calculating a ratio between a unit length in the predetermined coordinate system and a unit length in a real space by using the calculated distance in the predetermined coordinate system and a distance in a real world between the right-eye real camera and the left-eye real camera.