Storage medium having stored therein stereoscopic image display program, stereoscopic image display device, stereoscopic image display system, and stereoscopic image display method

• US 8,902,298 B2
• Filed: 06/07/2011
• Issued: 12/02/2014
• Est. Priority Date: 06/14/2010
• Status: Active Grant

• Current Assignee: HAL Laboratory, Inc., Nintendo Company Limited
• Original Assignee: HAL Laboratory, Inc., Nintendo Company Limited
• Inventors: Ito, Yuichiro, Nishimura, Yuki
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): WALKER, JARED T
• Application Number:
  US13/154,991
• Publication Number:
  US 20110304711A1
• Time in Patent Office: 1,274 Days
• Field of Search: 348/51
• US Class Current: 348/51
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/5252 using two or more virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character changes room or displaying a rear-mirror view in a car-driving game
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/30 characterized by output arrangements for receiving control signals generated by the game device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/302 : for viewing without the aid of special glasses, i.e. using autostereoscopic displays

Abstract

An example game device sets a reference distance for a distance from a real camera in an image-capturing direction, and displays a right captured image and a left captured image in a display area so that a reference point, which is a point that is away from the real camera by the reference distance, is displayed at a center in the display area. Thus, when an image produced by using an augmented reality technique is stereoscopically displayed, it is possible to display a virtual image in a manner easy to view.

Claims

1. A non-transitory computer-readable storage medium having stored therein an image display program for stereoscopically displaying a captured image on a stereoscopically-viewable stereoscopic display by displaying, in substantially a same display area, a right-eye image and a left-eye image by using a right captured image outputted from a right-eye real camera and a left captured image outputted from a left-eye real camera, wherein the image display program, when executed, causes a computer to at least:

set a reference distance for a distance from the real cameras in an image-capturing direction, the setting of the reference distance comprising calculating a distance to a predetermined image-capture object, and setting the reference distance based on the calculated distance;

display the right and left captured images in the display area so that a reference point common to the right and left captured images is displayed at a center of the display area, wherein the reference point is a point that is away from the real cameras by the reference distance in the image-capturing direction of the two real cameras on a straight line passing through a median point between the two real cameras and extending in the real camera image-capturing direction;

set a right-eye virtual camera and a left-eye virtual camera in a virtual space; and

stereoscopically display a virtual image by displaying in the display area a right virtual image obtained by capturing an image of the virtual space with the right-eye virtual camera and a left virtual image obtained by capturing an image of the virtual space with the left-eye virtual camera, wherein the left and right virtual images have a common virtual reference point and a projection matrix of the right virtual camera is set so that the virtual reference point in the right virtual image is displayed at the center of the display area and a projection matrix of the left virtual camera is set so that the virtual reference point in the left virtual image is displayed at the center of the display area,

wherein the virtual reference point is a point that is away from the virtual cameras by a distance corresponding to the reference distance in an image-capturing direction of the two virtual cameras on a straight line passing through a median point between the two virtual cameras and extending in the virtual camera image-capturing direction, and

wherein the image display program further causes the computer to recognize, using at least one of the right captured image and the left captured image, a position and an orientation of the predetermined image-capture object in the captured image so as to calculate position/orientation information representing a relative position and orientation relationship between the predetermined image-capture object and the real cameras, and to calculate a distance between the predetermined image-capture object and the real cameras based on the position/orientation information.

2. The storage medium according to claim 1, wherein the image display program further causes the computer to at least:

determine a right display area which is an area of the right captured image to be displayed in the display area and which is centered about a point corresponding to the reference point in the right captured image, and determine a left display area which is an area of the left captured image to be displayed in the display area and which is centered about a point corresponding to the reference point in the left captured image; and

display the right display area of the right captured image and the left display area of the left captured image in the display area based on the determined areas.

3. The storage medium according to claim 1, wherein the reference distance is fixedly set.

4. The storage medium according to claim 1,

wherein the image display program further causes the computer to set the reference distance using the calculated distance as a reference.

5. The storage medium according to claim 4, wherein the image display program further causes the computer to set, as the reference distance, a value obtained by adding or subtracting a value to/from the calculated distance.

6. The storage medium according to claim 1,

wherein the image display program further causes the computer to recognize, using both of the right captured image and the left captured image, the position and the orientation of the predetermined image-capture object in the captured image so as to calculate the position/orientation information representing the relative position and orientation

relationship between the predetermined image-capture object and the real cameras, thereby calculating a distance between the predetermined image-capture object and the real cameras based on the position/orientation information.

7. The storage medium according to claim 1, wherein the image display program further causes the computer to display the right captured image and the left captured image in the display area so that the reference point is displayed at the center of the display area by using an angle of view of the real cameras and a distance between the real cameras.

8. An image display device for stereoscopically displaying a captured image on a stereoscopically-viewable stereoscopic display by displaying, in substantially a same display area, a right-eye image and a left-eye image by using a right captured image outputted from a right-eye real camera and a left captured image outputted from a left-eye real camera, the image display device comprising:

processing circuitry including a processor, the processing circuitry being configured to control the image display device to at least;

set a reference distance for a distance from the real cameras in an image-capturing direction, the setting of the reference distance comprising calculating a distance to a predetermined image-capture object, and setting the reference distance based on the calculated distance;

display the right and left captured images in the display area so that a reference point common to the right and left captured images is displayed at a center of the display area, wherein the reference point is a point that is away from the real cameras by the reference distance in the image-capturing direction of the two real cameras on a straight line passing through a median point between the two real cameras and extending in the real camera image-capturing direction;

set a right-eye virtual camera and a left-eye virtual camera in a virtual space; and

stereoscopically display a virtual image by displaying in the display area a right virtual image obtained by capturing an image of the virtual space with the right-eye virtual camera and a left virtual image obtained by capturing an image of the virtual space with the left-eye virtual camera,

wherein the left and right virtual images have a common virtual reference point and the virtual camera setting section sets a projection matrix of the right virtual camera so that the virtual reference point in the right virtual image is displayed at the center of the display area and sets a projection matrix of the left virtual camera so that the virtual reference point in the left virtual image is displayed at the center of the display area,

wherein the virtual reference point is a point that is away from the virtual cameras by a distance corresponding to the reference distance in an image-capturing direction of the two virtual cameras on a straight line passing through a median point between the two virtual cameras and extending in the virtual camera image-capturing direction, and

wherein the processing circuitry is configured to control the image display device to recognize, using at least one of the right captured image and the left captured image, a position and an orientation of the predetermined image-capture object in the captured image so as to calculate position/orientation information representing a relative position and orientation relationship between the predetermined image-capture object and the real cameras, and to calculate a distance between the predetermined image-capture object and the real cameras based on the position/orientation information.

9. An image display system for stereoscopically

displaying a captured image on a stereoscopically-viewable stereoscopic display by displaying, in substantially a same display area, a right-eye image and a left-eye image by using a right captured image outputted from a right-eye real camera and a left captured image outputted from a left-eye real camera, the system comprising:

a reference distance setting section for setting a reference distance for a distance from the real cameras in an image-capturing direction, the reference distance setting section being configured to calculate a distance to a predetermined image-capture object, and to set the reference distance based on the calculated distance;

a stereoscopic display controller for displaying the right and left captured images in the display area so that a reference point common to the right and left captured images is displayed at a center of the display area, wherein the reference point is a point that is away from the real cameras by the reference distance in the image-capturing direction of the two real cameras on a straight line passing through a median point between the two real cameras and extending in the real camera image-capturing direction;

a virtual camera setting section for setting a right-eye virtual camera and a left-eye virtual camera in a virtual space; and

a virtual image stereoscopic display controller for stereoscopically displaying a virtual image by displaying in the display area a right virtual image obtained by capturing an image of the virtual space with the right-eye virtual camera and a left virtual image obtained by capturing an image of the virtual space with the left-eye virtual camera,

wherein the left and right virtual images have a common virtual reference point and a virtual camera setting section sets a projection matrix of the right virtual camera so that the virtual reference point in the right virtual image is displayed at the center of the display area and sets a projection matrix of the left virtual camera so that the virtual reference point in the left virtual image is displayed at the center of the display area,

wherein the virtual reference point is a point that is away from the virtual cameras by a distance corresponding to the reference distance in an image-capturing direction of the two virtual cameras on a straight line passing through a median point between the two virtual cameras and extending in the virtual camera image-capturing direction, and

wherein the system is configured to recognize, using at least one of the right captured image and the left captured image, a position and an orientation of the predetermined imagecapture object in the captured image so as to calculate position/orientation information representing a relative position and orientation relationship between the predetermined imagecapture object and the real cameras, and to calculate a distance between the predetermined image-capture object and the real cameras based on the position/orientation information.

10. An image display method to be executed by a computer for stereoscopically displaying a captured image on a stereoscopically-viewable stereoscopic display by displaying, in substantially a same display area, a right-eye image and a left-eye image by using a right captured image outputted from a right-eye real camera and a left captured image outputted from a left-eye real camera, the method comprising:

setting a reference distance for a distance from the real cameras in an image-capturing direction, the setting of the reference distance comprising calculating a distance to a predetermined image-capture object, and setting the reference distance based on the calculated distance;

displaying the right and left captured images in the display area so that a reference point common to the right and left captured images is displayed at a center of the display area, wherein the reference point is a point that is away from the real cameras by the reference distance in the image-capturing direction of the two real cameras on a straight line passing through a median point between the two real cameras and extending in the real camera image-capturing direction;

setting a right-eye virtual camera and a left-eye virtual camera in a virtual space; and

stereoscopically displaying a virtual image by displaying in the display area a right virtual image obtained by capturing an image of the virtual space with the right-eye virtual camera and a left virtual image obtained by capturing an image of the virtual space with the left eye virtual camera,

wherein the left and right images have a common virtual reference point and a projection matrix of the right virtual camera is set so that the virtual reference point in the right virtual image is displayed at the center of the display area and a projection matrix of the left virtual camera is set so that the virtual reference point in the left virtual image is displayed at the center of the display area,

wherein the virtual reference point is a point that is away from the virtual cameras by a distance corresponding to the reference distance in an image-capturing direction of the two virtual cameras on a straight line passing through a median point between the two virtual cameras and extending in the virtual camera image-capturing direction, and

wherein the method further comprises recognizing, using at least one of the right captured image and the left captured image, a position and an orientation of the predetermined image-capture object in the captured image so as to calculate position/orientation information representing a relative position and orientation relationship between the predetermined image-capture object and the real cameras, and to calculate a distance between the predetermined image-capture object and the real cameras based on the position/orientation information.