Storage medium having stored therein stereoscopic image display program, stereoscopic image display device, stereoscopic image display system, and stereoscopic image display method

• US 9,001,192 B2
• Filed: 06/07/2011
• Issued: 04/07/2015
• Est. Priority Date: 06/14/2010
• Status: Active Grant

• Current Assignee: HAL Laboratory, Inc., Nintendo Company Limited
• Original Assignee: HAL Laboratory, Inc., Nintendo Company Limited
• Inventors: Ito, Yuichiro, Nishimura, Yuki
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): WALKER, JARED T
• Application Number:
  US13/154,904
• Publication Number:
  US 20110304710A1
• Time in Patent Office: 1,400 Days
• Field of Search: None
• US Class Current: 348/51
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/5252 using two or more virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character changes room or displaying a rear-mirror view in a car-driving game
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/30 characterized by output arrangements for receiving control signals generated by the game device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/302 : for viewing without the aid of special glasses, i.e. using autostereoscopic displays

Abstract

A game device 10 displays an image on an upper LCD 22 capable of producing stereoscopic display using two images. Based on two captured images of a predetermined image-capture object captured respectively by two image-capturing devices, the game device 10 calculates a positional relationship between the predetermined image-capture object and each of the image-capturing devices. Then, the game device 10 determines a zero-parallax distance using at least one of the positional relationships so that it is a distance in accordance with a distance from the image-capturing device to the image-capture object, and determines display positions of the two captured images to be displayed on the upper LCD 22 in accordance with the zero-parallax distance. Moreover, the game device 10 produces two virtual images based on two virtual cameras which have been set using the positional relationships, and produces a synthesized image obtained by synthesizing the virtual image with the captured image to display the image on the upper LCD 22.

Claims

1. A non-transitory computer-readable storage medium having stored therein a stereoscopic image display program, which when executed by a computer of an information processing device which displays an image on a stereoscopic display device capable of producing stereoscopic display using two images, causes the information processing device to at least:

obtain two captured images of a predetermined image-capture object in a real space, captured respectively by two image-capturing devices;

calculate respective positional relationships between the predetermined image-capture object and each of the image-capturing devices based on the two captured images;

determine a zero-parallax distance using at least one of the positional relationships, wherein the zero-parallax distance corresponds to a distance, in a virtual space image-capturing direction, from at least one of two virtual cameras to a point in a virtual space that appears to be present at a position of a screen of the stereoscopic display device in stereoscopic display or a distance, in a real space image-capturing direction, from the image-capturing device associated with the used positional relationship to a point in the real space that appears to be present at the position of the screen of the stereoscopic display device in stereoscopic display;

determine respective display positions of the two captured images to be displayed on the stereoscopic display device so that a display object present at a position in accordance with the zero-parallax distance appears to be present at the position of the screen of the stereoscopic display device in stereoscopic display;

set each of the two virtual cameras for producing a respective virtual image representing an object in the virtual space by using a corresponding one of the positional relationships so that a range of the virtual space represented by the respective virtual image and a range of the real space represented by a corresponding captured image coincide with each other, and producing two virtual images based on the two virtual cameras which have been set; and

perform, for each captured image, a process of producing a synthesized image for display on the stereoscopic display device, each synthesized image obtained by synthesizing one of the virtual images with the corresponding captured image.

2. The storage medium according to claim 1, wherein the distance from the image-capturing device associated with the used positional relationship to the predetermined image-capture object or a distance to a point that is away from that distance by a predetermined distance is determined to be the zero-parallax distance.

3. The storage medium according to claim 1, wherein the zero-parallax distance is increased when the distance from the image-capturing device associated with the used positional relationship to the predetermined image-capture object increases, and is decreased when the distance from the image-capturing device associated with the used positional relationship to the predetermined image-capture object decreases.

4. The storage medium according to claim 1, wherein the stereoscopic image display program, when executed, further causes the information processing device to:

determine respective display areas to be displayed of the two captured images, thereby determining the respective display positions of the two captured images;

respectively set each of the two virtual cameras so that the range of the virtual space represented by the corresponding virtual image and the range of the real space represented by the display area of the corresponding captured image coincide with each other; and

synthesize each virtual image with the corresponding captured image in the display area.

5. The storage medium according to claim 4, wherein the stereoscopic image display program, when executed, further causes the information processing device to:

respectively calculate a position and an orientation of each of the two virtual cameras in the virtual space where the object in the virtual space is placed based on the corresponding positional relationship;

calculate respective viewing field ranges of each of the virtual cameras so that the virtual space corresponding to the real space in the corresponding display area is the range; and

produce respective virtual images, each virtual image to be synthesized with a corresponding captured image based on the position, the orientation and the viewing field of the corresponding virtual camera.

6. The storage medium according to claim 5, wherein the stereoscopic image display program, when executed, further causes the information processing device to calculate the respective viewing field range of each virtual camera so that a positional relationship of the display area with respect to an area of the corresponding captured image corresponds to a positional relationship of the respective viewing field range to be calculated with respect to the viewing field range of the virtual camera corresponding to the viewing field range of the image-capturing device.

7. The storage medium according to claim 4, wherein the stereoscopic image display program, when executed, further causes the information processing device to:

calculate a shift amount from a reference position so that, for the display area of each captured image, the zero-parallax distance in stereoscopic display is a distance in accordance with the distance from the corresponding image-capturing device to the image-capture object; and

wherein for at least one of the display areas, if the display area moved by the shift amount protrudes out of the corresponding captured image, change a size of the display area so that the display area does not protrude out of the corresponding captured image, wherein the respective virtual image is synthesized with an image in the display area of the corresponding captured image.

8. The storage medium according to claim 4, wherein the stereoscopic image display program, when executed, further causes the information processing device to determine the display areas of the two captured images so that a point at a position of the zero-parallax distance and at an equal distance from the two image-capturing devices is at center in each display area with respect to a left-right direction.

9. The storage medium according to claim 1, wherein the stereoscopic image display program, when executed, further causes the information processing device to:

successively obtain respective captured images from the two image-capturing devices;

successively calculate the respective positional relationships for the successively obtained captured images;

successively determine the zero-parallax distance each time the positional relationships are calculated;

successively determine the respective display positions each time the zero-parallax distance is determined;

successively perform a process of synthesizing the respective virtual images with corresponding captured images; and

successively display the synthesized images on the stereoscopic display device.

10. The storage medium according to claim 1, wherein the stereoscopic image display program, when executed, further causes the information processing device to:

calculate a length corresponding to a distance from the image-capturing device associated with the used positional relationship; and

determine the respective display positions of the two images based on the zero-parallax distance which is determined by the calculated length.

11. The storage medium according to claim 1, wherein the stereoscopic image display program, when executed, further causes the information processing device to determine the respective display positions of the two captured images so that a point at a position of the zero-parallax distance is displayed at the same position.

12. The storage medium according to claim 11, wherein the stereoscopic image display program, when executed, further causes the information processing device to:

determine the respective display area on each captured image so that a point at the position of the zero-parallax distance and at an equal distance from the two image-capturing devices is at center in the display area; and

perform, for each captured image, a process of producing a synthesized image obtained by synthesizing the corresponding virtual image with the captured image in the display area.

13. The storage medium according to claim 12, wherein the stereoscopic image display program, when executed, further causes the information processing device to:

determine the respective display areas by calculating a shift amount so that a proportion of the shift amount of the display area from a center of the corresponding captured image with respect to a length which is one half of a horizontal width of the corresponding captured image is a predetermined proportion; and

where a zero-parallax point is a point at a position of the zero-parallax distance and at an equal distance from two image-capturing devices, and a predetermined plane is set which vertically intersects with a viewing direction of the image-capturing devices, the predetermined proportion is a proportion of a length from an intersection between a line segment that connects between a position of one of the image-capturing devices and the zero-parallax point and the predetermined plane to a projection point with respect to a length from an intersection between a boundary line of an image-capturing range of the one image-capturing device and the predetermined plane to a projection point to which a position of the one image-capturing device is projected onto the predetermined plane.

17. A system comprising processing circuit and the non-transitory computer-readable storage medium according to claim 1.

14. A stereoscopic image display device for displaying an image on a stereoscopic display device capable of producing stereoscopic display using two images, comprising:

processing circuitry for obtaining two captured images of a predetermined image-capture object in a real space captured respectively by two image-capturing devices;

calculating respective positional relationships between the predetermined image-capture object and each of the image-capturing devices based on the two captured images;

determining a zero-parallax distance using at least one of the positional relationships, wherein the zero-parallax distance corresponds to a distance, in a virtual space image-capturing direction, from at least one of two virtual cameras to a point in a virtual space that appears to be present at a position of a screen of the stereoscopic display device in stereoscopic display or a distance, in a real space image-capturing direction, from the image-capturing device associated with the used positional relationship to a point in the real space that appears to be present at the position of the screen of the stereoscopic display device in stereoscopic display;

determining respective display positions of the two captured images to be displayed on the stereoscopic display device so that a display object present at a position in accordance with the zero-parallax distance appears to be present at the position of the screen of the stereoscopic display device in stereoscopic display;

setting each of the two virtual cameras for producing a respective virtual image representing an object in the virtual space by using a corresponding one of the positional relationships so that a range of the virtual space represented by the respective virtual image and a range of the real space represented by the corresponding captured image coincide with each other, and producing two virtual images based on the two virtual cameras which have been set; and

performing, for each captured image, a process of producing a synthesized image for display on the stereoscopic display device, each synthesized image obtained by synthesizing one of the virtual images with the corresponding captured image.

15. A stereoscopic image display system for displaying an image on a stereoscopic display device capable of producing stereoscopic display using two images, comprising:

a predetermined image-capture object in a real space;

processing circuitry for obtaining two captured images of the predetermined image-capture object captured respectively by two image capturing devices;

calculating respective positional relationships between the predetermined image-capture object and each of the image-capturing devices based on the two captured images;

determining a zero-parallax distance using at least one of the positional relationships, wherein the zero-parallax distance corresponds to a distance, in a virtual space image-capturing direction, from at least one of two virtual cameras to a point in a virtual space that appears to be present at a position of a screen of the stereoscopic display device in stereoscopic display or a distance, in a real space image-capturing direction, from the image-capturing device associated with the used positional relationship to a point in the real space that appears to be present at the position of the screen of the stereoscopic display device in stereoscopic display;

determining respective display positions of the two captured images to be displayed on the stereoscopic display device so that a display object present at a position in accordance with the zero-parallax distance appears to be present at the position of the screen of the stereoscopic display device in stereoscopic display;

setting each of the two virtual cameras for producing a respective virtual image representing an object in the virtual space by using a corresponding one of the positional relationships so that a range of the virtual space represented by the respective virtual image and a range of the real space represented by a corresponding captured image coincide with each other, and producing two virtual images based on the two virtual cameras which have been set; and

a performing, for each captured image, a process of producing a synthesized image for display on the stereoscopic display device, each synthesized image obtained by synthesizing one of the virtual images with the corresponding captured image.

16. A stereoscopic image display method for displaying an image on a stereoscopic display device capable of producing stereoscopic display using two images, comprising:

obtaining two captured images of a predetermined image-capture object in a real space captured respectively by two image-capturing devices;

calculating respective positional relationships between the predetermined image-capture object and each of the image-capturing devices based on the two captured images;

determining a zero-parallax distance using at least one of the positional relationships, wherein the zero-parallax distance corresponds to a distance, in a virtual space image-capturing direction, from at least one of two virtual cameras to a point in a virtual space that appears to be present at a position of a screen of the stereoscopic display device in stereoscopic display or a distance, in a real space image-capturing direction, from the image-capturing device associated with the used positional relationship to a point in the real space that appears to be present at the position of the screen of the stereoscopic display device in stereoscopic display;

determining respective display positions of the two captured images to be displayed on the stereoscopic display device so that a display object present at a position in accordance with the zero-parallax distance appears to be present at the position of the screen of the stereoscopic display device in stereoscopic display;

setting each of the two virtual cameras for producing a respective virtual image representing an object in the virtual space by using a corresponding one of the positional relationships so that a range of the virtual space represented by the respective virtual image and a range of the real space represented by a corresponding captured image coincide with each other, and producing two virtual images based on the two virtual cameras which have been set; and

performing, for each captured image, a process of producing a synthesized image for display on the stereoscopic display device, each synthesized image obtained by synthesizing one of the virtual images with the corresponding captured image.

18. A stereoscopic image display device comprising:

spaced-apart left and right real cameras for capturing respective left and right real images of a marker positioned in a real space;

a stereoscopic display; and

processing circuitry comprising a processor, the processing circuitry being configured to at least;

calculate respective positional relationships between the marker and each of the left and right real cameras based on the left and right real images;

determine a zero-parallax distance using at least one of the positional relationships;

determine respective display positions of the left and right real images to be displayed on the stereoscopic display so that a display object present at the zero-parallax distance appears to be present at a screen surface of the stereoscopic display;

producing left and right virtual images, each representing an object in a virtual space, using left and right virtual cameras set using the positional relationships; and

synthesizing the left virtual image and the left real image and synthesizing the right virtual image and the right real image to produce left and right images for display on the stereoscopic display.