Computer-readable storage medium, image display apparatus, image display system, and image display method

• US 8,780,183 B2
• Filed: 06/13/2011
• Issued: 07/15/2014
• Est. Priority Date: 06/11/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Ito, Yuichiro
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): HAQUE, MD NAZMUL
• Application Number:
  US13/158,736
• Publication Number:
  US 20110304701A1
• Time in Patent Office: 1,128 Days
• Field of Search: None
• US Class Current: 348/47
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

First, the interval between a left virtual camera and a right virtual camera is determined based on the relative position and the relative orientation of an outer imaging section (left) with respect to a marker, which are calculated from a result of marker recognition in the left real world image, and on the relative position and the relative orientation of an outer imaging section (right) with respect to the marker, which are calculated from a result of marker recognition in the right real world image. After the interval between the left virtual camera and the right virtual camera is determined, the positions and the orientations of the left virtual camera and the right virtual camera are determined based on the determined interval such that the relationship between the position and the orientation of the left virtual camera, and the position and the orientation of the right virtual camera is ideal.

Claims

1. A non-transitory computer-readable storage medium having stored therein an image display program which stereoscopically displays a real world image, onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, the image display program comprising instructions that cause a computer to:

obtain a first real world image shot by a left eye real camera;

obtain a second real world image shot by a right eye real camera;

calculate a relative position relationship between the left eye real camera and the right eye real camera, the relative position relationship calculated based on a manner in which a predetermined shooting target is visible on the first real world image and a manner in which the predetermined shooting target is visible on the second real world image;

determine the position of a left virtual camera for generating an image for a left eye, and the position of a right virtual camera for generating an image for a right eye, in a predetermined virtual space, so as to be separated from each other in accordance with the calculated relative position relationship;

generate a first virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera;

generate a second virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera;

superimpose the first virtual space image onto the first real world image to generate a first superimposed image;

superimpose the second virtual space image onto the second real world image to generate a second superimposed image; and

output the first and second superimposed images for a stereoscopic view to the stereoscopic display apparatus.

2. The non-transitory computer-readable storage medium according to claim 1, the image display program further comprises instructions that cause the computer to:

calculate at least relative positions of the left eye real camera and the predetermined shooting target with respect to each other, based on the position and the orientation of the predetermined shooting target in the first real world image; and

calculate at least relative positions of the right eye real camera and the predetermined shooting target with respect to each other, based on the position and the orientation of the predetermined shooting target in the second real world image,

wherein the relative position relationship is calculated between the left eye real camera and the right eye real camera, based on the relative position of the left eye real camera with respect to the predetermined shooting target and on the relative position of the right eye real camera with respect to the predetermined shooting target.

3. The non-transitory computer-readable storage medium according to claim 2, wherein the image display program further comprises instructions that cause the computer to:

calculate relative orientations of the corresponding one of the left eye real camera and the right eye real camera based on the position and the orientation of the predetermined shooting target in a corresponding one of the first real world image and the second real world image, and

set orientations of the left virtual camera and the right virtual camera, based on the calculated relative orientations of the corresponding one of the left eye real camera and the right eye real camera.

4. The non-transitory computer-readable storage medium according to claim 1, wherein the relative position relationship is an interval between the left eye real camera and the right eye real camera.

5. The non-transitory computer-readable storage medium according to claim 1, wherein the relative position relationship between the left eye real camera and the right eye real camera is further based on a difference between a relative position of the predetermined shooting target with respect to the left eye real camera and a relative position of the predetermined shooting target with respect to the right eye real camera.

6. The non-transitory computer-readable storage medium according to claim 1, the image display program further comprises instructions that cause the computer to:

calculate, based on the first real world image, a first transform matrix that transforms coordinates represented in a coordinate system based on the predetermined shooting target, into coordinates represented in a coordinate system based on the left eye real camera;

calculate, based on the second real world image, a second transform matrix that transforms coordinates represented in the coordinate system based on the predetermined shooting target, into coordinates represented in a coordinate system based on the right eye real camera; and

calculate a relative position vector indicating the relative position relationship between the left eye real camera and the right eye real camera by transforming an origin vector by one of the first transform and second transform matrices to obtain a resultant vector, and transform the resultant vector by an inverse matrix of the other one of the first transform matrix and the second transform matrix,

wherein the left virtual camera and the right virtual camera are set in accordance with a magnitude of the relative position vector.

7. The non-transitory computer-readable storage medium according to claim 1, wherein an orientation of the positioned left virtual camera is parallel to an orientation of the positioned right virtual camera.

8. The non-transitory computer-readable storage medium according to claim 1, the image display program further comprises instructions that cause the computer to execute, in the virtual space, predetermined processing in accordance with a scale in the real world, based on the calculated relative position relationship, and on a designed relative position relationship between the left eye real camera and the right eye real camera.

9. The non-transitory computer-readable storage medium according to claim 1, the image display program further comprises instructions that cause the computer to:

perform a determination of whether or not the depth distance from the left eye real camera or the right eye real camera to the predetermined shooting target has varied beyond a predetermined range in comparison to another depth distance previously determined when the calculated relative position relationship between the left eye real camera and the right eye real camera was calculated,

wherein the calculated relative position relationship is maintained without again calculated relative position relationship between the left eye real camera and the right eye real camera based on the depth distance not varying beyond the predetermined shooting range.

10. The non-transitory computer-readable storage medium according to claim 1, the image display program further comprises instructions that cause the computer to:

perform a determination of whether or not the depth distance from the left eye real camera or the right eye real camera to the predetermined shooting target has varied beyond a predetermined range in comparison with another previously calculated depth distance that is associated with the calculated relative position relationship between the left eye real camera and the right eye real camera; and

calculate a new relative position relationship between the left eye real camera and the right eye real camera in accordance with the performed determination varying beyond the predetermined range in comparison with the another previously calculated depth distance.

11. The non-transitory computer-readable storage medium according to claim 9, wherein the image display program further comprises instructions that cause the computer to:

calculate the depth distance from the left eye real camera or the right eye real camera to the predetermined shooting target based on obtained real world images that include the predetermined shooting target,

wherein the performed determination includes comparing a reference depth distance calculated at a time when the relative position relationship between the left eye real camera and the right eye real camera was calculated, with a latest depth distance calculated after calculation of the reference depth distance.

12. The non-transitory computer-readable storage medium according to claim 10, wherein the image display program further comprises instructions that cause the computer to:

calculate the depth distance from the left eye real camera or the right eye real camera to the predetermined shooting target based on obtained real world images that include the predetermined shooting target,

wherein the performed determination includes comparing a calculated reference depth distance at a time when the relative position relationship between the left eye real camera and the right eye real camera was calculated, with a latest depth distance calculated after the reference depth distance.

13. The non-transitory computer-readable storage medium according to claim 11, wherein processing of calculating the depth distance from the left eye real camera or the right eye real camera to the predetermined shooting target is executed every time a new real world image is obtained, and

the reference depth distance is calculated based on results of a plurality of the calculations of the depth distance obtained by the processing being executed a plurality of times.

14. The non-transitory computer-readable storage medium according to claim 13, wherein the reference depth distance is determined based on the results of the plurality of calculations of the depth distance only when all the results of the plurality of calculations of the depth distance are within a predetermined range.

15. An image display apparatus configured to stereoscopically display images to a stereoscopic display, the image display apparatus comprising:

a processing system including at least one processor that is coupled to a memory, the processing system configured to;

obtain a first real world image of a shooting target by using a left real camera;

obtain a second real world image of the shooting target by using a right real camera, the second real world image different from the first real world image;

calculate a relative position relationship between the left real camera and the right real camera based on a manner in which the shooting target is included in the first and second real world images;

set a first virtual camera in a virtual space at a first position;

set a second virtual camera in the virtual space at a second position, where at least one of the first and second positions is set using the calculated relative position relationship;

generate a first virtual image of the virtual space as viewed from the first virtual camera;

generate a second virtual image of the virtual space as viewed from the second virtual camera;

superimpose the first virtual image onto the first real world image;

superimpose the second virtual image onto the second real world image; and

output the superimposed images for stereoscopic view to the stereoscopic display.

18. The image display apparatus of claim 15, wherein the computing system is further configured to calculate a first transform matrix that describes a position, relative to the shooting target, from which the first real world image was obtained,

wherein the first virtual camera is set in the virtual space at the first position in accordance with calculated first transform matrix,

wherein second position is determined and set for the second virtual camera by translating a position indicated by the first transform matrix by the calculated relative position relationship.

16. An image display system comprising:

a stereoscopic display that is configured to display stereoscopic images;

a shooting target located in real space;

a left camera configured to obtain a first image of the shooting target;

a right camera configured to obtain a second image of the shooting target; and

a processing system that includes at least one processor coupled to a memory, the processing system configured to;

calculate a relative position relationship between the left camera and the right camera based on a manner in which the shooting target is visible on the first image and a manner in which the shooting target is visible on the second image;

determine a position of a left virtual camera and a position of a right virtual camera in a virtual space so as to be separated from each other in accordance with the calculated relative position relationship;

generate a first virtual space image of the virtual space as viewed from the positioned left virtual camera;

generate a second virtual space image of the virtual space as viewed from the positioned right virtual camera;

superimpose the first virtual space image onto the first image, and superimpose the second virtual space image onto the second image; and

output the superimposed images for a stereoscopic view on the stereoscopic display apparatus.

17. An image display method for stereoscopically displaying, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, a real world image onto which an image of a 3-dimensional virtual object is superimposed the image display method comprising:

obtaining a first real world image shot by a left real camera;

obtaining a second real world image shot by a right real camera;

calculating, using at least one processor, a relative position relationship between the left real camera and the right real camera based on visibility of a predetermined shooting in the first real world image and the second real world image;

determining a position of a left virtual camera and a position of a right virtual camera in a virtual space, so as to be separated from each other in accordance with the calculated relative position relationship;

generating a first virtual space image of the virtual space as viewed from the determined position of the left virtual camera;

generating a second virtual space image of the virtual space as viewed from the determine position of the right virtual camera; and

superimposing the first virtual space image onto the first real world image, and superimposing the second virtual space image onto the second real world image, to output images for a stereoscopic view to the stereoscopic display apparatus.