Computer-Readable Storage Medium, Image Display Apparatus, Image Display System, and Image Display Method

• US 20110304703A1
• Filed: 06/13/2011
• Published: 12/15/2011
• Est. Priority Date: 06/11/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: ITO, Yuichiro
• Granted Patent: US 10,015,473 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

First, the interval between a left virtual camera and a right virtual camera is determined based on the relative position and the relative orientation of an outer imaging section (left) with respect to a marker, which are calculated from a result of marker recognition in the left real world image, and on the relative position and the relative orientation of an outer imaging section (right) with respect to the marker, which are calculated from a result of marker recognition in the right real world image. After the interval between the left virtual camera and the right virtual camera is determined, the positions and the orientations of the left virtual camera and the right virtual camera are determined based on the determined interval such that the relationship between the position and the orientation of the left virtual camera, and the position and the orientation of the right virtual camera is ideal.

Claims

1. A computer-readable storage medium having stored therein an image display program which stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, by using outputs from a real camera imaging section for a right eye and a real camera imaging section for a left eye, the image display program causing a computer to function as:

position orientation calculation means for recognizing a predetermined shooting target in each of pieces of real world image data respectively outputted from the real camera imaging section for a right eye and the real camera imaging section for a left eye, and calculating position orientation information indicating relative positions and relative orientations of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and position orientation information indicating relative positions and relative orientations of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other;

position orientation integrating means for integrating the two pieces of position orientation information calculated by the position orientation calculation means, thereby calculating one piece of position orientation information;

virtual camera setting means for determining the position and the orientation of a right virtual camera for generating a virtual image for a right eye, and the position and the orientation of a left virtual camera for generating a virtual image for a left eye, in a predetermined virtual space, based on the one piece of position orientation information integrated by the first position orientation integrating means;

right virtual space image generation means for generating a right virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera;

left virtual space image generation means for generating a left virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera; and

display control means for superimposing the right virtual space image onto the piece of real world image data outputted from the real camera imaging section for a right eye, and superimposing the left virtual space image onto the piece of real world image data outputted from the real camera imaging section for a left eye, to output images for a stereoscopic view to the stereoscopic display apparatus.

2. The computer-readable storage medium according to claim 1, wherein the virtual camera setting means determines the positions and the orientations of the right virtual camera and the left virtual camera such that the relationship between relative orientations of the right virtual camera and the left virtual camera with respect to each other is the same as the relationship between designed relative orientations of the real camera imaging section for a right eye and the real camera imaging section for a left eye with respect to each other.

3. The computer-readable storage medium according to claim 1, wherein

the position orientation integrating means integrates the information about the relative orientations of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and the information about the relative orientations of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other, which are calculated by the position orientation calculation means, thereby calculating one orientation, and

the virtual camera setting means uses the one orientation integrated by the position orientation integrating means, as each of the orientations of the right virtual camera and the left virtual camera.

4. The computer-readable storage medium according to claim 1, wherein

the position orientation integrating means integrates the information about the relative positions of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and the information about the relative positions of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other, which are calculated by the position orientation calculation means, thereby calculating one position, and

the virtual camera setting means sets the positions of the right virtual camera and the left virtual camera so as to be symmetric about the one position integrated by the position orientation integrating means.

5. The computer-readable storage medium according to claim 1, wherein

the position orientation integrating means averages the information about the relative orientations of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and the information about the relative orientations of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other, which are calculated by the position orientation calculation means, thereby calculating an averaged orientation, and

the virtual camera setting means uses the averaged orientation calculated by the position orientation integrating means, as each of the orientations of the right virtual camera and the left virtual camera.

6. The computer-readable storage medium according to claim 1, wherein

the position orientation integrating means calculates the midpoint between the relative position of the real camera imaging section for a right eye and the relative position of the real camera imaging section for a left eye, by using the information about the relative positions of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and the information about the relative positions of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other, which are calculated by the position orientation calculation means, and

the virtual camera setting means sets the positions of the right virtual camera and the left virtual camera so as to be symmetric about the midpoint calculated by the position orientation integrating means.

7. The computer-readable storage medium according to claim 4, the image display program further causing the computer to function as virtual cameras distance determination means for determining the distance between the right virtual camera and the left virtual camera,

wherein the virtual camera setting means sets the positions of the right virtual camera and the left virtual camera so as to be symmetric about the one position integrated by the position orientation integrating means, and separated by the distance determined by the virtual cameras distance determination means.

8. The computer-readable storage medium according to claim 4, wherein the virtual cameras distance determination means determines the distance between the right virtual camera and the left virtual camera, based on a parallax between the two pieces of real world image data respectively outputted from the real camera imaging section for a right eye and the real camera imaging section for a left eye.

9. The computer-readable storage medium according to claim 4, wherein the virtual cameras distance determination means calculates the distance between the right virtual camera and the left virtual camera, by using the information about the relative positions and relative orientations of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and the information about the relative positions and relative orientations of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other, which are calculated by the position orientation calculation means.

10. An image display apparatus which stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, by using outputs from a real camera imaging section for a right eye and a real camera imaging section for a left eye, the image display apparatus comprising:

position orientation calculation means for recognizing a predetermined shooting target in each of pieces of real world image data respectively outputted from the real camera imaging section for a right eye and the real camera imaging section for a left eye, and calculating position orientation information indicating relative positions and relative orientations of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and position orientation information indicating relative positions and relative orientations of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other;

position orientation integrating means for integrating the two pieces of position orientation information calculated by the position orientation calculation means, thereby calculating one piece of position orientation information;

virtual camera setting means for determining the position and the orientation of a right virtual camera for generating a virtual image for a right eye, and the position and the orientation of a left virtual camera for generating a virtual image for a left eye, in a predetermined virtual space, based on the one piece of position orientation information integrated by the first position orientation integrating means;

right virtual space image generation means for generating a right virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera;

left virtual space image generation means for generating a left virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera; and

display control means for superimposing the right virtual space image onto the piece of real world image data outputted from the real camera imaging section for a right eye, and superimposing the left virtual space image onto the piece of real world image data outputted from the real camera imaging section for a left eye, to output images for a stereoscopic view to the stereoscopic display apparatus.

11. An image display system which stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, by using outputs from a real camera imaging section for a right eye and a real camera imaging section for a left eye, the image display system comprising:

a predetermined shooting target;

position orientation calculation means for recognizing the predetermined shooting target in each of pieces of real world image data respectively outputted from the real camera imaging section for a right eye and the real camera imaging section for a left eye, and calculating position orientation information indicating relative positions and relative orientations of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and position orientation information indicating relative positions and relative orientations of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other;

position orientation integrating means for integrating the two pieces of position orientation information calculated by the position orientation calculation means, thereby calculating one piece of position orientation information;

virtual camera setting means for determining the position and the orientation of a right virtual camera for generating a virtual image for a right eye, and the position and the orientation of a left virtual camera for generating a virtual image for a left eye, in a predetermined virtual space, based on the one piece of position orientation information integrated by the first position orientation integrating means;

right virtual space image generation means for generating a right virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera;

left virtual space image generation means for generating a left virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera; and

display control means for superimposing the right virtual space image onto the piece of real world image data outputted from the real camera imaging section for a right eye, and superimposing the left virtual space image onto the piece of real world image data outputted from the real camera imaging section for a left eye, to output images for a stereoscopic view to the stereoscopic display apparatus.

12. An image display method for stereoscopically displaying a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, by using outputs from a real camera imaging section for a right eye and a real camera imaging section for a left eye, the image display method comprising:

a position orientation calculation step of recognizing a predetermined shooting target in each of pieces of real world image data respectively outputted from the real camera imaging section for a right eye and the real camera imaging section for a left eye, and calculating position orientation information indicating relative positions and relative orientations of the real camera imaging section for a right eye and the predetermined shooting target with respect to each other, and position orientation information indicating relative positions and relative orientations of the real camera imaging section for a left eye and the predetermined shooting target with respect to each other;

a position orientation integrating step of integrating the two pieces of position orientation information calculated in the position orientation calculation step, thereby calculating one piece of position orientation information;

a virtual camera setting step of determining the position and the orientation of a right virtual camera for generating a virtual image for a right eye, and the position and the orientation of a left virtual camera for generating a virtual image for a left eye, in a predetermined virtual space, based on the one piece of position orientation information integrated in the first position orientation integrating step;

a right virtual space image generation step of generating a right virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera;

a left virtual space image generation step of generating a left virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera; and

a display control step of superimposing the right virtual space image onto the piece of real world image data outputted from the real camera imaging section for a right eye, and superimposing the left virtual space image onto the piece of real world image data outputted from the real camera imaging section for a left eye, to output images for a stereoscopic view to the stereoscopic display apparatus.