Computer-readable storage medium, information processing apparatus, information processing system, and information processing method

• US 9,278,281 B2
• Filed: 09/23/2011
• Issued: 03/08/2016
• Est. Priority Date: 09/27/2010
• Status: Active Grant

• Current Assignee: HAL Laboratory, Inc., Nintendo Company Limited
• Original Assignee: HAL Laboratory, Inc., Nintendo Company Limited
• Inventors: Ito, Yuichiro, Noge, Tetsuya
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): BEASLEY, DEIRDRE L
• Application Number:
  US13/243,622
• Publication Number:
  US 20120075430A1
• Time in Patent Office: 1,628 Days
• Field of Search: 348/47
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/5258 by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/818 Fishing
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6684 by dynamically adapting its position to keep a game object in its viewing frustrum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 2300/6692 using special effects, generally involving post-processing, e.g. blooming
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8029 : Fighting without shooting
• CPC Class Codes: A63F 2300/8035 : Virtual fishing
• CPC Class Codes: G06T 2207/10012 : Stereo images
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: H04N 13/156 : Mixing image signals
• CPC Class Codes: H04N 13/275 : from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

A marker present in a real space is detected from each of a real image for a left eye and a real image for a right eye. The position and the orientation of each of imaging sections relative to the marker are calculated. A left virtual camera and a right virtual camera are set in accordance with the positions and the orientations of the imaging sections. A virtual model having a planar shape is placed in a marker coordinate system, and areas in the real image for a left eye and the real image for a right eye that correspond to the virtual model are set as a left texture area and a right texture area, respectively. Each of images included in a left texture area and a right texture area is applied, as a texture, to a deformation object into which the virtual model is deformed.

Claims

1. A non-transitory computer-readable storage medium having stored therein instructions executed by a computer of an information processing apparatus connected to a left real camera and a right real camera for shooting a real space, and a display device capable of displaying a stereoscopic visible image, the stored instructions comprising instructions that are configured to cause the computer to:

obtain a real image for a left eye shot by the left real camera;

obtain a real image for a right eye shot by the right real camera;

set a left virtual camera in a virtual space that is three-dimensional, the left virtual camera set at a position and an orientation that corresponds to the left real camera in the real space;

set a right virtual camera in the virtual space in accordance with a position and an orientation of the right real camera in the real space;

locate a virtual model in the virtual space;

set, as a left texture area, an area in the real image for a left eye that is to be applied to the virtual model that is located in the virtual space;

set, as a right texture area, an area in the real image for a right eye that is to be applied to the virtual model that is located in the virtual space;

apply at least some portion of the set left texture area to the virtual model;

generate, based on at least the set left virtual camera and the portion of the set left texture area, a left virtual camera image of the virtual space that includes a view of the virtual model located in the virtual space;

apply at least some portion of the set right texture area to the virtual model;

generate, based on at least the right virtual camera and the portion of the set right texture area a right virtual camera image of the virtual space that includes a view of the virtual model located in the virtual space; and

display the generated left virtual camera image and the generated right virtual camera image on the display in a stereoscopic manner.

2. The non-transitory computer-readable storage medium according to claim 1, wherein

a left texture area is set corresponding to the position and the contour of the virtual model viewed by the left virtual camera, and

a right texture area is set corresponding to the position and the contour of the virtual model viewed by the right virtual camera.

3. The non-transitory computer-readable storage medium according to claim 1, wherein the stored instructions further comprise instructions that are configured to cause the computer to:

generate a superimposed image for a left eye by superimposing the left virtual camera image onto the real image for a left eye; and

generate a superimposed image for a right eye by superimposing the right virtual camera image onto the real image for a right eye,

wherein the superimposed image for a left eye and the superimposed image for a right eye are displayed to the display device.

4. The non-transitory computer-readable storage medium according to claim 3, wherein the stored instructions further comprise instructions that are configured to cause the computer to:

load a base virtual model; and

deform the base virtual model by adjusting geometry thereof to generate the virtual model,

wherein the virtual model that is located in the virtual space is the generated virtual model.

5. The non-transitory computer-readable storage medium according to claim 4, wherein

the virtual model has a planar shape or a curved-surface shape, and is set in the virtual space so as to follow the shape of a plane or a curved surface in the real space.

6. The non-transitory computer-readable storage medium according to claim 4, wherein no end portions of the base virtual model are deformed to generate the virtual model.

7. The non-transitory computer-readable storage medium according to claim 4, wherein the stored instructions further comprise instructions that are configured to cause the computer to locate a second object, different from the generated virtual model, in the virtual space such that the second object has a predetermined position relationship with the generated virtual model, wherein

the left virtual camera image shot by the left virtual camera includes the second object, and

the right virtual camera image shot by the right virtual camera includes the second object.

8. The non-transitory computer-readable storage medium according to claim 7,

wherein the second object is placed so as to be away from the generated virtual model in a shooting direction of each of the left virtual camera and the right virtual camera,

wherein the stored instructions further comprise instructions configured to cause the computer to use a predetermined mask object for masking the second object at an end portion of the generated virtual model.

9. The non-transitory computer-readable storage medium according to claim 3, wherein:

the obtained real image for the left eye and the obtained real image for the right eye includes an image of a marker, having a predetermined planar shape, that is located the real space,

the left virtual camera is set in accordance with the position and the orientation of the left real camera in the real space relative to the position and the orientation of the marker,

the right virtual camera is set in accordance with the position and the orientation of the right real camera in the real space relative to the position and the orientation of the marker,

the virtual model has a planar shape and, is placed on a plane in the virtual space that corresponds to a virtual plan upon which the marker is placed in the real space,

wherein the stored instructions further comprise instructions that are configured to cause the computer to;

deform a base virtual model by adjusting geometry thereof to generate the virtual model,

wherein the virtual model that is located in the virtual space is the generated virtual model.

10. The non-transitory computer-readable storage medium according to claim 1, wherein the stored instructions further comprise instructions that are configured to cause the computer to:

detect a specific subject included in the real image for a left eye and the real image for a right eye

obtain relative positions and relative orientations of the left real camera and the specific subject present in the real space with respect to each other based on a result of the detection of the specific subject in the real image for the left eye;

obtain relative positions and relative orientations of the right real camera and the specific subject present in the real space with respect to each other based on a result of the detection of the specific subject in the real image for the right eye;

set the left and right virtual cameras based on the respectively obtained relative positions and relative orientations of the left and right real cameras.

11. The non-transitory computer-readable storage medium according to claim 10, wherein the virtual model is placed based on a position, in the virtual space, that corresponds to the position of the specific subject in the real space.

12. An information processing apparatus connected to a left real camera and a right real camera for shooting a real space, and to display device capable of displaying a stereoscopic visible image, the information processing apparatus comprising:

a processing system that includes at least one processor, the processing system configured to;

obtain a real image for a left eye shot by the left real camera;

obtain a real image for a right eye shot by the right real camera;

set a left virtual camera in a virtual space that is three-dimensional, the left virtual camera set at a position and an orientation that corresponds to the left real camera in the real space;

set a right virtual camera in the virtual space in accordance with a position and an orientation the right real camera in the real space;

locate the virtual model in the virtual space;

determine a left texture image from an area of the real image for a left eye that is to be applied to the virtual model that is located in the virtual space;

determine a right texture image from the real image for a right eye that is to be applied to the virtual model that is located in the virtual space;

texture map at least some portion of the determined left texture image to the virtual model;

generate a left virtual camera image of the texture mapped virtual model as viewed in accordance with the set left virtual camera;

texture map the determined right texture image to the virtual model;

generate a right virtual camera image of the texture mapped virtual model as viewed in accordance with the set right virtual camera; and

display, in a stereoscopic manner, the generated left virtual camera image and the generated right virtual camera image on the display device.

13. An information processing system comprising:

a left real camera and a right real camera for shooting a real space;

a display device capable of displaying a stereoscopic visible image; and

a processing system that includes at least one computer processor, the processing system configured to;

obtain a real image for a left eye shot by the left real camera;

obtain a real image for a right eye shot by the right real camera;

set a left virtual camera in a virtual space that is three-dimensional, the left virtual camera set at a position and an orientation that corresponds to the left real camera in the real space;

set a right virtual camera in the virtual space in accordance with a position and an orientation the right real camera in the real space;

locate a virtual model in the virtual space;

set a left texture area in the real image for a left eye that is to be applied to the virtual model that is located in the virtual space;

set a right texture area in the real image for a right eye that that is to be applied to the virtual model that is located in the virtual space;

apply an image from the left texture area to the virtual model;

generate, based on the set left virtual camera, a left virtual camera image of the virtual space that includes a view of the virtual model located in the virtual space;

apply an image from the right texture area to the virtual model;

generate, based on the set right virtual camera, a right virtual camera image of the virtual space that includes a view of the virtual model located in the virtual space; and

output the generated left virtual camera image and the generated right virtual camera image to the display device such that the left virtual camera image is viewed by the left eye of a user and the right virtual camera image is viewed by the right eye of the user.

14. An information processing method comprising:

obtaining a real image for a left eye shot by a left real camera for shooting a real space;

obtaining a real image for a right eye shot by a right real camera for shooting the real space;

setting a left virtual camera in a virtual space that is three-dimensional, the left virtual camera set at a position and an orientation that corresponds to the left real camera in the real space,

setting a right virtual camera in a virtual space in accordance with a position and an orientation of the right real camera in the real space;

locating a virtual model in the virtual space;

setting, as a left texture area, an area in the real image for a left eye that is to be applied to the virtual model located in the virtual space;

setting, as a right texture area, an area in the real image for a right eye that is to be applied to the virtual model located in the virtual space;

applying at least a portion of the left texture area to the virtual model;

generating, based on at least the set left virtual camera and the applied portion of the left texture area, a left virtual camera image of the virtual space that includes a view of the virtual model located in the virtual space;

applying at least a portion of the right texture area to the virtual model;

generating, based on at least the set right virtual camera and the applied portion of the right texture area, a right virtual camera image of the virtual space that includes a view of the virtual model located in the virtual space shot; and

displaying the left virtual camera image and the right virtual camera image in a stereoscopic manner on a display device capable of displaying a stereoscopic visible image.