STORAGE MEDIUM HAVING STORED THEREIN IMAGE PROCESSING PROGRAM, IMAGE PROCESSING APPARATUS, IMAGE PROCESSING SYSTEM, AND IMAGE PROCESSING METHOD

• US 20120075285A1
• Filed: 11/15/2010
• Published: 03/29/2012
• Est. Priority Date: 09/28/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: OYAGI, Yasuyuki, Takadera, Tatsuya
• Granted Patent: US 8,854,356 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/5252 using two or more virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character changes room or displaying a rear-mirror view in a car-driving game
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8058 Virtual breeding, e.g. tamagotchi
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 : Stereo images
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 7/73 : using feature-based methods

Abstract

An image taken by a real camera is repeatedly obtained, and position and orientation information determined in accordance with a position and an orientation of a real camera in a real space is repeatedly calculated. A virtual object or a letter to be additionally displayed on the taken image is set as an additional display object, and based on a result of recognition of a sound inputted into a sound input device, at least one selected from the group consisting of a display position, an orientation, and a display form of the additional display object is set. A combined image repeatedly generated by superimposing on the taken image the set additional display object with reference to a position in the taken image in accordance with the position and orientation information is displayed on a display device.

Claims

1. A computer-readable storage medium having stored therein an image processing program executed by a computer of an image processing apparatus for displaying an image on a display device, the image processing program causing the computer to function as:

taken image obtaining means for repeatedly obtaining an image taken by a real camera;

position and orientation calculation means for repeatedly calculating position and orientation information determined in accordance with a position and an orientation of the real camera in a real space;

sound data obtaining means for obtaining sound data representing a sound signal from a sound input device;

sound recognition means for recognizing a sound inputted into the sound input device;

setting means for setting one of a virtual object and a letter to be additionally displayed on the taken image, as an additional display object, and for setting, based on a sound recognition result by the sound recognition means, at least one selected from the group consisting of a display position, an orientation, and a display form of the additional display object;

image generation means for repeatedly generating a combined image by superimposing, with reference to a position in the taken image in accordance with the position and orientation information, the additional display object set by the setting means; and

display control means for causing the display device to repeatedly display the combined image.

2. The computer-readable storage medium having stored therein the image processing program according to claim 1, wherein

the position and orientation calculation means detects one of a certain imaging target and a certain characteristic point included in the taken image and calculates, based on a result of the detection, information representing a relative position and a relative orientation between the real camera and the one of the imaging target and the characteristic point, as the position and orientation information.

3. The computer-readable storage medium having stored therein the image processing program according to claim 1, wherein

the position and orientation calculation means calculates the position and orientation information by using at least one of a geographic position of the real camera and an azimuth of an imaging direction of the real camera in the real space.

4. The computer-readable storage medium having stored therein the image processing program according to claim 1, wherein

the setting means sets the virtual object as the additional display object, and

the image generation means includes:

virtual camera setting means for setting in a virtual world a position and an orientation of a virtual camera based on the position and orientation information;

virtual object arrangement means for arranging the virtual object set by the setting means in the virtual world; and

virtual world image generation means for generating as a virtual world image an image in the virtual world seen from the virtual camera, and

the image generation means generates as the combined image an image obtained by superimposing the virtual world image on the taken image.

5. The computer-readable storage medium having stored therein the image processing program according to claim 4, wherein

the setting means sets, when the sound inputted into the sound input device is recognized as a first sound and the position and orientation information satisfies a first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a first action, and sets, when the sound inputted into the sound input device is recognized as the first sound and the position and orientation information does not satisfy the first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a second action.

6. The computer-readable storage medium having stored therein the image processing program according to claim 2, wherein

the position and orientation calculation means calculates, as the position and orientation information, information representing a relative position and a relative orientation between the imaging target and the real camera,

the setting means sets the virtual object as the additional display object, and

the image generation means includes:

virtual camera setting means for setting in a virtual world a position and a direction corresponding to the imaging target, based on the position and orientation information, and for setting in the virtual world a position and an orientation of the virtual camera, with reference to the position and the direction corresponding to the imaging target, based on the position and orientation information;

virtual object arrangement means for arranging the virtual object set by the setting means in the virtual world, with reference to the position corresponding to the imaging target; and

virtual world image generation means for generating as a virtual world image an image of the virtual world seen from the virtual camera,

the setting means sets, when the sound inputted into the sound input device is recognized as a first sound, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object, based on a first action to be performed with reference to the direction corresponding to the imaging target in the virtual world, and sets, when the sound inputted into the sound input device is recognized as a second sound, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object, based on a second action to be performed with reference to the direction to the virtual camera in the virtual world, and

the image generation means generates as the combined image an image obtained by superimposing the virtual world image on the taken image.

7. The computer-readable storage medium having stored therein the image processing program according to claim 1, wherein

the setting means sets, at one of a case where the sound is not inputted into the sound input device and a case where the sound recognized by the sound recognition means is not a sound that causes the additional display object to be set, at least one selected from the group consisting of the display position, the orientation, and the display form of the additional display object, to a predetermined content.

8. The computer-readable storage medium having stored therein the image processing program according to claim 1, the image processing program further causing the computer to function as photograph saving means for saving in storage means a latest combined image currently generated by the image generation means, in accordance with a photographing instruction given by a user.

9. The computer-readable storage medium having stored therein the image processing program according to claim 1, wherein

the taken image obtaining means repeatedly obtains a first image taken by a first real camera and a second image taken by a second real camera which is provided at a position spaced from the first real camera by a predetermined distance,

the position and orientation calculation means repeatedly calculates first position and orientation information determined in accordance with a position and an orientation of the first real camera in the real space and second position and orientation information determined in accordance with a position and an orientation of the second real camera in the real space,

the image generation means repeatedly generates a first combined image by superimposing, with reference to a position in the first taken image in accordance with the first position and orientation information, the additional display object set by the setting means, and repeatedly generates a second combined image by superimposing, with reference to a position in the second taken image in accordance with the second position and orientation information, the additional display object set by the setting means, and

the display control means outputs the first combined image and the second combined image to a display device capable of displaying a stereoscopically visible image, and causes the display device to repeatedly stereoscopically display the combined image including the additional display object.

10. The computer-readable storage medium having stored therein the image processing program according to claim 1, wherein

the sound recognition means determines whether the sound inputted into the sound input device is a first sound at least by verifying the sound inputted into the sound input device against a sound registered in advance as a registered sound, and determines whether the sound inputted into the sound input device is a second sound based only on the level of the sound waveform of the sound inputted into the sound input device.

11. The computer-readable storage medium having stored therein the image processing program according to claim 1, wherein

the sound recognition means determines whether the sound inputted into the sound input device is a first sound, based on the likelihood of a feature parameter time series registered in advance as a registered sound against a feature parameter time series obtained from the sound input pattern inputted into the sound input device, and determines whether the sound inputted into the sound input device is a third sound, based only on at least one of the level of the sound waveform and the spectrum information of the sound inputted into the sound input device.

12. The computer-readable storage medium having stored therein the image processing program according to claim 10, the image processing program further causing the computer to function as:

sound registration means for allowing a user to input a sound corresponding to an instruction to be given to the additional display object, and for registering, in storage means, sound data corresponding to the sound as the registered sound corresponding to the instruction, wherein

the sound recognition means determines whether the sound inputted into the sound input device is the first sound by using the sound data registered as the registered sound by the sound registration means, and

when the sound recognition means has determined that the sound inputted into the sound input device is the first sound, the setting means sets at least one selected from the group consisting of the display position, the orientation, and the display form of the additional display object, based on the instruction corresponding to the first sound and registered in advance by the sound registration means.

13. The computer-readable storage medium having stored therein the image processing program according to claim 3, wherein

the setting means estimates a subject whose image has been taken in the taken image, based on at least one of the geographic position of the real camera and the azimuth of the imaging direction of the real camera represented by the position and orientation information, and sets a letter which represents information about the subject as the additional display object,

the image generation means repeatedly generates the combined image by superimposing, with reference to a position in the taken image in accordance with the position and orientation information, the letter corresponding to the subject set by the setting means,

when a letter which coincides with a word recognized through sound recognition by the sound recognition means is set as the additional display object, the setting means changes a display form into that indicating that the letter has been selected, and sets the letter, and

the image generation means generates the combined image by superimposing on the taken image the letter which coincides with the word in the display form updated by the setting means.

14. The computer-readable storage medium having stored therein the image processing program according to claim 1, wherein

the taken image obtaining means obtains the taken image from the real camera incorporated in a housing which incorporates the image processing apparatus,

the sound data obtaining means obtains the sound data from the sound input device incorporated in the housing, and

the display control means causes the display device incorporated in the housing to display the combined image.

15. An image processing apparatus for displaying an image on a display device, comprising:

taken image obtaining means for repeatedly obtaining an image taken by a real camera;

position and orientation calculation means for repeatedly calculating position and orientation information determined in accordance with a position and an orientation of the real camera in a real space;

sound data obtaining means for obtaining sound data representing a sound signal from a sound input device;

sound recognition means for recognizing a sound inputted into the sound input device;

setting means for setting one of a virtual object and a letter to be additionally displayed on the taken image as an additional display object, and for setting, based on a sound recognition result by the sound recognition means, at least one selected from the group consisting of a display position, an orientation, and a display form of the additional display object;

image generation means for repeatedly generating a combined image by superimposing, with reference to a position in the taken image in accordance with the position and orientation information, the additional display object set by the setting means; and

display control means for causing the display device to repeatedly display the combined image.

16. The image processing apparatus according to claim 15, wherein

the image processing apparatus incorporates the real camera, the display device, and the sound input device in at least one housing which incorporates the image processing apparatus.

17. A image processing system which includes a plurality of apparatuses communicable with each other, and which displays an image on a display device, the image processing system comprising:

taken image obtaining means for repeatedly obtaining an image taken by a real camera;

position and orientation calculation means for repeatedly calculating position and orientation information determined in accordance with a position and an orientation of the real camera in a real space;

sound data obtaining means for obtaining sound data representing a sound signal from a sound input device;

sound recognition means for recognizing a sound inputted into the sound input device;

setting means for setting one of a virtual object and a letter to be additionally displayed on the taken image as an additional display object, and for setting, based on a sound recognition result by the sound recognition means, at least one selected from the group consisting of a display position, an orientation, and a display form of the additional display object;

image generation means for repeatedly generating a combined image by superimposing, with reference to a position in the taken image in accordance with the position and orientation information, the additional display object set by the setting means; and

display control means for causing the display device to repeatedly display the combined image.

18. An image processing method performed by a processor or a cooperation by a plurality of processors included in an image processing system which includes at least one information processing apparatus capable of performing image processing for causing a display device to display an image, the image processing method comprising:

a taken image obtaining step for repeatedly obtaining an image taken by a real camera;

a position and orientation calculation step for repeatedly calculating position and orientation information determined in accordance with a position and an orientation of the real camera in a real space;

a sound data obtaining step for obtaining sound data representing a sound signal from a sound input device;

a sound recognition step for recognizing a sound inputted into the sound input device;

a setting step for setting one of a virtual object and a letter to be additionally displayed on the taken image as an additional display object, and for setting, based on a sound recognition result in the sound recognition step, at least one selected from the group consisting of a display position, an orientation, and a display form of the additional display object;

an image generation step for repeatedly generating a combined image by superimposing, with reference to a position in the taken image in accordance with the position and orientation information, the additional display object set in the setting step; and

a display control step for causing the display device to repeatedly display the combined image.