Storage medium having stored therein image processing program, image processing apparatus, image processing system, and image processing method

• US 8,854,356 B2
• Filed: 11/15/2010
• Issued: 10/07/2014
• Est. Priority Date: 09/28/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Oyagi, Yasuyuki, Takadera, Tatsuya
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): CHEN, FRANK S
• Application Number:
  US12/946,248
• Publication Number:
  US 20120075285A1
• Time in Patent Office: 1,422 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/5252 using two or more virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character changes room or displaying a rear-mirror view in a car-driving game
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8058 Virtual breeding, e.g. tamagotchi
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 : Stereo images
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 7/73 : using feature-based methods

Abstract

An image taken by a real camera is repeatedly obtained, and position and orientation information determined in accordance with a position and an orientation of a real camera in a real space is repeatedly calculated. A virtual object or a letter to be additionally displayed on the taken image is set as an additional display object, and based on a result of recognition of a sound inputted into a sound input device, at least one selected from the group consisting of a display position, an orientation, and a display form of the additional display object is set. A combined image repeatedly generated by superimposing on the taken image the set additional display object with reference to a position in the taken image in accordance with the position and orientation information is displayed on a display device.

Claims

1. A non-transitory computer-readable storage medium having stored therein an image processing program executed by a computer of an image processing apparatus for displaying an image on a display device, the image processing program causing the computer to perform functionality comprising:

obtaining an image taken by a real camera;

calculating position and orientation information determined in accordance with a position and an orientation of the real camera in a real space;

obtaining sound data representing a sound signal from a sound input device;

recognizing a sound inputted into the sound input device;

setting, based on the taken image, a virtual object to be additionally displayed on the taken image and setting, based on a sound recognition result, a display animation of the virtual object;

setting, when the sound inputted into the sound input device is recognized as a first sound and the position and orientation information satisfies a first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a first action, and setting, when the sound inputted into the sound input device is recognized as the first sound and the position and orientation information does not satisfy the first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a second action;

setting in a virtual world a position and an orientation of a virtual camera based on the position and orientation information;

arranging the virtual object set in the virtual world;

generating as a virtual world image an image in the virtual world seen from the virtual camera;

generating as a combined image an image obtained by superimposing the virtual world image, including the virtual object, on the taken image; and

causing the display device to display the combined image.

2. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising:

detecting one of a certain imaging target and a certain characteristic point included in the taken image and calculating, based on a result of the detection, information representing a relative position and a relative orientation between the real camera and the one of the imaging target and the characteristic point, as the position and orientation information.

3. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 2, wherein the image processing program further causing the computer to perform functionality comprising:

calculating, as the position and orientation information, information representing a relative position and a relative orientation between the imaging target and the real camera;

setting in a virtual world a position and a direction corresponding to the imaging target, based on the position and orientation information, and setting in the virtual world a position and an orientation of the virtual camera, with reference to the position and the direction corresponding to the imaging target, based on the position and orientation information;

arranging the virtual object set in the virtual world, with reference to the position corresponding to the imaging target; and

setting, when the sound inputted into the sound input device is recognized as a first sound, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object, based on a first action to be performed with reference to the direction corresponding to the imaging target in the virtual world, and setting, when the sound inputted into the sound input device is recognized as a second sound, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object, based on a second action to be performed with reference to the direction to the virtual camera in the virtual world.

4. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising calculating the position and orientation information by using at least one of a geographic position of the real camera and an azimuth of an imaging direction of the real camera in the real space.

5. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising setting, at one of a case where the sound is not inputted into the sound input device and a case where the sound recognized is not a sound that causes the virtual object to be set, the display animation of the virtual object, to a predetermined content.

6. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising saving in a storage device a latest combined image currently generated in accordance with a photographing instruction given by a user.

7. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising:

obtaining a first image taken by a first real camera and a second image taken by a second real camera which is provided at a position spaced from the first real camera by a predetermined distance,

calculating first position and orientation information determined in accordance with a position and an orientation of the first real camera in the real space and second position and orientation information determined in accordance with a position and an orientation of the second real camera in the real space,

generating a first combined image by superimposing, with reference to a position in the first taken image in accordance with the first position and orientation information, the set virtual object, and generating a second combined image by superimposing, with reference to a position in the second taken image in accordance with the second position and orientation information, the set virtual object, and

outputting the first combined image and the second combined image to a display device capable of displaying a stereoscopically visible image, and causing the display device to stereoscopically display the output first and second combined images including the virtual object.

8. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising determining whether the sound inputted into the sound input device is a first sound at least by verifying the sound inputted into the sound input device against a sound registered in advance as a registered sound, and determining whether the sound inputted into the sound input device is a second sound based only on the level of a sound waveform of the sound inputted into the sound input device.

9. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 8, wherein the image processing program further causing the computer to perform functionality comprising:

allowing a user to input a sound corresponding to an instruction to be given to the virtual object, and registering, in a storage device, sound data corresponding to the sound as the registered sound corresponding to the instruction;

determining whether the sound inputted into the sound input device is the first sound by using the sound data registered as the registered sound; and

setting, when the sound inputted into the sound input device is recognized as the first sound, the display animation of the virtual object based on the instruction corresponding to the first sound and registered in advance.

10. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising:

determining whether the sound inputted into the sound input device is a first sound, based on the likelihood of a feature parameter time series registered in advance as a registered sound against a feature parameter time series obtained from the sound input pattern inputted into the sound input device, and determining whether the sound inputted into the sound input device is a third sound, based only on at least one of the level of a sound waveform and spectrum information of the sound inputted into the sound input device.

11. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising:

obtaining the taken image from the real camera incorporated in a housing which incorporates the image processing apparatus,

obtaining the sound data from the sound input device incorporated in the housing, and

causing the display device incorporated in the housing to display the combined image.

12. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the display position is selected from the group as being set based on the sound recognition result.

13. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the orientation is selected from the group as being set based on the sound recognition result.

14. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising detecting one of a certain imaging target and a certain characteristic point from the taken image, and setting the virtual object based on the detected imaging target or characteristic point.

15. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising determining, when a direction of a certain imaging target included in the taken image is within a predetermined range in the image recognition result of the taken image, that the image recognition result satisfies the first condition.

16. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 1, wherein the image processing program further causing the computer to perform functionality comprising detecting one of a certain imaging target and a certain characteristic point from the taken image in the image recognition, and setting an action of the virtual object based on the detected imaging target or characteristic point.

17. An image processing apparatus for displaying an image on a display device, comprising at least one processor, the image processing apparatus configured to:

obtain an image taken by a real camera;

calculate position and orientation information determined in accordance with a position and an orientation of the real camera in a real space,

obtain sound data representing a sound signal from a sound input device;

recognize a sound inputted into the sound input device;

set, based on the taken image, a virtual object to be additionally displayed on the taken image, and set, based on a sound recognition result, a display animation of the virtual object;

set, when the sound inputted into the sound input device is recognized as a first sound and the position and orientation information satisfies a first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a first action, and set, when the sound inputted into the sound input device is recognized as the first sound and the position and orientation information does not satisfy the first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a second action;

set in a virtual world a position and an orientation of a virtual camera based on the position and orientation information;

arrange the virtual object set in the virtual world;

generate as a virtual world image an image in the virtual world seen from the virtual camera;

generate as a combined image an image obtained by superimposing the virtual world image, including the virtual object, on the taken image; and

cause the display device to display the combined image.

18. The image processing apparatus according to claim 17, wherein

the image processing apparatus incorporates the real camera, the display device, and the sound input device in at least one housing which incorporates the image processing apparatus.

19. An image processing system which includes a plurality of apparatuses communicable with each other, and which displays an image on a display device, the image processing system comprising:

a processing system having at least one processor, the processing system configured to;

obtain an image taken by a real camera,

calculate position and orientation information determined in accordance with a position and an orientation of the real camera in a real space,

obtain sound data representing a sound signal from a sound input device,

recognize a sound inputted into the sound input device,

set, based on the taken image, a virtual object to be additionally displayed on the taken image, and set, based on a sound recognition result, a display animation of the virtual object,

set, when the sound inputted into the sound input device is recognized as a first sound and the position and orientation information satisfies a first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a first action, and set, when the sound inputted into the sound input device is recognized as the first sound and the position and orientation information does not satisfy the first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a second action,

set in a virtual world a position and an orientation of a virtual camera based on the position and orientation information;

arrange the virtual object set in the virtual world;

generate as a virtual world image an image in the virtual world seen from the virtual camera;

generate as a combined image an image obtained by superimposing the virtual world image, including the virtual object, on the taken image; and

cause the display device to display the combined image.

20. A non-transitory computer-readable storage medium having stored therein an image processing program executed by a computer of an image processing apparatus for displaying an image on a display device, the image processing program causing the computer to perform functionality comprising:

repeatedly obtaining an image taken by a real camera;

repeatedly calculating position and orientation information determined in accordance with a position and an orientation of the real camera in real space;

obtaining sound data representing a sound signal from a sound input device;

recognizing a sound inputted into the sound input device;

setting a virtual object to be additionally displayed on the taken image;

setting, when the sound inputted into the sound input device is recognized as a first sound and the position and orientation information satisfies a first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a first action, and setting, when the sound inputted into the sound input device is not recognized as the first sound and the position and orientation information satisfies the first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a second action;

repeatedly generating a combined image by, based on a position included in the taken image and corresponding to the position and orientation information, superimposing the set virtual object;

setting in a virtual world a position and an orientation of a virtual camera based on the position and orientation information;

arranging the virtual object set in the virtual world;

generating as a virtual world image an image in the virtual world seen from the virtual camera; and

repeatedly causing the display device to display the combined image.

21. The non-transitory computer-readable storage medium having stored therein the image processing program according to claim 20, wherein the image processing program further causing the computer to perform functionality comprising detecting one of a certain imaging target and a certain characteristic point from the taken image in the image recognition, and setting an action of the virtual object based on the detected imaging target or characteristic point.

22. An image processing apparatus for displaying an image on a display device, comprising at least one processor, the image processing apparatus configured to:

repeatedly obtain an image taken by a real camera;

repeatedly calculate position and orientation information determined in accordance with a position and an orientation of the real camera in real space;

obtain sound data representing a sound signal from a sound input device;

recognize a sound inputted into the sound input device;

set a virtual object to be additionally displayed on the taken image;

set, when the sound inputted into the sound input device is recognized as a first sound and the position and orientation information satisfies a first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a first action, and set, when the sound inputted into the sound input device is not recognized as the first sound and the position and orientation information satisfies the first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a second action;

repeatedly generate a combined image by, based on a position included in the taken image and corresponding to the position and orientation information, superimposing the set virtual object;

set in a virtual world a position and an orientation of a virtual camera based on the position and orientation information;

arrange the virtual object set in the virtual world;

generate as a virtual world image an image in the virtual world seen from the virtual camera; and

repeatedly cause the display device to display the combined image.

23. An image processing system which includes a plurality of apparatuses communicable with each other, and which displays an image on a display device, the image processing system comprising:

a processing system having at least one processor, the processing system configured to;

repeatedly obtain an image taken by a real camera,

repeatedly calculate position and orientation information determined in accordance with a position and an orientation of the real camera in real space,

obtain sound data representing a sound signal from a sound input device,

recognize a sound inputted into the sound input device,

set a virtual object to be additionally displayed on the taken image,

set, when the sound inputted into the sound input device is recognized as a first sound and the position and orientation information satisfies a first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a first action, and set, when the sound inputted into the sound input device is not recognized as the first sound and the position and orientation information satisfies the first condition, at least one selected from the group consisting of the display position, the orientation, and the display form of the virtual object based on a second action,

repeatedly generate a combined image by, based on a position included in the taken image and corresponding to the position and orientation information, superimposing the set virtual object,

set in a virtual world a position and an orientation of a virtual camera based on the position and orientation information,

arrange the virtual object set in the virtual world,

generate as a virtual world image an image in the virtual world seen from the virtual camera, and

repeatedly cause the display device to display the combined image.

24. A non-transitory computer-readable storage medium having stored therein an image processing program executed by a computer of an image processing apparatus for displaying an image on a display device, the image processing program causing the computer to perform functionality comprising:

obtaining an image taken by a real camera;

calculating position and orientation information determined in accordance with a position and an orientation of the real camera in a real space by using at least one of a geographic position of the real camera and an azimuth of an imaging direction of the real camera in the real space;

obtaining sound data representing a sound signal from a sound input device;

recognizing a sound inputted into the sound input device;

setting, based on the taken image, a letter to be additionally displayed on the taken image, and setting, based on a sound recognition result, a display animation of the letter;

estimating a subject whose image has been taken in the taken image, based on at least one of the geographic position of the real camera and the azimuth of the imaging direction of the real camera represented by the position and orientation information, the letter representing information about the subject;

changing, when the letter coincides with a word recognized through sound recognition, a display form into that indicating that the letter has been selected, and setting the letter;

generating a combined image by superimposing, with reference to the position in the taken image in accordance with the position and orientation information, the letter corresponding to the subject and which coincides with the word in the updated display form; and

causing the display device to display the combined image,

wherein in setting the display animation at least one selected from a group consisting of a display position, an orientation, and a display form of the letter is set.

25. An image processing apparatus for displaying an image on a display device, comprising at least one processor, the image processing apparatus configured to:

obtain an image taken by a real camera;

calculate position and orientation information determined in accordance with a position and an orientation of the real camera in a real space by using at least one of a geographic position of the real camera and an azimuth of an imaging direction of the real camera in the real space;

obtain sound data representing a sound signal from a sound input device;

recognize a sound inputted into the sound input device;

set, based on the taken image, a letter to be additionally displayed on the taken image, and set, based on a sound recognition result, a display animation of the letter;

estimate a subject whose image has been taken in the taken image, based on at least one of the geographic position of the real camera and the azimuth of the imaging direction of the real camera represented by the position and orientation information, the letter representing information about the subject;

change, when the letter coincides with a word recognized through sound recognition, a display form into that indicating that the letter has been selected;

generate a combined image by superimposing, with reference to the position in the taken image in accordance with the position and orientation information, the letter corresponding to the subject and which coincides with the word in the updated display form; and

cause the display device to display the combined image,

wherein in setting the display animation at least one selected from a group consisting of a display position, an orientation, and a display form of the letter is set.

26. An image processing system which includes a plurality of apparatuses communicable with each other, and which displays an image on a display device, the image processing system comprising:

a processing system having at least one processor, the processing system configured to;

obtain an image taken by a real camera,

calculate position and orientation information determined in accordance with a position and an orientation of the real camera in a real space by using at least one of a geographic position of the real camera and an azimuth of an imaging direction of the real camera in the real space,

obtain sound data representing a sound signal from a sound input device,

recognize a sound inputted into the sound input device,

set, based on the taken image, a letter to be additionally displayed on the taken image, and set, based on a sound recognition result, a display animation of the letter,

estimate a subject whose image has been taken in the taken image, based on at least one of the geographic position of the real camera and the azimuth of the imaging direction of the real camera represented by the position and orientation information, the letter representing information about the subject,

change, when the letter coincides with a word recognized through sound recognition, a display form into that indicating that the letter has been selected,

generate a combined image by superimposing, with reference to the position in the taken image in accordance with the position and orientation information, the letter corresponding to the subject and which coincides with the word in the updated display form, and

cause the display device to display the combined image,

wherein in setting the display animation at least one selected from a group consisting of a display position, an orientation, and a display form of the letter is set.