Storage medium having stored thereon image processing program, image processing apparatus, image processing system, and image processing method

• US 9,101,832 B2
• Filed: 12/29/2010
• Issued: 08/11/2015
• Est. Priority Date: 06/09/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Ito, Yuichiro
• Primary Examiner(s): Tseng, Charles
• Application Number:
  US12/980,620
• Publication Number:
  US 20110304607A1
• Time in Patent Office: 1,686 Days
• Field of Search: 345/629
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/6018 where the game content is authored by the player, e.g. level editor or by game device at runtime, e.g. level is created from music data on CD
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8094 Unusual game types, e.g. virtual cooking
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models

Abstract

Data of a coordinate input provided on a second image displayed on a display device is acquired, and input position data representing the coordinate input provided on the second image is generated. At least a first feature point is extracted from a first image, the first feature point being a feature point having a first feature on the first image; a predetermined superimposition image is superimposed on the first image, at a position on the first image based on the first feature point, the position corresponding to a position, represented by the input position data, on the second image based on a second feature point, the second feature point being a feature point having the first feature on the second image; and the first image on which the superimposition has been made is displayed on the display device.

Claims

1. A non-transitory computer-readable storage medium having stored thereon an image processing program executed by a computer of an image processing apparatus that displays an image on a display device, the image processing program comprising instructions configured to cause the computer to:

acquire a first image;

extract at least a first feature point from the first image, the first feature point being a feature point having a first feature on the first image;

display the first image at a first location on the display device;

display a second image at a second location on the display device, where the first and second images are displayed at different, non-overlapping, locations on the display device;

while the first image is displayed at the first location, acquire data of a coordinate input provided on the second image displayed at the second location on the display device;

generate input position data representing a position of the coordinate input provided on the second image using the acquired data;

superimpose a predetermined superimposition image on the first image to create superimposed first image, the predetermined superimposition image being superimposed at a position on the first image based on the first feature point, the position on the first image corresponding to a position, represented by the input position data, on the second image based on a second feature point, the second feature point being a feature point having the first feature on the second image; and

update the first image displayed at the first location to be the superimposed first image, the superimposed first image being concurrently displayed with the second image.

2. The non-transitory computer-readable storage medium of claim 1, wherein the instructions further the computer to:

based on the input position data, superimpose on the second image an input image representing the coordinate input,

wherein the second image is displayed on which the input image representing the coordinate input has been superimposed, and

the position, represented by the input position data, is the position of the input image.

3. The non-transitory computer-readable storage medium of claim 2, wherein

the predetermined superimposition image is an image based on the input image.

4. The non-transitory computer-readable storage medium of claim 2, wherein the instructions are further configured to cause the computer to:

extract from the second image the input image included in a region determined based on the second feature point,

wherein the predetermined superimposition image is the extracted input image.

5. The non-transitory computer-readable storage medium of claim 4, wherein

the input image that is extracted is included in a region surrounded by three or more feature points, with each feature point having a unique feature on the second image,

three or more feature points are extracted from the first image, the three or more feature points having the unique features and corresponding to the three or more feature points on the second image, and

the extracted input image is superimposed on the first image in a region surrounded by the extracted three or more feature points.

6. The non-transitory computer-readable storage medium of claim 2, wherein the instructions are further configured to cause the computer to:

generate a texture representing the input image superimposed on the second image;

set texture coordinates of a polygon corresponding to the generated texture based on the second feature point on the second image;

map the generated texture on the polygon; and

place the polygon on the first image based on the first feature point on the first image.

7. The non-transitory computer-readable storage medium of claim 2, wherein the instructions are further configured to cause the computer to:

acquire the second image;

extract a first plurality of feature points from an image representing an object recognized in a process of recognizing a predetermined object performed on the acquired second image, each feature point having a unique feature on the first image representing the object;

extract a first plurality of out-of-region points provided outside the image representing the object, in a radial manner from the corresponding feature points;

extract a second plurality of feature points from an image representing an object recognized in a process of recognizing the predetermined object performed on the acquired first image, the second plurality of feature points having the unique features and corresponding to the extracted first plurality of feature points;

extract a second plurality of out-of-region points provided outside the image representing the object, in a radial manner, so as to correspond to the extracted first plurality of out-of-region points; and

extract from the second image the input image included in a region surrounded by three or more of the first plurality of feature points and/or the first plurality of out-of-region points,

wherein the predetermined superimposition image is superimposed on the first image, in a region surrounded by three or more of the second plurality of feature points and/or the second plurality of out-of-region points corresponding to the region from which the input image has been extracted, the superimposition image being the extracted input image.

8. The non-transitory computer-readable storage medium of claim 1, wherein the instructions further cause the computer to:

acquire the second image; and

extract the second feature point having the first feature from the acquired second image.

9. The non-transitory computer-readable storage medium of claim 8, wherein

the first image and the second image are each an image including a face image representing a person'"'"'s face,

the second feature point is extracted from the face image of a person'"'"'s face recognized in a face recognition process performed on the acquired second image, the second feature point being a point having the first feature in accordance with the face image, and

the first feature point is extracted from the face image of a person'"'"'s face recognized in a face recognition process performed on the acquired first image, the first feature point being a point having the first feature in accordance with the face image.

10. The non-transitory computer-readable storage medium of claim 1, wherein

the first image is a moving image,

the first feature point is extracted from a frame image included in the moving image, and

the predetermined superimposition image is superimposed on the frame image included in the moving image.

11. The non-transitory computer-readable storage medium of claim 10, wherein

the first feature point is extracted from each of a plurality of frame images included in the acquired moving image, and

the predetermined superimposition image is superimposed on each of the plurality of frame images included in the moving image.

12. The non-transitory computer-readable storage medium of claim 11, wherein the instructions are further configured to cause the computer to:

sequentially update the input position data every time coordinate input is acquired, and

the predetermined superimposition image is superimposed on the corresponding frame image, at a position on the corresponding frame image based on the first feature point, the position on the first image corresponding to a position represented by the most recently updated input position data.

13. The non-transitory computer-readable storage medium of claim 10, wherein the instructions are further configured to cause the computer to:

store in a storage device the frame image included in the acquired moving image, wherein,

when the first feature point is extracted from the frame image, the frame image is read from the storage device from which the extraction has been made.

14. The non-transitory computer-readable storage medium of claim 10, wherein

the first image is a moving image of a real world captured in real time by a real camera available to the image processing apparatus.

15. The non-transitory computer-readable storage medium of claim 1, wherein

the second image is a still image displayed on the display device.

16. The non-transitory computer-readable storage medium of claim 1, wherein the instructions are further configured to cause the computer to:

extract at least a third feature point from the acquired first image, the third feature point being a feature point having a second feature on the first image,

wherein the predetermined superimposition image is superimposed on the first image at a position on the first image that is based on the first feature point and the third feature point, the position on the first image corresponding to a position, represented by the input position data, on the second image that is based on the second feature point and a fourth feature point, the fourth feature point being a feature point having the second feature on the second image.

17. The non-transitory computer-readable storage medium of claim 16, wherein

the superimposed predetermined superimposition image is set to a size in accordance with a relationship between (1) distance between the second feature point and the fourth feature point on the second image and (2) a distance between the first feature point and the third feature point on the first image.

18. The non-transitory computer-readable storage medium of claim 16, wherein the instructions are further configured to cause the computer to:

determine a position and an orientation of the superimposition image to be superimposed on the first image, such that a relative positional relationship between (1) the first feature point and the third feature point on the first image and (2) the superimposition image, is the same as a relative positional relationship between (1) the second feature point and the fourth feature point on the second image and (2) the position represented by the input position data.

19. The non-transitory computer-readable storage medium of claim 1, wherein

the first image and the second image are each an image including a face image representing a person'"'"'s face,

the first feature point is extracted from the face image of a person'"'"'s face recognized in a face recognition process performed on the acquired first image, the first feature point being a point having the first feature in accordance with the face image, and

the superimposition image is superimposed on the first image such that the second feature point is a point having the first feature in accordance with the face image included in the second image.

20. The non-transitory computer-readable storage medium of claim 1, wherein

the display device includes at least a first display screen and a second display screen,

the second image is displayed on the second display screen, and

the first image on which the superimposition has been made is displayed on the first display screen.

21. The non-transitory computer-readable storage medium of claim 20, wherein

the display device includes a touch panel that covers the second display screen,

the data of the coordinate input represents a touch position of a touch performed on the touch panel, and

the input position data is generated to represent a position on the second image that overlaps the touch position.

22. The non-transitory computer-readable storage medium of claim 21, wherein

the first display screen is capable of displaying a stereoscopically visible image, using a left-eye image and a right-eye image,

the first image is a stereoscopically visible image including a left-eye image and a right-eye image, the first feature point is extracted from each of the left-eye image and the right-eye image of the first image,

the superimposition image is superimposed on the first image, at a position of the left-eye image and a position of the right-eye image on the first image based on the first feature point, each position corresponding to the position, represented by the input position data, on the second image based on the second feature point, and

the stereoscopically visible image is displayed on the first display screen, using the left-eye image and the right-eye image of the first image on which the superimposition has been made.

23. The non-transitory computer-readable storage medium of claim 1, wherein drawing input that corresponds to the data of the coordinate input provided on the second input temporally corresponds to display of corresponding image information used in the predetermined superimposition image that is superimposed on the simultaneously displayed first image.

24. The non-transitory computer-readable storage medium of claim 1, wherein the first and second images are different images.

25. An image processing apparatus that displays an image on a display device, the image processing apparatus comprising:

a processing system that includes at least one processor, the processing system configured to;

accept a first image;

extract at least a first feature point from the first image, the first feature point being a feature point having a first feature on the first image;

output a second image to the display device for display at a first location;

acquire data of a coordinate input provided on the display device and within the first location where the second image is displayed;

generate input position data representing a position of the coordinate input provided on the second image, using the acquired data;

superimpose a predetermined superimposition image on the first image, at a position on the first image based on the first feature point, the position corresponding to a position, represented by the input position data, on the second image based on a second feature point, the second feature point being a feature point having the first feature on the second image; and

output the first image, on which the predetermined superimposition image has been superimposed, to the display device for display at a second location that is non-overlapping and different from the first location where the second image is displayed, the first and second images being output for concurrent display on the display device,

where the first image is updated to include the predetermined superimposition image as the coordinate input is provided that the displayed first image is based upon.

26. An image processing system that includes a plurality of apparatuses configured to communicate with each other and displays an image on a display device, the image processing system comprising:

first image acquisition unit configured to acquire a first image;

first feature point extraction unit configured to extract at least a first feature point from the first image, the first feature point being a feature point having a first feature on the first image;

second image display control unit configured to display a second image on the display device;

coordinate input acquisition unit configured to acquire data of a coordinate input provided on the second image displayed on the display device;

input position data generation unit configured generate input position data representing a position of the coordinate input provided on the second image, using the data acquired by the coordinate input acquisition unit;

first image superimposition unit configured to superimpose a predetermined superimposition image on the first image, at a position on the first image based on the first feature point, the position corresponding to a position, represented by the input position data, on the second image based on a second feature point, the second feature point being a feature point having the first feature on the second image; and

first image display control unit configured to display on the display device the first image on which the superimposition has been made by the first image superimposition unit,

wherein the first image, which has been superimposed with the predetermined superimposition image, and the second image are concurrently displayed on the display device at different, non-overlapping, locations thereon such that the first image is displayed while the coordinate input is provided.

27. The image processing system according to claim 26, wherein

at least a first apparatus and a second apparatus are configured to communicate with each other,

the first apparatus includes:

the second image display control unit;

the coordinate input acquisition unit;

the input position data generation unit;

a second image superimposition unit configured to, based on the input position data, superimpose on the second image an input image representing the coordinate input; and

a first transceiver configured to transmit data of the input image to the second apparatus, wherein

the second image display control unit is further configured to display, on a display device available to the first apparatus, the second image on which the superimposition has been made by the second image superimposition unit, and

the second apparatus includes;

a second transceiver configured to receive the data of the input image from the first apparatus;

the first image acquisition unit;

the first feature point extraction unit;

the first image superimposition unit; and

the first image display control unit, wherein

based on the data of the input image received by the second transceiver, the first image superimposition unit is further configured to superimpose the superimposition image on the first image, at the position on the first image based on the first feature point, the position corresponding to a position of the input image on the second image based on the second feature point, the superimposition image being an image based on the input image, and

the first image display control unit is further configured to display, on a display device available to the second apparatus, the first image on which the superimposition has been made by the first image superimposition unit.

28. An image processing method performed by a processor or by plurality of processors in cooperation, the processor and/or the plurality of processors included in an image processing system that includes at least one information processing apparatus capable of performing image processing for displaying an image on a display device, the image processing method comprising:

acquiring a first image;

extracting at least a first feature point from the first image, the first feature point being a feature point having a first feature on the first image;

displaying the first image at a first location on the display device;

displaying a second image at a second location on the display device such that the first and second images are displayed at different, non-overlapping, locations on the display device;

while the first image is being displayed at the first location, acquiring data of a coordinate input provided on the second image displayed at the second location;

generating input position data representing a position of the coordinate input provided on the second image, using the acquired data;

responsive to the generation of the input position data, superimposing a predetermined superimposition image on the first image to generate a superimposed first image, the predetermined superimposition image being superimposed at a position on the first image based on the first feature point, the position corresponding to a position, represented by the input position data, on the second image based on a second feature point, the second feature point being a feature point having the first feature on the second image; and

updating the first image displayed at the first location to be the generated superimposed first image at the first location, the updated first image being concurrently displayed with the second image.