STORAGE MEDIUM HAVING STORED THEREON IMAGE PROCESSING PROGRAM, IMAGE PROCESSING APPARATUS, IMAGE PROCESSING SYSTEM, AND IMAGE PROCESSING METHOD

• US 20110304607A1
• Filed: 12/29/2010
• Published: 12/15/2011
• Est. Priority Date: 06/09/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Ito, Yuichiro
• Granted Patent: US 9,101,832 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/204 the platform being a handheld device
• CPC Class Codes: A63F 2300/6018 where the game content is authored by the player, e.g. level editor or by game device at runtime, e.g. level is created from music data on CD
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8094 Unusual game types, e.g. virtual cooking
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models

Abstract

Data of a coordinate input provided on a second image displayed on a display device is acquired, and input position data representing the coordinate input provided on the second image is generated. At least a first feature point is extracted from a first image, the first feature point being a feature point having a first feature on the first image; a predetermined superimposition image is superimposed on the first image, at a position on the first image based on the first feature point, the position corresponding to a position, represented by the input position data, on the second image based on a second feature point, the second feature point being a feature point having the first feature on the second image; and the first image on which the superimposition has been made is displayed on the display device.

Claims

1. A computer-readable storage medium having stored thereon an image processing program executed by a computer of an image processing apparatus that displays an image on a display device, the image processing program causing the computer to function as:

first image acquisition means for acquiring a first image;

first feature point extraction means for extracting at least a first feature point from the first image, the first feature point being a feature point having a first feature on the first image;

second image display control means for displaying a second image on the display device;

coordinate input acquisition means for acquiring data of a coordinate input provided on the second image displayed on the display device;

input position data generation means for generating input position data representing a position of the coordinate input provided on the second image, using the data acquired by the coordinate input acquisition means;

first image superimposition means for superimposing a predetermined superimposition image on the first image, at a position on the first image based on the first feature point, the position corresponding to a position, represented by the input position data, on the second image based on a second feature point, the second feature point being a feature point having the first feature on the second image; and

first image display control means for displaying on the display device the first image on which the superimposition has been made by the first image superimposition means.

2. The computer-readable storage medium having stored thereon the image processing program according to claim 1, further causing the computer to function as:

second image superimposition means for, based on the input position data, superimposing on the second image an input image representing the coordinate input, wherein

the second image display control means displays on the display device the second image on which the superimposition has been made by the second image superimposition means, and

the first image superimposition means superimposes the superimposition image on the first image, at a position on the first image based on the first feature point, the position corresponding to a position of the input image on the second image based on the second feature point.

3. The computer-readable storage medium having stored thereon the image processing program according to claim 2, wherein

the first image superimposition means superimposes the superimposition image on the first image, at the position on the first image based on the first feature point, the position corresponding to the position of the input image on the second image based on the second feature point, the superimposition image being an image based on the input image.

4. The computer-readable storage medium having stored thereon the image processing program according to claim 1, further causing the computer to function as:

second image acquisition means for acquiring the second image; and

second feature point extraction means for extracting the second feature point having the first feature from the second image acquired by the second image acquisition means.

5. The computer-readable storage medium having stored thereon the image processing program according to claim 1, wherein

the first image acquisition means acquires a moving image as the first image,

the first feature point extraction means extracts the first feature point from a frame image included in the moving image acquired by the first image acquisition means, and

the first image superimposition means superimposes the superimposition image on the frame image included in the moving image, at a position on the frame image based on the first feature point, the position corresponding to the position, represented by the input position data, on the second image based on the second feature point.

6. The computer-readable storage medium having stored thereon the image processing program according to claim 1, wherein

the second image display control means displays a still image as the second image on the display device,

the coordinate input acquisition means acquires data of a coordinate input provided on the still image displayed on the display device, and

the input position data generation means generates input position data representing a position of the coordinate input provided on the still image, using the data acquired by the coordinate input acquisition means.

7. The computer-readable storage medium having stored thereon the image processing program according to claim 5, wherein

the first feature point extraction means extracts the first feature point from each of a plurality of frame images included in the moving image acquired by the first image acquisition means, and

the first image superimposition means superimposes the superimposition image on said each of the plurality of frame images included in the moving image, at a position on said each of the plurality of frame images based on the first feature point, the position corresponding to the position, represented by the input position data, on the second image based on the second feature point.

8. The computer-readable storage medium having stored thereon the image processing program according to claim 7, wherein

the input position data generation means includes input position data updating means for sequentially updating the input position data every time the coordinate input acquisition means acquires the coordinate input, and

the first image superimposition means superimposes the superimposition image on the corresponding frame image, at a position on the corresponding frame image based on the first feature point, the position corresponding to a position represented by the input position data most recently updated by the input position data updating means.

9. The computer-readable storage medium having stored thereon the image processing program according to claim 5, further causing the computer to function as:

frame image storage control means for temporarily storing in a storage device the frame image included in the moving image acquired by the first image acquisition means, wherein,

when the first feature point extraction means has extracted the first feature point from the frame image acquired by the first image acquisition means, the first image superimposition means reads from the storage device the frame image from which the extraction has been made, and superimposes the superimposition image on the frame image, at the position on the frame image based on the first feature point.

10. The computer-readable storage medium having stored thereon the image processing program according to claim 1, wherein

the first feature point extraction means extracts at least the first feature point and a third feature point from the first image, the third feature point being a feature point having a second feature on the first image, and

the first image superimposition means superimposes the superimposition image on the first image, at a position on the first image based on the first feature point and the third feature point, the position corresponding to a position, represented by the input position data, on the second image based on the second feature point and a fourth feature point, the fourth feature point being a feature point having the second feature on the second image.

11. The computer-readable storage medium having stored thereon the image processing program according to claim 10, wherein

in accordance with a relationship between:

a distance between the second feature point and the fourth feature point on the second image; and

a distance between the first feature point and the third feature point on the first image, the first image superimposition means sets a size of the superimposition image corresponding to the position represented by the input position data, and superimposes the superimposition image on the first image.

12. The computer-readable storage medium having stored thereon the image processing program according to claim 10, wherein

the first image superimposition means determines a position and an orientation of the superimposition image to be superimposed on the first image, such that a relative positional relationship between:

the first feature point and the third feature point on the first image; and

the superimposition image, is the same as a relative positional relationship between;

the second feature point and the fourth feature point on the second image; and

the position represented by the input position data.

13. The computer-readable storage medium having stored thereon the image processing program according to claim 2, wherein

the first image superimposition means includes:

image extraction means for extracting from the second image the input image included in a region determined based on the second feature point; and

extracted-image superimposition means for superimposing the superimposition image on the first image based on the first feature point, the superimposition image being the input image extracted by the image extraction means.

14. The computer-readable storage medium having stored thereon the image processing program according to claim 13, wherein

the image extraction means extracts from the second image the input image included in a region surrounded by three or more feature points, each having a unique feature on the second image,

the first feature point extraction means extracts three or more feature points from the first image, the three or more feature points having the unique features and corresponding to the three or more feature points on the second image, and

the extracted-image superimposition means superimposes the input image extracted by the image extraction means on the first image, in a region surrounded by the three or more feature points extracted by the first feature point extraction means.

15. The computer-readable storage medium having stored thereon the image processing program according to claim 2, wherein

the first image superimposition means includes texture generation means for generating a texture representing the input image superimposed on the second image, and

the first image superimposition means sets texture coordinates of a predetermined polygon corresponding to the texture based on the second feature point on the second image, maps the texture on the polygon, places the polygon on the first image based on the first feature point on the first image, and superimposes the superimposition image on the first image.

16. The computer-readable storage medium having stored thereon the image processing program according to claim 1, wherein

the first image and the second image are each an image including a face image representing a person'"'"'s face,

the first feature point extraction means extracts the first feature point from the face image of a person'"'"'s face recognized in a face recognition process performed on the first image acquired by the first image acquisition means, the first feature point being a point having the first feature in accordance with the face image, and

the first image superimposition means superimposes the superimposition image on the first image such that the second feature point is a point having the first feature in accordance with the face image included in the second image.

17. The computer-readable storage medium having stored thereon the image processing program according to claim 4, wherein

the first image and the second image are each an image including a face image representing a person'"'"'s face,

the second feature point extraction means extracts the second feature point from the face image of a person'"'"'s face recognized in a face recognition process performed on the second image acquired by the second image acquisition means, the second feature point being a point having the first feature in accordance with the face image, and

the first feature point extraction means extracts the first feature point from the face image of a person'"'"'s face recognized in a face recognition process performed on the first image acquired by the first image acquisition means, the first feature point being a point having the first feature in accordance with the face image.

18. The computer-readable storage medium having stored thereon the image processing program according to claim 5, wherein

the first image acquisition means acquires, as the first image, a moving image of a real world captured in real time by a real camera available to the image processing apparatus.

19. The computer-readable storage medium having stored thereon the image processing program according to claim 2, further causing the computer to function as:

second image acquisition means for acquiring the second image; and

second feature point extraction means for;

extracting a plurality of feature points from an image representing an object recognized in a process of recognizing a predetermined object performed on the second image acquired by the second image acquisition means, each feature point having a unique feature on the image representing the object; and

further extracting a plurality of out-of-region points provided outside the image representing the object, in a radial manner from the corresponding feature points, wherein

the first feature point extraction means extracts a plurality of feature points from an image representing an object recognized in a process of recognizing the predetermined object performed on the first image acquired by the first image acquisition means, the plurality of feature points having the unique features and corresponding to the plurality of feature points extracted by the second feature point extraction means, and further extracts a plurality of out-of-region points provided outside the image representing the object, in a radial manner, so as to correspond to the plurality of out-of-region points extracted by the second feature point extraction means, and

the first image superimposition means includes;

image extraction means for extracting from the second image the input image included in a region surrounded by three or more of the feature points and/or the out-of-region points set by the second feature point extraction means; and

extracted-image superimposition means for superimposing the superimposition image on the first image, in a region surrounded by three or more of the feature points and/or the out-of-region points corresponding to the region from which the input image has been extracted, the superimposition image being the input image extracted by the image extraction means.

20. The computer-readable storage medium having stored thereon the image processing program according to claim 1, wherein

the display device includes at least a first display screen and a second display screen,

the second image display control means displays the second image on the second display screen, and

the first image display control means displays on the first display screen the first image on which the superimposition has been made by the first image superimposition means.

21. The computer-readable storage medium having stored thereon the image processing program according to claim 20, wherein

the display device includes a touch panel that covers the second display screen,

the coordinate input acquisition means acquires, as the data of the coordinate input, data representing a touch position of a touch performed on the touch panel, and

the input position data generation means generates, as the input position data, data representing a position on the second image that overlaps the touch position.

22. The computer-readable storage medium having stored thereon the image processing program according to claim 21, wherein

the first display screen is capable of displaying a stereoscopically visible image, using a left-eye image and a right-eye image,

the first image acquisition means acquires, as the first image, a stereoscopically visible image including a left-eye image and a right-eye image,

the first feature point extraction means extracts the first feature point from each of the left-eye image and the right-eye image of the first image,

the first image superimposition means superimposes the superimposition image on the first image, at a position of the left-eye image and a position of the right-eye image on the first image based on the first feature point, each position corresponding to the position, represented by the input position data, on the second image based on the second feature point, and

the first image display control means displays the stereoscopically visible image on the first display screen, using the left-eye image and the right-eye image of the first image on which the superimposition has been made by the first image superimposition means.

23. An image processing apparatus that displays an image on a display device, the image processing apparatus comprising:

first image acquisition means for acquiring a first image;

first feature point extraction means for extracting at least a first feature point from the first image, the first feature point being a feature point having a first feature on the first image;

second image display control means for displaying a second image on the display device;

coordinate input acquisition means for acquiring data of a coordinate input provided on the second image displayed on the display device;

input position data generation means for generating input position data representing a position of the coordinate input provided on the second image, using the data acquired by the coordinate input acquisition means;

first image superimposition means for superimposing a predetermined superimposition image on the first image, at a position on the first image based on the first feature point, the position corresponding to a position, represented by the input position data, on the second image based on a second feature point, the second feature point being a feature point having the first feature on the second image; and

first image display control means for displaying on the display device the first image on which the superimposition has been made by the first image superimposition means.

24. An image processing system that includes a plurality of apparatuses configured to communicate with each other and displays an image on a display device, the image processing system comprising:

first image acquisition means for acquiring a first image;

first feature point extraction means for extracting at least a first feature point from the first image, the first feature point being a feature point having a first feature on the first image;

second image display control means for displaying a second image on the display device;

coordinate input acquisition means for acquiring data of a coordinate input provided on the second image displayed on the display device;

input position data generation means for generating input position data representing a position of the coordinate input provided on the second image, using the data acquired by the coordinate input acquisition means;

first image superimposition means for superimposing a predetermined superimposition image on the first image, at a position on the first image based on the first feature point, the position corresponding to a position, represented by the input position data, on the second image based on a second feature point, the second feature point being a feature point having the first feature on the second image; and

first image display control means for displaying on the display device the first image on which the superimposition has been made by the first image superimposition means.

25. The image processing system according to claim 24, wherein

at least a first apparatus and a second apparatus are configured to communicate with each other,

the first apparatus includes:

the second image display control means;

the coordinate input acquisition means;

the input position data generation means;

second image superimposition means for, based on the input position data, superimposing on the second image an input image representing the coordinate input; and

data transmission means for transmitting data of the input image to the second apparatus, wherein

the second image display control means displays, on a display device available to the first apparatus, the second image on which the superimposition has been made by the second image superimposition means, and

the second apparatus includes;

data reception means for receiving the data of the input image from the first apparatus;

the first image acquisition means;

the first feature point extraction means;

the first image superimposition means; and

the first image display control means, wherein

based on the data of the input image received by the data reception means, the first image superimposition means superimposes the superimposition image on the first image, at the position on the first image based on the first feature point, the position corresponding to a position of the input image on the second image based on the second feature point, the superimposition image being an image based on the input image, and

the first image display control means displays, on a display device available to the second apparatus, the first image on which the superimposition has been made by the first image superimposition means.

26. An image processing method performed by a processor or by a cooperation of a plurality of processors, the processor and the plurality of processors included in an image processing system that includes at least one information processing apparatus capable of performing image processing for displaying an image on a display device, the image processing method comprising:

a first image acquisition step of acquiring a first image;

a first feature point extraction step of extracting at least a first feature point from the first image, the first feature point being a feature point having a first feature on the first image;

a second image display control step of displaying a second image on the display device;

a coordinate input acquisition step of acquiring data of a coordinate input provided on the second image displayed on the display device;

an input position data generation step of generating input position data representing a position of the coordinate input provided on the second image, using the data acquired in the coordinate input acquisition step;

a first image superimposition step of superimposing a predetermined superimposition image on the first image, at a position on the first image based on the first feature point, the position corresponding to a position, represented by the input position data, on the second image based on a second feature point, the second feature point being a feature point having the first feature on the second image; and

a first image display control step of displaying on the display device the first image on which the superimposition has been made in the first image superimposition step.