Image Processing Apparatus, Image Processing Method, and Program

• US 20080199044A1
• Filed: 02/19/2008
• Published: 08/21/2008
• Est. Priority Date: 02/20/2007
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Tsurumi, Shingo
• Granted Patent: US 8,126,206 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/269 using gradient-based methods

Abstract

Disclosed herein is an image processing apparatus for recognizing, from a taken image, an object corresponding to a registered image registered in advance, including, an image taker configured to take an image of a subject to obtain the taken image of the subject, a recognizer configured to recognize, from the taken image, an object corresponding to the registered image, a first specified area tracker configured to execute first specified area tracking processing for tracking, in the taken image, a first tracking area specified on the basis of a result of recognition by the recognizer, and a second specified area tracker configured to execute second specified area tracking processing for tracking a second specified area specified on the basis of a result of the first specified area tracking processing.

Claims

1. An image processing apparatus for recognizing, from a taken image, an object corresponding to a registered image registered in advance, comprising:

an image taker configured to take an image of a subject to obtain said taken image of said subject;

a recognizer configured to recognize, from said taken image, an object corresponding to said registered image;

a first specified area tracker configured to execute first specified area tracking processing for tracking, in said taken image, a first tracking area specified on the basis of a result of recognition by said recognizer; and

a second specified area tracker configured to execute second specified area tracking processing for tracking a second specified area specified on the basis of a result of said first specified area tracking processing.

2. The image processing apparatus according to claim 1, wherein said first specified area tracker executes said first specified area tracking processing on each taken image in a period of time ranging from the taking of an image for use in the recognition by said recognizer to the ending of said first specified area tracking processing.

3. The image processing apparatus according to claim 1, further comprising:

a generator configured to generate, on the basis of a result of said second specified area tracking processing and said registered image, a registered image having substantially a same size and a same posture as those of said object as a correction image for correcting said taken image; and

a synthesizer configured to synthesize, on the basis of a result of said second specified area tracking processing, said correction image with said second specified area in said taken image.

4. The image processing apparatus according to claim 3, wherein, if a ratio of an area of a screen for displaying said taken image to an area of said correction image is equal to or higher than a reference value, said synthesizer synthesizes said correction image with said second specified area in said taken image and, if said ratio is below said reference value, synthesizes said correction image with said taken image as an image for said screen in the entirety thereof.

5. The image processing apparatus according to claim 1, further comprising:

a generator configured to generate, on the basis of a result of said second specified area tracking processing and said registered image, a registered image having substantially a same size as that of said object and a predetermined posture as a correction image for correcting said taken image; and

a synthesizer configured to synthesize, on the basis of a result of said second specified area tracking processing, said correction image with a location of said second specified area in said taken image.

6. The image processing apparatus according to claim 1, further comprising:

a generator configured to generate, on the basis of a result of said second specified area tracking processing and an associated image associated with said registered image, an associated image having substantially a same size and a same posture as those of said object as a correction image for correcting said taken image; and

a synthesizer configured to synthesize, on the basis of a result of said second specified area tracking processing, said correction image with said second specified area in said taken image.

7. The image processing apparatus according to claim 1, wherein said first specified area tracker has a first specifier configured to specify said first specified area;

a first extractor configured to extract a first feature point in said first specified area in said taken image;

a first calculator configured to calculate first movement information indicative of a movement of said first feature point;

a first parameter calculator configured to calculate a first parameter indicative of a location and a posture of said first specified area from said first movement information;

a first error calculator configured to calculate a first error that is an error between a multiplication value obtained by multiplying said first parameter by a value indicative of a location of a first feature point in said first specified area of a taken image taken before said taken image and a value indicative of a location of a first feature point in said first specified area in said taken image; and

a second score calculator configured to calculate a first score of said first parameter on the basis of said first error;

wherein said first specifier shifts said first specified area on the basis of said first parameter having said first score having the highest thereof,

said second specified area tracker has

a second specifier configured to specify said second specified area on the basis of said first parameter;

a second extractor configured to extract a second feature point in said second specified area in said taken image;

a second calculator configured to calculate second movement information indicative of a movement of said second feature point;

a second parameter calculator configured to calculate a second parameter indicative of a location and a posture of said second specified area from said second movement information;

a second error calculator configured to calculate a second error that is an error between a multiplication value obtained by multiplying said second parameter by a value indicative of a location of a first feature point in said second specified area of a taken image taken before said taken image and a value indicative of a location of a second feature point in said second specified area in said taken image; and

a score calculator configured to calculate a second score of said second parameter on the basis of said second error;

wherein said second specifier shifts said second specified area on the basis of said second parameter having said second score having the highest thereof.

8. The image processing apparatus according to claim 7, wherein said first movement information and said second movement information are each an optical flow.

9. The image processing apparatus according to claim 7, wherein said first parameter and said second parameter are each at least one of parameters of affine transformation and projective transformation.

10. The image processing apparatus according to claim 7, wherein said first parameter calculator calculates said first parameter from said first movement information of at least one of at least three of said first feature points and at least four of said first feature points;

and

said second parameter calculator calculates said second parameter from said second movement information of at least one of at least three of said second feature points and at least four of said second feature points.

11. An image processing method for an image processing apparatus for recognizing, from a taken image, an object corresponding to a registered image, comprising the steps of:

taking an image of a subject to obtain said taken image of said subject;

recognizing, from said taken image, an object corresponding to said registered image;

executing first specified area tracking processing for tracking, in said taken image, a first tracking area specified on the basis of a result of recognition in said recognizing step; and

executing second specified area tracking processing for tracking a second specified area specified on the basis of a result of said first specified area tracking processing.

12. A program configured to make a computer execute recognition processing for recognizing, from a taken image, an object corresponding to a registered image registered in advance, comprising the steps of:

taking an image of a subject to obtain said taken image of said subject;

recognizing, from said taken image, an object corresponding to said registered image;

executing first specified area tracking processing for tracking, in said taken image, a first tracking area specified on the basis of a result of recognition in said recognizing step; and

executing second specified area tracking processing for tracking a second specified area specified on the basis of a result of said first specified area tracking processing.

13. An image processing apparatus for recognizing, from a taken image, an object corresponding to a registered image registered in advance, comprising:

an image taker configured to take an image of a subject to obtain said taken image corresponding to said subject;

a recognizer configured to recognize, from said taken image, an object corresponding to said registered image;

two specified area trackers configured to execute a first specified area tracking processing for tracking, in said taken image, a first specified area specified on the basis of a result of recognition by said recognizer and second specified area tracking processing for tracking, in said taken image, a second specified area specified on the basis of a result of said first specified area tracking processing,

wherein said two specified area trackers alternately execute said second specified area tracking processing with one of said two specified area trackers starting said first specified area tracking processing while the other is executing said second specified area tracking processing.

14. The image processing apparatus according to claim 13, wherein one of said two specified area trackers executes said first specified area tracking processing in each taken image taken in a period of time ranging from the taking of a taken image for use in the recognition by said recognizer to the end of said first specified area tracking processing and executes said second specified area tracking processing in each taken image taken in a period of time ranging from the end of said first specified area tracking processing to the end of said first specified area tracking processing executed by the other specified area tracker.

15. An image processing method for an image processing apparatus for recognizing, from a taken image, an object corresponding to a registered image registered in advance, comprising the steps of:

taking an image of a subject to obtain said taken image corresponding to said subject;

recognizing, from said taken image, an object corresponding to said registered image;

executing, by two specified area trackers, a first specified area tracking processing for tracking, in said taken image, a first specified area specified on the basis of a result of recognition by said recognizer and second specified area tracking processing for tracking, in said taken image, a second specified area specified on the basis of a result of said first specified area tracking processing,

wherein, while one of first specified area tracking processing for tracking, in said taken image, a first specified area specified on the basis of a result of said recognition and second specified area tracking processing for tracking, in said taken image, a second specified area on the basis of a result of said first specified area tracking processing is executing said second specified area tracking processing, the other starts said first specified area tracking processing, thereby alternately executing said second specified area tracking processing.

16. A program configured to make a computer execute recognition processing for recognizing, from a taken image, an object corresponding to a registered image registered in advance, comprising the steps of:

taking an image of a subject to obtain said taken image corresponding to said subject;

recognizing, from said taken image, an object corresponding to said registered image;

executing, by two specified area trackers, a first specified area tracking processing for tracking, in said taken image, a first specified area specified on the basis of a result of recognition by said recognizer and second specified area tracking processing for tracking, in said taken image, a second specified area specified on the basis of a result of said first specified area tracking processing,

wherein, while one of first specified area tracking processing for tracking, in said taken image, a first specified area specified on the basis of a result of said recognition and second specified area tracking processing for tracking, in said taken image, a second specified area on the basis of a result of said first specified area tracking processing is executing said second specified area tracking processing, the other starts said first specified area tracking processing, thereby alternately executing said second specified area tracking processing.

17. An image processing apparatus for recognizing an object corresponding to a registered image from a taken image, comprising:

image taking means configured to take an image of a subject to obtain said taken image of said subject;

recognizing means configured to recognize, from said taken image, an object corresponding to said registered image;

first specified area tracking means configured to execute first specified area tracking processing for tracking, in said taken image, a first tracking area specified on the basis of a result of recognition by said recognizing means; and

second specified area tracking means configured to execute second specified area tracking processing for tracking a second specified area specified on the basis of a result of said first specified area tracking processing.