Apparatus and method for image processing, image processing program and image processor

• US 8,005,295 B2
• Filed: 02/28/2008
• Issued: 08/23/2011
• Est. Priority Date: 03/06/2007
• Status: Expired due to Fees

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Nobori, Kunio
• Primary Examiner(s): Ahmed; Samir
• Assistant Examiner(s): Bayat; Ali
• Application Number:
  US12/278,845
• Publication Number:
  US 20100166299A1
• Time in Patent Office: 1,272 Days
• Field of Search: 382/162, 382/167, 382/141, 382/147, 382/151, 382/201, 382/169, 382/266, 382/267, 382/268, 382/274, 348/87, 348/95, 348/126, 348/349, 348/453, 358/1.9, 358/3.23, 358/518, 345/589, 345/590, 345/604, 345/605, 345/690, 345/204, 345/594
• US Class Current: 382/162
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Corresponding points or a motion vector is generated with reduced positional error even if incoming images have been shot under different illumination conditions.

An image processing apparatus (100, 120) includes an initial corresponding point computing section (113) for computing and outputting multiple sets of corresponding points across multiple images and a corresponding point recomputing section (104), which selects reference corresponding points, consisting of multiple sets of initial corresponding points with small error, from the multiple sets of initial corresponding points by using a photometric constraint equation and a geometric constraint equation, newly computes aimed corresponding points associated with those reference corresponding points, and then outputs the reference corresponding points and the aimed corresponding points as corresponding points.

Claims

1. An image processing apparatus comprising:

an initial corresponding point computing section for computing, based on image data representing a number N of images, n sets of initial corresponding points across the N images, where N and n are both integers, N≧

4 and n≧

4;

a reference candidate generating section for selecting reference corresponding point candidates from the n sets of initial corresponding points to make M groups of reference corresponding point candidates such that each of the M groups is made up of s sets of initial corresponding points, where M and s are both integers, M≧

2 and 4≦

s ≦

n;

a coefficient calculating section for calculating photometric coefficient as a coefficient for a photometric constraint equation such that in each of the M groups of reference corresponding point candidates, luminance values of respective points on the images satisfy the photometric constraint equation;

a new candidate generating section for generating multiple sets of aimed corresponding point candidates, each said set being comprised of new corresponding points that are associated with each other between the images, such that each of the M groups of reference corresponding point candidates satisfies a geometric constraint equation that is defined for that group between the images;

a first selecting section for estimating, based on the luminance values of the respective points on the images and the photometric coefficient calculated, how much those sets of aimed corresponding point candidates satisfy the photometric constraint equation and selecting a single set of aimed corresponding point candidates based on a result of the estimation and outputting the set as aimed corresponding points; and

a second selecting section for evaluating the M groups of reference corresponding point candidates based on the result of the estimation obtained by the first selecting section, selecting a single set of reference corresponding point candidates from the M groups based on a result of the evaluation, and outputting the selected single set of reference corresponding point candidates and aimed corresponding points associated with the selected candidates as corresponding points across the N images.

2. The image processing apparatus of claim 1, wherein the reference candidate generating section estimates the degree of independence of an N×

s luminance matrix, comprised of the luminance values of the selected s sets of initial corresponding points on the N images, and selects the M groups of reference corresponding point candidates based on a result of the estimation.

3. The image processing apparatus of claim 2, wherein the reference candidate generating section uses the ratio of a third eigenvalue to a first eigenvalue of the N×

s luminance matrix as an estimated value and starts selecting the reference corresponding point candidates sequentially from a group of s initial corresponding points that is associated with an N×

s luminance matrix that has produced the greatest estimated value.

4. The image processing apparatus of claim 2, wherein if the ratio of a third eigenvalue to a first eigenvalue of the N×

s luminance matrix is smaller than a predefined threshold value, the reference candidate generating section excludes the s sets of initial corresponding points, associated with the N×

s luminance matrix, from the reference corresponding point candidates.

5. The image processing apparatus of claim 1, wherein the reference candidate generating section estimates the magnitudes of spatial gradients of the luminance values in the vicinity of the points on the respective images with respect to the selected s sets of initial corresponding points and selects the s sets of initial corresponding points with the greatest spatial gradient preferentially as the reference corresponding point candidates.

6. The image processing apparatus of claim 1, wherein the N images are color pictures, and wherein the reference candidate generating section estimates a degree of correlation between the color pictures in the vicinity of the points on the respective images with respect to the selected s sets of initial corresponding points and selects the s sets of initial corresponding points with the lowest degree of correlation preferentially as the reference corresponding point candidates.

7. The image processing apparatus of claim 1, wherein the coefficient calculating section applies a spatial low pass filter, of which the cutoff frequency is determined by either a point-to-point distance between the points on the respective images or a zoom ratio of a subject on the respective images, to the image data and uses resultant luminance values as those of mutually associated points on the respective images.

8. The image processing apparatus of claim 1, wherein the coefficient calculating section calculates the photometric coefficient by replacing the luminance value of a point representing each said reference corresponding point candidate on its associated image with that of at least one point located in the vicinity of the former point.

9. The image processing apparatus of claim 8, wherein the coefficient calculating section calculates the photometric coefficient by using the luminance value of at least one point, which is located in proximity of said point and of which the luminance value is different from that of said point by no more than a predetermined magnitude.

10. The image processing apparatus of claim 8, wherein if the difference between the luminance value of said point and that of a proximate point located around said point exceeds a predetermined range, the coefficient calculating section calculates the photometric coefficient using the luminance value of at least one point that is located in the vicinity of said point.

11. The image processing apparatus of claim 1, further comprising:

a coordinate transformation section for generating a set of coordinate transformed images by magnifying at least one of the N images at a predetermined zoom ratio and by mapping, as pixel values of the magnified image, respective pixel values of at least k images (where k is an integer and 3≦

k≦

N) among the N images according to the predetermined zoom ratio; and

an image generating section, which selects k coordinate transformed images from those coordinate transformed images, calculates a transform coefficient as a coefficient for the photometric constraint equation such that the luminance values of points on the respective coordinate transformed images, which are associated with each other between the k coordinate transformed images, satisfy the photometric constraint equation, and linearly combines the N coordinate transformed images with the transform coefficient, thereby generating a synthetic image.

12. The image processing apparatus of claim 11, wherein the image generating section selects the k coordinate transformed images according to the magnitude of a subject area ratio, which is defined as the ratio of the area of a subject in any of the coordinate transformed images to that of the same subject in its original one yet to be transformed.

13. An image processing method comprising the steps of:

using a processor, computing, based on image data representing a number N of images, n sets of initial corresponding points across the N images, where N and n are both integers, N≧

4 and n≧

4;

selecting s sets of initial corresponding points from the n sets of initial corresponding points, where s is an integer and 4≦

s≦

n;

making M groups of reference corresponding point candidates such that each of the M groups is made up of s sets of initial corresponding points, where M is an integer and M≧

2;

calculating a photometric coefficient as a coefficient for a photometric constraint equation such that in each of the M groups of reference corresponding point candidates, luminance values of respective points on the images satisfy the photometric constraint equation;

generating multiple sets of aimed corresponding point candidates, each said set being comprised of new corresponding points that are associated with each other between the images, such that each of the M groups of reference corresponding point candidates satisfies a geometric constraint equation that is defined for that group between the images;

estimating, based on the luminance values of the respective points on the images and the photometric coefficient calculated, how much those sets of aimed corresponding point candidates satisfy the photometric constraint equation;

selecting a single set of aimed corresponding point candidates based on a result of the estimation and outputting the set as aimed corresponding points;

evaluating the M groups of reference corresponding point candidates based on the result of the estimation, and selecting a single set of reference corresponding point candidates from the M groups based on a result of the evaluation; and

outputting the selected single set of reference corresponding point candidates and aimed corresponding points associated with the selected candidates as corresponding points across the N images.

14. A non-transitory storage medium on which a computer program is stored, the computer program being defined so as to make a computer perform the steps of:

computing, based on image data representing a number N of images, n sets of initial corresponding points across the N images, where N and n are both integers, N≧

4 and n≧

4;

making M groups of reference corresponding point candidates such that each of the M groups is made up of s sets of initial corresponding points across the N images, where M and s are both integers, M≧

2 and 4≦

s≦

n;

calculating coefficient as a coefficient for a photometric constraint equation such that in each of the M groups of reference corresponding point candidates, luminance values of respective points on the images satisfy the photometric constraint equation;

generating multiple sets of aimed corresponding point candidates, each said set being comprised of new corresponding points that are associated with each other between the images, such that each of the M groups of reference corresponding point candidates satisfies a geometric constraint equation that is defined for that group between the images;

estimating, based on the luminance values of the respective points on the images and the photometric coefficient calculated, how much those sets of aimed corresponding point candidates satisfy the photometric constraint equation;

selecting a single set of aimed corresponding point candidates based on a result of the estimation and outputting the set as aimed corresponding points;

evaluating the M groups of reference corresponding point candidates based on the result of the estimation, and selecting a single set of reference corresponding point candidates from the M groups based on a result of the evaluation; and

outputting the selected single set of reference corresponding point candidates and aimed corresponding points associated with the selected candidates as corresponding points across the N images.

15. A processor having the ability to execute a computer program, wherein by executing an image processing program, the processor performs the steps of:

the processor computing, based on image data representing a number N of images, n sets of initial corresponding points across the N images, where N and n are both integers, N≧

4 and n≧

4;

making M groups of reference corresponding point candidates such that each of the M groups is made up of s sets of initial corresponding points across the N images, where M is an integer and M≧

2;

calculating a photometric coefficient as a coefficient for a photometric constraint equation such that in each of the M groups of reference corresponding point candidates, luminance values of respective points on the images, which are associated with each other between the images, satisfy the photometric constraint equation;

generating multiple sets of aimed corresponding point candidates, each said set being comprised of new corresponding points that are associated with each other between the images, such that each of the M groups of reference corresponding point candidates satisfies a geometric constraint equation that is defined for that group between the images;

estimating, based on the luminance values of the respective points on the images, which are associated with each other between the images, and the photometric coefficient calculated, how much those sets of aimed corresponding point candidates satisfy the photometric constraint equation;

selecting a single set of aimed corresponding point candidates based on a result of the estimation and outputting the set as aimed corresponding points;

evaluating the M groups of reference corresponding point candidates based on the result of the estimation, and selecting a single set of reference corresponding point candidates from the M groups based on a result of the evaluation; and

outputting the selected single set of reference corresponding point candidates and aimed corresponding points associated with the selected candidates as corresponding points across the N images.

16. An image processing apparatus comprising:

an initial corresponding point computing section for computing, based on image data representing a number N of images, n sets of initial corresponding points across the N images, where N and n are both integers, N≧

4 and n≧

4; and

a corresponding point recomputing section, which selects s sets (where s is an integer and 4≦

s≦

n) of initial corresponding points, which satisfy a geometric constraint equation and a photometric constraint equation, from the n sets of initial corresponding points and outputs the s sets as reference corresponding points and which computes, based on the reference corresponding points, aimed corresponding points that satisfy the geometric and photometric constraint equations and outputs the aimed corresponding points.