Image processing method and apparatus

• US 5,680,150 A
• Filed: 11/23/1994
• Issued: 10/21/1997
• Est. Priority Date: 11/29/1993
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Urushiya, Hiroyuki, Shimizu, Yukihiko, Okutomi, Masatoshi, Yoshizaki, Osamu
• Primary Examiner(s): Powell, Mark R.
• Assistant Examiner(s): Kovalick, Vincent E.
• Application Number:
  US08/347,259
• Time in Patent Office: 1,063 Days
• Field of Search: 395/100, 395/133, 395/135, 395/138, 382/9, 382/22, 382/25, 382/26, 382/27, 382/41, 382/42, 382/44, 382/45, 382/46, 382/47, 382/48, 358/537, 358/538, 358/539, 358/540, 364/413.19, 364/413.13, 345/115, 357/537, 357/538
• US Class Current: 345/634
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 3/153 using elastic snapping
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/337 involving reference images or patches

Abstract

In an image processing apparatus, a plurality of substantially corresponding points are designated on a reference image and a peripheral image. The degree of coincidence is detected between a window in the peripheral image with a corresponding point as the center, and windows successively cut from the reference image, whose center is a corresponding point. The corresponding point at the center of the window having the highest degree of coincidence in the reference image is moved and an offset between this corresponding point and the corresponding point on the peripheral image is detected. When a corresponding point whose offset is greater than a threshold exists, the peripheral image is subjected to an affine transformation using a corresponding point on the reference image whose offset is less than the threshold. The processing is repeated by using the corresponding point on the transformed peripheral image. The two images are merged by an affine transformation when the offsets of all corresponding points become smaller than the threshold.

Claims

1. An image processing apparatus comprising:

display means for displaying first and second images;

designating means for designating a first point on the first image and a second on the second image, the second point corresponding to the first point;

acquiring means for acquiring a peripheral image of the first point in the first image, which has a first predetermined size;

determination means for determining a search zone in the second image, which includes the second point and has a second predetermined size;

searching means for searching an area having a maximum degree of coincidence with the peripheral image from the search zone;

correcting means for correcting a position of the second point based on the searched area; and

merging means for merging the first and second images on the basis of the first point and the second point corrected by said correcting means.

2. The apparatus according to claim 1, wherein said designating means designates a plurality of pairs of corresponding points, each of the plurality of pairs consisting of a first point on the first image and a second point on the second image and said merging means merges the first and second images on the basis of the plurality of first points located on the first image and the plurality of second points located on the second image, which are corrected by said correcting means.

3. The apparatus according to claim 2, wherein said merging means merges the first and second images using an affine transformation.

4. The apparatus according to claim 1, wherein said correcting means moves the second point located on the second image within the search zone inclusive of the second point on the basis of the peripheral image of the first point, which is located on the first image.

5. The apparatus according to claim 4, wherein said searching means:

acquires a plurality of partial images, which have a size identical with that of the peripheral image, within said search zone; and

extracts a partial image having a maximum degree of coincidence with the peripheral image from among the plurality of partial images.

6. The apparatus according to claim 5, wherein the degree of coincidence in said searching means is acquired on the basis of degree of correlation between the peripheral image and each of the partial images.

7. An image processing apparatus comprising:

display means for displaying first and second images;

designating means for designating a plurality of pairs of corresponding points, each of the plurality of pairs of corresponding points consisting of a first point on the first image and a second point on the second image;

acquiring means for acquiring, regarding each of the first points, a peripheral image which has a first predetermined size;

determination means for determining, regarding each of the second points, a search zone which includes the second point and having a second predetermined size;

searching means for searching an area having a maximum degree of coincidence with the peripheral image from the search zone;

correcting means for correcting a position of the second point based on the searched area; and

merging means for merging the first and second image based on the first points and the second points corrected by said correction means.

8. The apparatus according to claim 7, wherein the degree of coincidence in said searching means is acquired on the basis of a degree of correlation between the peripheral image and each of partial images which has a size identical with that of the peripheral image, within the search zone.

9. The apparatus according to claim 8, wherein the first and second images are color images and the degree of coincidence between the partial image and a peripheral image is an average of degrees of coincidence acquired for each color component of the color image.

10. The apparatus according to claim 7, further comprising correcting means which, in a case where the plurality of pairs of corresponding points includes a pair for which the distance acquired by said acquisition means is greater than the predetermined value, is for correcting the position of the corresponding point, which is located on the first image, in such a manner that said distance becomes smaller than the predetermined value;

wherein operations performed by said moving means, said acquisition means and said merging means are re-executed using the corresponding point, which is located on the first image, corrected by said correcting means and the corresponding point designated on the second image.

11. The apparatus according to claim 10, wherein said correcting means includes:

second transforming means which, in a case where the plurality of pairs of corresponding points includes a pair for which the distance acquired by said acquisition means is greater than the predetermined value, is for obtaining a second transformed image by transforming the second image using a pair of corresponding points for which the distance acquired by said acquisition means is smaller than the predetermined value; and

altering means for altering the position of the corresponding point located on the first image on the basis of the position of each corresponding point located on the second transformed image.

12. The apparatus according to claim 11, wherein said merging means merges the first and second images on the basis of finally obtained results of transformation by said second transformation means.

13. The apparatus according to claim 11, wherein said altering means moves the position of the corresponding point, which is located on the first image, of a pair of corresponding points among said plurality of pairs for which the distance acquired by said acquisition means is greater than the predetermined value, to position indicating a corresponding point, which is located on the second transformed image, forming the pair with the corresponding point located on the first image.

14. The apparatus according to claim 11, wherein said altering means moves all positions of corresponding points located on the first image to positions indicating corresponding points on the second transformed image.

15. The apparatus according to claim 11, wherein size of the prescribed range in said moving means is reduced whenever moving by said moving means is re-executed after operations of said second transformation means and altering means.

16. The apparatus according to claim 10, further comprising notification means which, in a case where the number of pairs of corresponding points for which the distances are smaller than the predetermined value is insufficient for executing the affine transformation in said second transformation means, gives notification to this effect.

17. The apparatus according to claim 10, wherein in a case where the number of pairs of corresponding points for which the distances are smaller than the predetermined value is insufficient for executing the affine transformation in said second transformation means, selection of a pair of corresponding points used in the affine transformation is performed based upon a value greater than the predetermined value.

18. An image processing method comprising:

a display step of displaying first and second images;

a designating step of designating a first point on the first image and a second point on the second image, the second point corresponding to the first point;

an acquiring step of acquiring a peripheral image of the first point in the first image, which has a first predetermined size;

a determination step of determining a search zone in the second image, which includes the second point and has a second predetermined size;

a searching step of searching an area having a maximum degree of coincidence with the peripheral image from the search zone;

a correcting step of correcting a position of the second point based on said searched area; and

a merging step of merging the first and second images on the basis of the first point and the second point corrected by said correcting means.

19. The method according to claim 18, wherein said designating step designates a plurality of pairs of corresponding points, each of the plurality of pairs consisting of a first point on the first image and a second point on the second image and said merging step merges the first and second images on the basis of the plurality of first points located on the first image and the plurality of second points located on the second image, which are corrected at said correcting step.

20. The method according to claim 19, wherein said merging step merges the first and second images using an affine transformation.

21. The method according to claim 18, wherein said correcting step moves the second point located on the second image within the search zone inclusive of the second point on the basis of the peripheral image of the first point, which is located on the first image.

22. The method according to claim 21, wherein said searching step:

acquires a plurality of partial images, which have a size identical with that of the peripheral image, within said search zone; and

extracts a partial image having a maximum degree of coincidence with the peripheral image from among the plurality of partial image.

23. The method according to claim 22, wherein the degree of coincidence at said searching step is acquired on the basis of degree of correlation between the peripheral image and each of the partial images.

24. An image processing method comprising:

a display step of displaying first and second images;

a designating step of designating a plurality of pairs of corresponding points, each of the plurality of pairs of corresponding points consisting of a first point on the first image and a second point on the second image;

an acquiring step of acquiring, regarding each of the first points, a peripheral image which has a first predetermined size;

a determination step of determining, regarding each of the second points, a search zone which includes the second point and has a second predetermined size;

a searching step of searching an area having a maximum degree of coincidence with the peripheral image from the search zone;

a correcting step of correcting a position of the second point based on the searched area; and

a merging step of merging the first and second image based on the first points and the second points corrected by said correction means.

25. The method according to claim 24, wherein the degree of coincidence at said searching step is acquired on the basis of a degree of correlation between the peripheral image and each of partial images which has a size identical with that of the peripheral image, within the search zone.

26. The method according to claim 25, wherein the first and second images are color images and the degree of coincidence between the partial image and a peripheral image is an average of degrees of coincidence acquired for each color component of the color image.

27. The method according to claim 24, further comprising a correcting step which, in a case where the plurality of pairs of corresponding points includes a pair for which the distance acquired at said acquisition step is greater than the predetermined value, is for correcting the position of the corresponding point, which is located on the first image, in such a manner that said distance becomes smaller than the predetermined value;

wherein said moving step, said acquisition step and said merging step are re-executed using the corresponding point, which is located on the first image, corrected at said correcting step and the corresponding point designated on the second image.

28. The method according to claim 27, wherein said correcting step includes:

a second transforming step which, in a case where the plurality of pairs of corresponding points includes a pair for which the distance acquired by said acquisition means is greater than the predetermined value, is for obtaining a second transformed image by transforming the second image using a pair of corresponding points for which the distance acquired at said acquisition step is smaller than the predetermined value; and

an altering step of altering the position of the corresponding point located on the first image on the basis of the position of each corresponding point located on the second transformed image.

29. The method according to claim 28, wherein said merging step merges the first and second images on the basis of finally obtained results of transformation at said second transformation step.

30. The method according to claim 28, wherein said altering step moves the position of the corresponding point, which is located on the first image, of a pair of corresponding points among said plurality of pairs for which the distance acquired at said acquisition step is greater than the predetermined value, to position indicating a corresponding point, which is located on the second transformed image, forming the pair with the corresponding point located on the first image.

31. The method according to claim 28, wherein said altering step moves all positions of corresponding points located on the first image to positions indicating corresponding points on the second transformed image.

32. The method according to claim 28, wherein size of the prescribed range at said moving step is reduced whenever said moving step is re-executed after operations at said second transformation step and altering step.

33. The method according to claim 27, further comprising a notification step which, in a case where the number of pairs of corresponding points for which the distances are smaller than the predetermined value is insufficient for executing the affine transformation at said second transformation step, gives notification to this effect.

34. The method according to claim 27, wherein in a case where the number of pairs of corresponding points for which the distances are smaller than the predetermined value is insufficient for executing the affine transformation at said second transformation step, selection of a pair of corresponding points used in the affine transformation is performed based upon a value greater than the predetermined value.

35. An image processing apparatus comprising:

display means for displaying first and second images;

designating means for designating a plurality of pairs of corresponding points indicating substantially corresponding positions on the first and second images;

transforming means for obtaining a transformed image by subjecting the second image to an affine transformation on the basis of the plurality of pairs of corresponding points designated by said designating means;

moving means for moving the corresponding points of the pairs located on the first image based upon peripheral images of respective corresponding points, which are located on the transformed image, forming the pairs with the corresponding points located on the first image, and with regard to each of the corresponding points, said moving means acquires a peripheral image of the corresponding point located on the transformed image, acquires a plurality of partial images, which have a size identical with that of the peripheral image, within a prescribed range of the corresponding point, which is located on the first image, forming the pair with the corresponding point located on the transformed image, extracts a partial image having a maximum degree of coincidence with the peripheral image from among the plurality of partial images, and moves the corresponding point, which is located on the first image, inside the partial image extracted, wherein the degree of coincidence in said moving means is acquired on the basis of a degree of correlation between the peripheral image and each of the partial images;

acquisition means for acquiring distances between the corresponding points located on the first image moved by said moving means and the corresponding points, which are located on the transformed image, forming the pairs with the corresponding points located on the first image; and

merging means for merging the first and second image by the affine transformation, which is performed by said transformation means, in a case where all distances acquired by said acquisition means are smaller than a predetermined value,

wherein the first and second images are color images and the degree of coincidence between the partial image and a peripheral image is an average of degrees of coincidence acquired for each color component of the color image.

36. An image processing apparatus comprising:

display means for displaying first and second images;

designating means for designating a plurality of pairs of corresponding points indicating substantially corresponding positions on the first and second images;

transforming means for obtaining a transformed image by subjecting the second image to an affine transformation on the basis of the plurality of pairs of corresponding points designated by said designating means;

moving means for moving the corresponding points of the pairs located on the first image based upon peripheral images of respective corresponding points, which are located on the transformed image, forming the pairs with the corresponding points located on the first image;

acquisition means for acquiring distances between the corresponding points located on the first image moved by said moving means and the corresponding points, which are located on the transformed image, forming the pairs with the corresponding points located on the first image;

merging means for merging the first and second image by the affine transformation, which is performed by said transformation means, in a case where all distances acquired by said acquisition means are smaller than a predetermined value; and

correcting means which, in a case where the plurality of pairs of corresponding points includes a pair for which the distance acquired by said acquisition means is greater than the predetermined value, is for correcting the position of the corresponding point, which is located on the first image, in such a manner that the distance becomes smaller than the predetermined value,

wherein operations performed by said moving means, said acquisition means and said merging means are re-executed using the corresponding point, which is located on the first image, corrected by said correcting means and the corresponding point designated on the second image.

37. An image processing method comprising:

a display step of displaying first and second images;

a designating step of designating a plurality of pairs of corresponding points indicating substantially corresponding positions on the first and second images;

a transforming step of obtaining a transformed image by subjecting the second image to an affine transformation on the basis of the plurality of pairs of corresponding points designated at said designating step;

a moving step of moving the corresponding points of the pairs located on the first image based upon peripheral images of respective corresponding points, which are located on the transformed image, forming the pairs with the corresponding points located on the first image, and with regard to each of the corresponding points, said moving step acquiring a peripheral image of the corresponding point located on the transformed image, acquiring a plurality of partial images, which have a size identical with that of the peripheral image, within a prescribed range of the corresponding point, which is located on the first image, forming the pair with the corresponding point located on the transformed image, extracting a partial image having a maximum degree of coincidence with the peripheral image from among the plurality of partial images, and moving the corresponding point, which is located on the first image, inside the partial image extracted, wherein the degree of coincidence at said moving step is acquired on the basis of degree of correlation between the peripheral image and each of the partial images;

an acquisition step of acquiring distances between the corresponding points located on the first image moved at said moving step and the corresponding points, which are located on the transformed image, forming the pairs with the corresponding points located on the first image; and

a merging step of merging the first and second image by the affine transformation, which is performed at said transformation step, in a case where all distances acquired by said acquisition means are smaller than a predetermined value,

wherein the first and second images are color images and the degree of coincidence between the partial image and a peripheral image is an average of degrees of coincidence acquired for each color component of the color image.

38. An image processing method comprising:

a display step of displaying first and second images;

a designating step of designating a plurality of pairs of corresponding points indicating substantially corresponding positions on the first and a second images;

a transforming step of obtaining a transformed image by subjecting the second image to an affine transformation on the basis of the plurality of pairs of corresponding points designated at said designating step;

a moving step of moving the corresponding points of the pairs located on the first image based upon peripheral images of respective corresponding points, which are located on the transformed image, forming the pairs with the corresponding points located on the first image;

an acquisition step of acquiring distances between the corresponding points located on the first image moved at said moving step and the corresponding points, which are located on the transformed image, forming the pairs with the corresponding points located on the first image;

a merging step of merging the first and second image by the affine transformation, which is performed at said transformation step, in a case where all distances acquired by said acquisition means are smaller than a predetermined value; and

a correcting step which, in a case where the plurality of pairs of corresponding points includes a pair for which the distance acquired at said acquisition step is greater than the predetermined value, is for correcting the position of the corresponding point, which is located on the first image, in such a manner that the distance becomes smaller than the predetermined value,

wherein said moving step, said acquisition step and said merging step are re-executed using the corresponding point, which is located on the first image, corrected at said correcting step and the corresponding point designated on the second image.