Motion and disparity estimation method, image synthesis method, and apparatus for implementing same methods

• US 5,768,404 A
• Filed: 04/13/1995
• Issued: 06/16/1998
• Est. Priority Date: 04/13/1994
• Status: Expired due to Term

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Azuma, Takeo, Morimura, Atsushi
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Johnson, Timothy M.
• Application Number:
  US08/421,324
• Time in Patent Office: 1,160 Days
• Field of Search: 382/107, 382/154, 382/199, 382/294, 382/295, 356/12, 356/13, 395/121, 395/122, 348/218, 348/699
• US Class Current: 382/107
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/238 using non-full search, e.g. three-step search
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

To achieve this, the invention has a base image frame memory for storing a base image, a reference image frame memory for storing a reference image, a block correlation computing circuit for computing block correlations and confidence measure for estimation by using a plurality of block sizes, a representative pixel correlation computing circuit, and an estimation integrating computing circuit for evaluating the reliability of the result of estimation by block matching on the basis of a luminance gradient, image noise, a minimum value of an evaluation yardstick for differences between blocks, and block size, and for integrating the results of estimation obtained with the plurality of block sizes.

Claims

1. A disparity estimation apparatus, comprising:

correspondence determining means, for determining correspondence between a first image and a second image;

correspondence evaluation means for evaluating a confidence of correspondence determined by the correspondence determining means to produce evaluation data;

disparity data computing means for computing disparity data for the first image and the second image based upon correspondence determined by the correspondence determining means;

occlusion judging means for judging an occlusion portion of the second image which is not included in the first image, based upon the disparity data and the evaluation data;

first edge extracting means, for extracting edges of the evaluation data;

second edge extracting means, for extracting edges of the first and second images;

first selecting means for selecting at least one edge of the evaluation data which corresponds to a non-occlusion portion of the first image;

integrating means for integrating at least one edge of the first image with the selected at least one edge of the evaluation data to produce at least one non-occlusion disparity edge;

correspondence converting means, for obtaining the vicinity in the second image corresponding to the non-occlusion disparity edge produced by the integrating means, based upon the disparity data;

second selecting means, for selecting at least one edge of the second image from the occlusion portion of the second image;

third selecting means, for selecting the selected at least one edge of the second image and the vicinity produced by the correspondence converting means, to produce at least one occlusion disparity edge;

disparity supplementing means for supplementing disparity data for the occlusion portion of the second image, based upon the at least one occlusion disparity edge and the disparity data;

disparity extrapolating means, for extrapolating disparity data for the non-occlusion portion of the first image based upon the non-occlusion disparity edge and the disparity data; and

disparity integrating means, for integrating the disparity data, the supplemented disparity data, and the extrapolated disparity data.

2. A disparity estimation apparatus as recited in claim 1, further including noise suppressing means for suppressing noise in the disparity data.

3. A disparity estimation apparatus as recited in claim 1, further including evaluation data storing means for storing evaluation data produced by the correspondence evaluation means.

4. A disparity estimation apparatus as recited in claim 1, further including occlusion data storing means for storing occlusion data corresponding to occlusion portions judged by the occlusion judging means.

5. A disparity estimation apparatus as recited in claim 1,

wherein the correspondence determining means determines correspondence between at least one correspondence region of the second image and the first image;

and

further including region shifting means,

for shifting the at least one correspondence region when the at least one correspondence region includes a portion of the second image for which corresponding evaluation data does not meet a second predetermined threshold, and

for providing the shifted at least one correspondence region to the correspondence determining means for determining correspondence between the shifted at least one correspondence region of the second image and the first image.

6. An image synthesizing apparatus, comprising:

the disparity estimation apparatus as recited in claim 1;

disparity distance converting means, for converting the integrated disparity data into distance data;

image generating means for generating hidden image data for a hidden portion of the second image based upon the occlusion portion, the integrated disparity data, and the image data; and

image synthesizing means for synthesizing a third image from the image data and the hidden image data.

7. An image synthesizing apparatus as recited in claim 6, wherein the image generating means

detects which side of the occlusion portion has a greater disparity;

determines the hidden portion to be on the detected side of the occlusion portion; and

employs image data for the occlusion portion and image data for a side of the occlusion portion in the second image opposite the selected side as the hidden image data.

8. An image synthesizing apparatus as recited in claim 6, wherein the image generating means

detects which side of the occlusion portion has a greater disparity;

determines the hidden portion to be on the detected side of the occlusion portion; and

employs image data for a side of the occlusion portion in the second image opposite the selected side as the hidden image data.

9. An image synthesizing apparatus as recited in claim 6, further including noise suppressing means for suppressing noise in the disparity data.

10. An image synthesizing apparatus as recited in claim 6, further including evaluation data storing means for storing evaluation data produced by the correspondence evaluation means.

11. An image synthesizing apparatus as recited in claim 6, further including occlusion data storing means for storing occlusion data corresponding to occlusion portions judged by the occlusion judging means.

12. An image synthesizing apparatus as recited in claim 6,

wherein the correspondence determining means determines correspondence between at least one correspondence region of the second image and the first image;

and

further including region shifting means,

for shifting the at least one correspondence region when the at least one correspondence region includes a portion of the second image for which corresponding evaluation data does not meet a second predetermined threshold, and

for providing the shifted at least one correspondence region to the correspondence determining means for determining correspondence between the shifted at least one correspondence region of the second image and the first image.

13. An image synthesizing apparatus, comprising:

the disparity estimation apparatus as recited in claim 1;

image shifting means for shifting at least the second image according to a designated viewing direction;

image dropout detection means, for detecting no-image data regions which occur in at least the second image as a result of the image shift;

image data extrapolating means, for extrapolating image data for the no-image data regions based upon the image data; and

image synthesizing means for synthesizing a third image from the image data and the extrapolated image data.

14. An image synthesizing apparatus as recited in claim 13, wherein the image generating means

detects which side of the occlusion portion has a greater disparity;

determines the hidden portion to be on the detected side of the occlusion portion; and

employs image data for the occlusion portion and image data for a side of the occlusion portion in the second image opposite the selected side as the hidden image data.

15. An image synthesizing apparatus as recited in claim 13, wherein the image generating means

detects which side of the occlusion portion has a greater disparity;

determines the hidden portion to be on the detected side of the occlusion portion; and

employs image data for a side of the occlusion portion in the second image opposite the selected side as the hidden image data.

16. An image synthesizing apparatus as recited in claim 13, further including noise suppressing means for suppressing noise in the disparity data.

17. An image synthesizing apparatus as recited in claim 13, further including evaluation data storing means for storing evaluation data produced by the correspondence evaluation means.

18. An image synthesizing apparatus as recited in claim 13, further including occlusion data storing means for storing occlusion data corresponding to occlusion portions judged by the occlusion judging means.

19. An image synthesizing apparatus as recited in claim 13,

wherein the correspondence determining means determines correspondence between at least one correspondence region of the second image and the first image;

and

further including region shifting means,

for shifting the at least one correspondence region when the at least one correspondence region includes a portion of the second image for which corresponding evaluation data does not meet a second predetermined threshold, and

for providing the shifted at least one correspondence region to the correspondence determining means for determining correspondence between the shifted at least one correspondence region of the second image and the first image.

20. A method of estimating disparity, comprising:

determining correspondence between a first image and a second image;

evaluating a confidence of the determined correspondence to produce evaluation data;

computing disparity data for the first image and the second image based upon the determined correspondence;

judging an occlusion portion of the second image which is not included in the first image, based upon the disparity data and the evaluation data;

extracting edges of the evaluation data;

extracting edges of the first and second images;

selecting at least one edge of the evaluation data which corresponds to a non-occlusion portion of the first image;

integrating at least one edge of the first image with the selected at least one edge of the evaluation data to produce at least one non-occlusion disparity edge;

obtaining the vicinity in the second image corresponding to the non-occlusion disparity edge produced by the integrating means, based upon the disparity data;

selecting at least one edge of the second image from the occlusion portion of the second image;

selecting the selected at least one edge of the second image and the vicinity produced by the correspondence converting means, to produce at least one occlusion disparity edge;

supplementing disparity data for the occlusion portion of the second image, based upon the at least one occlusion disparity edge and the disparity data;

extrapolating disparity data for the non-occlusion portion of the first image based upon the non-occlusion disparity edge and the disparity data; and

integrating the disparity data, the supplemented disparity data, and the extrapolated disparity data.

21. A method of estimating disparity, as recited in claim 20, wherein the integration of at least one edge of the first image with the selected at least one edge of the evaluation data to produce at least one non-occlusion disparity edge includes:

obtaining an absolute value of an edge component of the at least one edge of the first image;

obtaining an absolute value of an edge component of the selected at least one edge of the evaluation data; and

averaging the absolute value of the edge component of the at least one edge of the first image with the absolute value of the edge component of the selected at least one edge of the evaluation data.

22. A method of estimating disparity, as recited in claim 20, wherein the supplementing of disparity data for the occlusion portion of the second image includes:

determining which portion of the second image adjacent the occlusion portion has the smallest disparity; and

employing the disparity data for the determined portion of the second image adjacent the occlusion portion to supplement the disparity data for the occlusion portion, the disparity data being supplemented in a direction from the determined portion of the second image to the at least one occlusion disparity edge.

23. A method of estimating disparity, as recited in claim 20, wherein the extrapolation of disparity data for the non-occlusion portion of the first image includes:

selecting a first point in the first image having a stable disparity and being positioned on a first side of the non-occlusion disparity edge;

extrapolating disparity data from the first point to the non-occlusion disparity edge;

selecting a second point in the first image having a stable disparity and being positioned on a second side of the non-occlusion disparity edge opposite the first side; and

extrapolating disparity data from the second point to the non-occlusion disparity edge.

24. A method of estimating disparity, as recited in claim 20, further including filtering noise from the disparity data.

25. A method of estimating disparity, as recited in claim 20, wherein the integration of the disparity data, the supplemented disparity data, and the extrapolated disparity data includes the steps of:

first, adopting disparity data that is neither for a region of the non-occlusion disparity edge nor in a region of the occlusion disparity edge;

second, adopting disparity data for the region of the non-occlusion disparity edge; and

third, adopting disparity data for the region of the occlusion disparity edge.

26. A method of estimating disparity as recited in claim 20, wherein the determination of correspondence between the first image and the second image includes:

determining correspondence between at least one correspondence region of the second image and the first image;

shifting the at least one correspondence region when the at least one correspondence region includes a portion of the second image for which corresponding evaluation data does not meet a second predetermined threshold, and

determining correspondence between the shifted at least one correspondence region of the second image and the first image.

27. A method of synthesizing an image, comprising:

estimating disparity for a first image and a second image as recited in claim 20, to obtain integrated disparity data;

converting the integrated disparity data into distance data;

generating hidden image data for a hidden portion of the second image based upon the occlusion portion, the integrated disparity data, and the image data; and

synthesizing a third image from the image data and the hidden image data.

28. A method of synthesizing an image, as recited in claim 27, wherein the integration of at least one edge of the first image with the selected at least one edge of the evaluation data to produce at least one non-occlusion disparity edge includes:

obtaining an absolute value of an edge component of the at least one edge of the first image;

obtaining an absolute value of an edge component of the selected at least one edge of the evaluation data; and

averaging the absolute value of the edge component of the at least one edge of the first image with the absolute value of the edge component of the selected at least one edge of the evaluation data.

29. A method of synthesizing an image, as recited in claim 27, wherein the supplementing of disparity data for the occlusion portion of the second image includes:

determining which portion of the second image adjacent the occlusion portion has the smallest disparity; and

employing the disparity data for the determined portion of the second image adjacent the occlusion portion to supplement the disparity data for the occlusion portion, the disparity data being supplemented in a direction from the determined portion of the second image to the at least one occlusion disparity edge.

30. A method of synthesizing an image, as recited in claim 27, wherein the extrapolation of disparity data for the non-occlusion portion of the first image includes:

selecting a first point in the first image having a stable disparity and being positioned on a first side of the non-occlusion disparity edge;

extrapolating disparity data from the first point to the non-occlusion disparity edge;

selecting a second point in the first image having a stable disparity and being positioned on a second side of the non-occlusion disparity edge opposite the first side; and

extrapolating disparity data from the second point to the non-occlusion disparity edge.

31. A method of synthesizing an image, as recited in claim 27, further including filtering noise from the disparity data.

32. A method of synthesizing an image, as recited in claim 27, wherein the integration of the disparity data, the supplemented disparity data, and the extrapolated disparity data includes the steps of:

first, adopting disparity data that is neither for a region of the non-occlusion disparity edge nor in a region of the occlusion disparity edge;

second, adopting disparity data for the region of the non-occlusion disparity edge; and

third, adopting disparity data for the region of the occlusion disparity edge.

33. A method of synthesizing an image as recited in claim 27, wherein the determination of correspondence between the first image and the second image includes:

determining correspondence between at least one correspondence region of the second image and the first image;

shifting the at least one correspondence region when the at least one correspondence region includes a portion of the second image for which corresponding evaluation data does not meet a second predetermined threshold, and

determining correspondence between the shifted at least one correspondence region of the second image and the first image.

34. A method of synthesizing an image as recited in claim 27, further including supplementing disparity data for the uncertain portions of the first and second images.

35. A method of synthesizing an image as recited in claim 27, wherein the generation of hidden image data for a hidden portion of the second image includes:

detecting which side of the occlusion portion has a greater disparity;

determining the hidden portion to be on the detected side of the occlusion portion; and

employing image data for the occlusion portion and image data for a side of the occlusion portion in the second image opposite the selected side as the hidden image data.

36. A method of synthesizing an image as recited in claim 27, wherein the generation of hidden image data for a hidden portion of the second image includes:

detecting which side of the occlusion portion has a greater disparity;

determining the hidden portion to be on the detected side of the occlusion portion; and

employing image data for a side of the occlusion portion in the second image opposite the selected side as the hidden image data.

37. A method of synthesizing an image, comprising:

estimating disparity for a first image and a second image as recited in claim 20, to obtain integrated disparity data;

shifting at least the second image according to a designated viewing direction;

detecting no-image data regions which occur in at least the second image as a result of the image shift;

extrapolating image data for the no-image data regions based upon the image data; and

synthesizing a third image from the image data and the extrapolated image data.

38. A method of synthesizing an image as recited in claim 37, further including replacing suppressed disparity data with a median value of neighboring disparity data or a mean weighted value of neighboring disparity data.

39. A method of synthesizing an image as recited in claim 37, wherein the extrapolation of image data for the no-image data regions includes extrapolating data for a no-image data region in a first direction from a start of the no-image data region by employing image data obtained in a second direction, opposite the first direction, from the start of the no-image data region.

40. A method of synthesizing an image as recited in claim 37, wherein the integration of at least one edge of the first image with the selected at least one edge of the evaluation data to produce at least one non-occlusion disparity edge includes:

obtaining an absolute value of an edge component of the at least one edge of the first image;

obtaining an absolute value of an edge component of the selected at least one edge of the evaluation data; and

averaging the absolute value of the edge component of the at least one edge of the first image with the absolute value of the edge component of the selected at least one edge of the evaluation data.

41. A method of synthesizing an image, as recited in claim 37, wherein the supplementing of disparity data for the occlusion portion of the second image includes:

determining which portion of the second image adjacent the occlusion portion has the smallest disparity; and

employing the disparity data for the determined portion of the second image adjacent the occlusion portion to supplement the disparity data for the occlusion portion, the disparity data being supplemented in a direction from the determined portion of the second image to the at least one occlusion disparity edge.

42. A method of synthesizing an image, as recited in claim 37, wherein the extrapolation of disparity data for the non-occlusion portion of the first image includes:

selecting a first point in the first image having a stable disparity and being positioned on a first side of the non-occlusion disparity edge;

extrapolating disparity data from the first point to the non-occlusion disparity edge;

selecting a second point in the first image having a stable disparity and being positioned on a second side of the non-occlusion disparity edge opposite the first side; and

extrapolating disparity data from the second point to the non-occlusion disparity edge.

43. A method of synthesizing an image, as recited in claim 37, further including filtering noise from the disparity data.

44. A method of synthesizing an image, as recited in claim 37, wherein the integration of the disparity data, the supplemented disparity data, and the extrapolated disparity data includes the steps of:

first, adopting disparity data that is neither for a region of the non-occlusion disparity edge nor in a region of the occlusion disparity edge;

second, adopting disparity data for the region of the non-occlusion disparity edge; and

third, adopting disparity data for the region of the occlusion disparity edge.

45. A method of synthesizing an image as recited in claim 37, wherein the determination of correspondence between the first image and the second image includes:

determining correspondence between at least one correspondence region of the second image and the first image;

shifting the at least one correspondence region when the at least one correspondence region includes a portion of the second image for which corresponding evaluation data does not meet a second predetermined threshold, and

determining correspondence between the shifted at least one correspondence region of the second image and the first image.

46. A method of synthesizing an image as recited in claim 37, further including supplementing disparity data for the uncertain portions of the first and second images.