Motion and disparity estimation method, image synthesis method, and apparatus for implementing same methods

• US 6,215,899 B1
• Filed: 02/26/1998
• Issued: 04/10/2001
• Est. Priority Date: 04/13/1994
• Status: Expired due to Term

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Azuma, Takeo, Morimura, Atsushi
• Primary Examiner(s): Au, Amelia
• Assistant Examiner(s): Johnson, Timothy M.
• Application Number:
  US09/031,193
• Time in Patent Office: 1,139 Days
• Field of Search: 382/106, 382/154, 348/42, 348/46, 348/47, 348/699, 345/421, 356/12, 396/324, 396/325
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/238 using non-full search, e.g. three-step search
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

An image computing and processing apparatus for computing correspondence between images from a plurality of images and for obtaining disparity corresponding to the depth of an image. Also, an apparatus for synthesizing from the disparity and image data an image as viewed from a designated viewing direction. The apparatus has a base image frame memory for storing a base image, a reference image frame memory for storing a reference image, a block correlation computing circuit for computing block correlations and confidence measure for estimation by using a plurality of block sizes, a representative pixel correlation computing circuit, and an estimation integrating computing circuit for evaluating the reliability of the result of estimation by block matching on the basis of a luminance gradient, image noise, a minimum value of an evaluation yardstick for differences between blocks, and block size, and for integrating the results of estimation obtained with the plurality of block sizes.

Claims

1. An image synthesizing apparatus, comprising:

a disparity estimation apparatus, said disparity estimation apparatus comprising correspondence determining means, for determining correspondence between a first image and a second image;

disparity data computing means for computing disparity data for the first image and the second image based upon correspondence determined by the correspondence determining means;

occlusion judging means for judging occlusion portions of the second image which are not included in the first image, based upon the disparity data; and

disparity supplementing means for supplementing disparity data for the occlusion portions of the second image;

disparity distance converting means, for converting the supplemented disparity data into distance data;

image generating means for generating hidden image data for a hidden portion of the second image based upon the occlusion portion, the supplemented disparity data, and the image data; and

image synthesizing means for synthesizing a third image from the image data and the hidden image data,

wherein the image generating means

detects which side of the occlusion portion has a greater disparity;

determines the hidden portion to be on the detected side of the occlusion portion; and

employs image data for the occlusion portion and image data for a side of the occlusion portion in the second image opposite the selected side as the hidden image data.

2. An image synthesizing apparatus, comprising:

a disparity estimation apparatus, said disparity estimation apparatus comprising correspondence determining means, for determining correspondence between a first image and a second image;

disparity data computing means for computing disparity data for the first image and the second image based upon correspondence determined by the correspondence determining means;

occlusion judging means for judging occlusion portions of the second image which are not included in the first image, based upon the disparity data; and

disparity supplementing means for supplementing disparity data for the occlusion portions of the second image;

disparity distance converting means, for converting the supplemented disparity data into distance data;

image generating means for generating hidden image data for a hidden portion of the second image based upon the occlusion portion, the supplemented disparity data, and the image data; and

image synthesizing means for synthesizing a third image from the image data and the hidden image data,

wherein the image generating means

detects which side of the occlusion portion has a greater disparity;

determines the hidden portion to be on the detected side of the occlusion portion; and

employs image data for a side of the occlusion portion in the second image opposite the selected side as the hidden image data.

3. A method of estimating disparity for a first image and a second image, comprising:

determining correspondence between the first image and the second image;

computing disparity data for the first image and the second image based upon the determined correspondence;

judging occlusion portions of the second image which are not included in the first image, based upon the disparity data; and

supplementing disparity data for the occlusion portions of the second image,

wherein the supplementation of disparity data for the occlusion portions of the second image includes;

selecting an occlusion portion;

determining which portion of the second image adjacent the occlusion portion has the smallest disparity; and

employing the disparity data for the determined portion of the second image adjacent the occlusion portion to supplement the disparity data for the occlusion portion, the disparity data being supplemented in a direction from the determined portion of the second image to a side of the occlusion portion opposite the determined portion of the second image.

4. A method of synthesizing an image, comprising:

estimating disparity for a first image and a second image, to obtain supplemented disparity data, by determining correspondence between the first image and the second image;

computing disparity data for the first image and the second image based upon the determined correspondence;

judging occlusion portions of the second image which are not included in the first image, based upon the disparity data;

supplementing disparity data for the occlusion portions of the second image;

storing evaluation data of the determined correspondence; and

suppressing disparity data for uncertain portions of the first and second images for which corresponding evaluation data does not meet a first predetermined threshold, wherein judging occlusion portions of the second image which are not included in the first image, is further based on the evaluation data;

converting the supplemented disparity data into distance data;

generating hidden image data for a hidden portion of the second image based upon the occlusion portion, the supplemented disparity data, and the image data; and

synthesizing a third image from the image data and the hidden image data,

wherein the supplementation of disparity data for the occlusion portions of the second image includes;

selecting an occlusion portion;

determining which portion of the second image adjacent the occlusion portion has the smallest disparity; and

employing the disparity data for the determined portion of the second image adjacent the occlusion portion to supplement the disparity data for the occlusion portion, the disparity data being supplemented in a direction from the determined portion of the second image to a side of the occlusion portion opposite the determined portion of the second image.

5. A method of synthesizing an images, comprising:

estimating disparity for a first image and a second image, to obtain supplemented disparity data, by determining correspondence between the first image and the second image;

computing disparity data for the first image and the second image based upon the determined correspondence;

judging occlusion portions of the second image which are not included in the first image, based upon the disparity data;

supplementing disparity data for the occlusion portions of the second image;

storing evaluation data of the determined correspondence; and

suppressing disparity data for uncertain portions of the first and second images for which corresponding evaluation data does not meet a first predetermined threshold, wherein judging occlusion portions of the second image which are not included in the first image, is further based on the evaluation data;

converting the supplemented disparity data into distance data;

generating hidden image data for a hidden portion of the second image based upon the occlusion portion, the supplemented disparity data, and the image data; and

synthesizing a third image from the image data and the hidden image data,

wherein the generation of hidden image data for a hidden portion of the second image includes;

detecting which side of the occlusion portion has a greater disparity;

determining the hidden portion to be on the detected side of the occlusion portion; and

employing image data for the occlusion portion and image data for a side of the occlusion portion in the second image opposite the selected side as the hidden image data.

6. A method of synthesizing an images, comprising:

estimating disparity for a first image and a second image, to obtain supplemented disparity data, by determining correspondence between the first image and the second image;

computing disparity data for the first image and the second image based upon the determined correspondence;

judging occlusion portions of the second image which are not included in the first image, based upon the disparity data;

supplementing disparity data for the occlusion portions of the second image;

storing evaluation data of the determined correspondence; and

suppressing disparity data for uncertain portions of the first and second images for which corresponding evaluation data does not meet a first predetermined threshold, wherein judging occlusion portions of the second image which are not included in the first image, is further based on the evaluation data;

converting the supplemented disparity data into distance data;

generating hidden image data for a hidden portion of the second image based upon the occlusion portion, the supplemented disparity data, and the image data; and

synthesizing a third image from the image data and the hidden image data,

wherein the generation of hidden image data for a hidden portion of the second image includes;

detecting which side of the occlusion portion has a greater disparity;

determining the hidden portion to be on the detected side of the occlusion portion; and

employing image data for a side of the occlusion portion in the second image opposite the selected side as the hidden image data.

7. A method of synthesizing an image, comprising:

estimating disparity for a first image and a second image by determining correspondence between the first image and the second image;

computing disparity data for the first image and the second image based upon the determined correspondence;

judging occlusion portions of the second image which are not included in the first image, based upon the disparity data; and

supplementing disparity data for the occlusion portions of the second image, to obtain supplemented disparity data;

shifting at least the second image according to a designated viewing direction;

detecting no-image data regions which occur in at least the second image as a result of the image shift;

extrapolating image data for the no-image data regions based upon the image data; and

synthesizing a third image from the image data and the extrapolated image data,

wherein the supplementation of disparity data for the occlusion portions of the second image includes;

selecting an occlusion portion;

determining which portion of the second image adjacent the occlusion portion has the smallest disparity; and

employing the disparity data for the determined portion of the second image adjacent the occlusion portion to supplement the disparity data for the occlusion portion, the disparity data being supplemented in a direction from the determined portion of the second image to a side of the occlusion portion opposite the determined portion of the second image.