Three-dimensional reference image segmenting method and apparatus

• US 5,684,890 A
• Filed: 02/24/1995
• Issued: 11/04/1997
• Est. Priority Date: 02/28/1994
• Status: Expired due to Term

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Okajima, Kenji, Miyashita, Masanobu
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Chen, Wenpeng
• Application Number:
  US08/394,278
• Time in Patent Office: 984 Days
• Field of Search: 382/154, 382/153, 382/173, 382/190, 382/191, 382/195, 382/282, 382/209, 382/216, 382/236, 382/260, 382/261, 382/279, 382/280, 382/291, 382/242, 382/106, 382/255, 382/302, 382/278, 364/822, 395/119, 340/815.47, 348/20, 348/47, 348/207
• US Class Current: 382/154
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/168 involving transform domain methods
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

The images of an object equivalent to those obtained when as object is viewed respectively by the right and left eyes are subdivided into small areas with overlaps allowed therebetween and are Fourier transformed to achieve a local Fourier transform for the images. Phase difference of Fourier transforms is obtained according to the right and left local Fourier transform images of a reference image which is stored beforehand in a storage memory. Alternatively, a Fourier transform power spectrum is obtained. Disparity is represented in terms of a phase difference or in the form of a Fourier transform power spectral pattern. According to either one of the above results, and on the basis of a local Fourier transform of an input image including the reference image, an identical object image is estimated. For the image, a local inverse Fourier transform is effected such that geometric mean values are attained between the right and left images so as to segment an image matching the reference image and, also, with respect to disparity.

Claims

1. A three-dimensional reference image segmenting apparatus, comprising:

means for inputting an image of an object, the image having a left image and a right image, respectively equivalent to those obtained when the object is viewed by a right eye and a left eye, respectively;

means for subdividing the right and left input images of the image into predetermined areas, calculating a Fourier transform for each of the areas;

means for representing a disparity between respective local areas of said left and right image in terms of a phase difference between their respective local Fourier transforms;

means for obtaining a maximum power spectral pattern by computing a sum of a right power spectrum and a left power spectrum when the phases of the local Fourier transform of said left and right images match each other for each spatial frequency;

storage means for storing a maximum power spectral pattern of a reference image;

means for estimating, according to the local Fourier transforms of the input images, the maximum power spectral pattern of the input image, and the maximum power spectral pattern of the reference image, a local Fourier transform of an image existing in the input image and having a power spectral pattern analogous to that of the reference image, thereby obtaining a local Fourier transform of an identical object image identical to the reference image; and

means for computing a local inverse Fourier transform according to the local Fourier transform estimated by the estimating means, thereby segmenting the reference image from the input image even when the object of the reference image is included in the input image and a positional difference exists in a depth direction between the reference image and the object included in the input image.

2. A three-dimensional reference image segmenting method, comprising the steps of:

inputting a pixel image including right pixel image and a left pixel image, equivalent to images obtained when an object is viewed by a right eye and a left eye, respectively;

dividing the input right pixel and left pixel images each into a plurality of areas, each represented as f.sub.μ

(x), where f.sub.μ

(x) is a value of a pixel at a position identified by a space vector x, and μ

is a parameter having a value of "L" and "R", respectively, to represent the right and the left images;

calculating, for each f.sub.μ

(x) area of the right input pixel image and the left input pixel image, a local Fourier transform F.sub.μ

(k,X), where X is a space vector pointing to a center of a corresponding area f.sub.μ

(x), and k is a space frequency; and

segmenting a right and a left target image corresponding to the object from the input images in accordance with the local Fourier transform F.sub.μ

(k,X) and a prior stored local Fourier transform F.sub.μ

(k,X) of each of the areas in each of a right and left reference image of the object, and such a phase difference δ

φ

₁ satisfying a relation that;

space="preserve" listing-type="equation">F.sub.R (k,X)⃡

F.sub.L (k,X)exp(-iδ

φ

.sub.1).

3. A method according to claim 2, wherein said step of segmenting a right and a left target image comprises the steps of:

calculating, for each of the areas in each of the left input pixel images and right input pixel images, a local Fourier transform equivalent F'"'"'.sub.μ

(k,X) from the prior stored local Fourier transform F.sub.μ

(k,X) and said phase difference on the basis of a local disparity between the input images f_R (x) and f_L (x) being expressed as the phase difference;

calculating an inverse Fourier transform image f'"'"'μ

(x), corresponding to the left input pixel images and right input pixel images, from the local Fourier transform equivalents F'"'"'μ

(k,X); and

finding a geometric mean fμ

(x) between each fμ

(x) of the left input pixel images and right input pixel images and the corresponding inverse Fourier transform image f'"'"'μ

(x), where fμ

(x) is an image which only contains an objective image corresponding to the object.

4. A method according to claim 3, wherein the step of calculating a local Fourier transform equivalent F'"'"'μ

(k,X) comprises the step of;

calculating a local Fourier transform equivalent according to the following equations;

space="preserve" listing-type="equation">F.sub.R '"'"'(k,X)=F.sub.R (k,X)exp(-iδ

φ

.sub.1), and

space="preserve" listing-type="equation">F.sub.L '"'"'(k,X)=F.sub.L (k,X)exp(-iδ

φ

.sub.1).

5. A method according to claim 3, wherein the step of calculating a local Fourier transform equivalent F'"'"'μ

(k,X) comprises the step of;

calculating a local Fourier transform equivalent according to the following equations;

space="preserve" listing-type="equation">F.sub.R '"'"'(k,X)=|F.sub.R (k,X)|w(F.sub.L,F.sub.R)exp(-iδ

φ

.sub.1), and

space="preserve" listing-type="equation">F.sub.L '"'"'(k,X)=|F.sub.L (k,X)|w(F.sub.R,F.sub.L)exp(iδ

φ

.sub.1),

where

space="preserve" listing-type="equation">w(Z.sub.1,Z.sub.2)=Z.sub.1 /|Z.sub.1 |;

ε

<

|Z.sub.1 |,ε

<

|Z.sub.2 |=0;

otherwise.

6. A method according to claim 3, wherein the step of calculating a local Fourier transform equivalent F'"'"'μ

(k,X) comprises the step of;

obtaining the local Fourier transform equivalent F'"'"'μ

(k,X) from the following equations;

space="preserve" listing-type="equation">E.sub.R Σ

.sub.δ

φ

(|F.sub.L (k,X)+F.sub.R '"'"'(k,X)exp(iδ

φ

)|-λ

.sub.R P.sub.B (k,X,δ

φ

)).sup.2, and

space="preserve" listing-type="equation">E.sub.L =Σ

.sub.δ

φ

(|F.sub.L '"'"'(k,X)+F.sub.R( k,X)exp(iδ

φ

)|-λ

.sub.L P.sub.B (k,X,δ

φ

).sup.2, where

space="preserve" listing-type="equation">P.sub.B (k,X,δ

φ

)=|F.sub.L (k,X)+F.sub.L (k,X)exp(iδ

φ

)|.sup.2 and λ

.sub.R and λ

.sub.L are positive auxiliary functions.

7. A method according to claim 2, wherein said step of dividing each of the left input pixel image and right input pixel image into predetermined areas provides overlaps between said predetermined areas.

8. A method according to claim 2, wherein said step of segmenting a right and a left target image corresponding to the object comprises the step of:

compensating a positional difference in depth direction between the reference image and the target image in the left input pixel image and right input pixel image.

9. A method according to claim 8, wherein said step of compensating a positional difference comprises the steps of:

obtaining an apparent disparity d caused by the positional difference in depth direction between the reference image and the target image in the left input pixel image and right input pixel image; and

replacing the phase difference δ

φ

₁ with δ

φ

₁ +kd.

10. A three-dimensional reference image segmenting apparatus comprising:

means for inputting a right pixel image and a left pixel image equivalent to images obtained when an object is viewed by a right eye and a left eye, respectively;

means for dividing the input right pixel and left pixel images each into a plurality of predetermined areas, each represented as f.sub.μ

(x) where f.sub.μ

(x) is a value of a pixel at a position identified by a space vector x, and μ

is a parameter having a value of "L" and "R", respectively, to represent the right and the left images;

means for calculating, for each f.sub.μ

(x) area of the right input pixel image and the left input pixel image, a local Fourier transform F.sub.μ

(k,X), where X is a space vector pointing to a center of a corresponding area f.sub.μ

(x), and k is a space frequency; and

means for segmenting a right and a left target image corresponding to the object from the input pixel image by using the local Fourier transform fμ

(k,X), a prior stored local Fourier transform Fμ

(k,X) of each of the predetermined areas in each of a right and a left reference image of the object, and such a phase difference δ

φ

₁ satisfying a relation that;

space="preserve" listing-type="equation">F.sub.R (k,X)⃡

F.sub.L (k,X)exp(-iδ

φ

.sub.1).

11. An apparatus according to claim 10, wherein the means for segmenting a right and a left target image comprises:

means for calculating, for each of the areas in each of the input pixel image, a local Fourier transform equivalent F'"'"'μ

(k,X) from the local Fourier transform Fμ

(k,X), the prior stored local Fourier transform and the phase difference on the basis of a local disparity between the input images f_R (x) and f_L (X) being expressed as the phase difference;

means for calculating, for each of the input images, an inverse Fourier transform image fμ

(x) from all the local Fourier transform equivalents F'"'"'μ

(k,X); and

means for finding a geometric mean fμ

(x) between each fμ

(x) of the input pixel image and the corresponding inverse Fourier transform image f'"'"'μ

(x), where fμ

(x) is an image which only contains an objective image corresponding to the object.

12. An apparatus according to claim 11, wherein the means for calculating a local Fourier transform equivalent F'"'"'μ

(k,X) comprises;

means for calculating the following equations;

space="preserve" listing-type="equation">F'"'"'.sub.R (k,X)=F.sub.R (k,X)exp(-iδ

φ

.sub.1), and

space="preserve" listing-type="equation">F'"'"'.sub.L (k,X)exp(-iδ

φ

.sub.1).

13. An apparatus according to claim 11, wherein the means for calculating a local Fourier transform equivalent F'"'"'μ

(k,X) comprises;

means for calculating the following equations;

F'"'"'_R (k,X)=|F_R (k,X)|w(F_L,F_R)exp(-iδ

φ

₁), and

space="preserve" listing-type="equation">F'"'"'.sub.L (k,X)=|F.sub.L (k,X)|w(F.sub.R, F.sub.L)exp(-δ

φ

.sub.1),

where

space="preserve" listing-type="equation">w(Z.sub.1,Z.sub.2)=Z.sub.1 /|Z.sub.1 |;

ε

<

|Z.sub.1 |,ε

<

|Z.sub.2 |=0;

otherwise.

14. An apparatus according to claim 11, wherein the means for calculating a local Fourier transform equivalent Fμ

(k,X) comprises;

means for obtaining such the local Fourier transform equivalents F'"'"'μ

(k,X) as minimizes the following equations;

space="preserve" listing-type="equation">E.sub.R =Σ

.sub.δ

φ

(|F.sub.L (k,X)+F'"'"'.sub.R (k,X)exp(iδ

φ

)|-λ

.sub.R P.sub.B (k,X,δ

φ

)).sup.2),

space="preserve" listing-type="equation">E.sub.L =Σ

.sub.δ

φ

(|F.sub.L '"'"'(k,X)+F.sub.R (k,X)exp(iδ

φ

)|-λ

.sub.L P.sub.B (k,X,δ

φ

)).sup.2, and

space="preserve" listing-type="equation">P.sub.B (k,X,δ

φ

)=|F.sub.L (k,X)+F.sub.L (k,X)exp(iδ

φ

)|.sup.2, where λ

.sub.R and λ

.sub.L are positive auxiliary functions.

15. An apparatus according to claim 10, wherein the means for dividing the input pixel image into predetermined areas comprises means for providing overlaps between said predetermined areas.

16. An apparatus according to claim 10, wherein the means for segmenting a right and a left target image corresponding to the object comprises:

means for compensating a positional difference in depth direction between the reference image and the target image in the input pixel image.

17. An apparatus according to claim 16, wherein the means for compensating a positional difference comprises:

means for obtaining an apparent disparity d caused by the positional difference in depth direction between the reference image and the target image in the input pixel image; and

means for replacing the phase difference δ

φ

₁ with δ

φ

₁ +kd.