3D stereo real-time sensor system, method and computer program therefor

• US 6,980,210 B1
• Filed: 11/24/1998
• Issued: 12/27/2005
• Est. Priority Date: 11/24/1997
• Status: Expired due to Fees

• Current Assignee: 3D-Image-Prozessing GmbH
• Original Assignee: 3-D Image Processing GmbH
• Inventors: Henkel, Rolf, Weiglhofer, Gerhard, Pauker, Fritz
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US09/199,888
• Time in Patent Office: 2,590 Days
• Field of Search: 345/418, 345/419, 345/426, 345/427, 345/118, 345/121, 345/433
• US Class Current: 345/419
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The invention pertains to an image processing device, in particular a device for stereoscopic image processing, and to a coherence detector used thereon. The image processing device can process stereoscopic images in real time and can be implemented with simple means and low expense. Two alternative embodiments of the coherence detector are disclosed. The image signals can be processed at the speed that they are supplied by image receivers, and thus image data processing is possible to attain depth information with little expense and in real time.

Claims

1. A coherence detector having a number of n signal inputs (EST_i(1<

=I<

=n);

E1, E2, E3, E4), to which one input signal value each is applied, comprising in operative combination;

(a) a first sorting device (S1) for sorting of the n input signal values (E1, . . . , E4) according to their value (A, B, C, D) and for parallel output of the n-sorted values;

(b) a number of n−

1 subtracting devices (“

−

”

) for subtraction of two neighboring, sorted values ((D−

C), (C−

B), (B−

A));

(c) a second sorting device (S2) for sorting of the obtained n−

1 differences ((D−

C), (C−

B), (B−

A)) with regard to their value and for output of the smallest determined difference value (“

<

<

”

);

(d) a selection device (“

=”

, MUX1) which outputs the value pair of the input signal values belonging to the smallest determined difference value (“

<

<

”

); and

(e) an output device (“

Σ

/2”

) to which the output value pair is sent, and which outputs an average value (KE).

2. A coherence detector according to claim 1, wherein:

(a) the selection device (“

=”

, MUX1) has a comparison unit which determines the particular difference value which corresponds to the smallest difference value by comparison of the smallest difference value with the n−

1 difference values; and

(b) a multiplexer unit (MUX1) which outputs that value pair of the input signal values with the smallest difference, based on the comparison result provided by the comparison unit.

3. A signal processing device for processing coherence detector signals having a number of n signal inputs (EST_i), with 1<

=i<

=n, to which a particular input signal value is applied, comprising in operative combination;

(a) a window limiting device (FB) for output of n-input signals having input signal values within a default defined, finite window value range with m+1 values at a spacing of 2×

/m;

(b) a rounding device (R) for rounding of the values output by the window-limiter device (FB) to the next nearest of the m+1 values of the default set window-value range;

(c) an addressing driver unit (AA), and a convolution device which has a number of m+1 signal inputs corresponding to the number of values of the window-value range, said addressing-driver unit supplies the n-signals to a particular k-th input, which n-signals have values of −

x+(k−

1)*2×

/m, to drive a corresponding k-th of m+1 radiation emission element and disposed so that emission radiation is acquired by a number of m+1 radiation reception elements corresponding to the number of values of the window-value range; and

(d) an evaluation device which determines the address k of at least one m+1 radiation receiving element which acquires the greatest radiation intensity, and outputs the value k of this address to an output terminal (KE), which output value represents the result of the convolution of the signals supplied to the convolution unit.

4. A signal processing device according to claim 3, wherein said radiation emission elements of the convolution device are light emitting diodes (LED) and the radiation receiving elements are photo diodes (PD).

5. A signal processing device according to claim 4, including a transparent substrate (SUB) having a top side and a lower side, and wherein said light emitting diodes are located at the top side of a transparent substrate and the photo diodes are located opposite the light emitting diodes at the lower side of said substrate.

6. A signal processing device according to claim 5, wherein said transparent substrate (SUB) is a glass substrate.

7. An image processing device having displacement features (VSEL, VSER;

Δ

X_L1, . . . , Δ

X_Ln, Δ

X_R1, . . . , Δ

X_Rn) for mutual shifting of the image data of a first supplied image and of a second supplied image taken at a different recording angle, comprising in operative combination;

(a) means for picking off the image data in parallel from displacement features (VSEL, VSER) and supplying in pairs ((Δ

X_L1, Δ

X_Rn), (Δ

X_L2, Δ

X<sub>Rn−

1</sub>), . . . (Δ

X<sub>Ln−

1</sub>, Δ

X_R2), (Δ

X_Ln, Δ

X_R1)) to an outlet-connected disparity-detection device (DD;

EST₁, . . . , EST_n);

(b) a disparity-detection device (DD) having a particular disparity element (EST₁, . . . , EST_n) for determining a disparity value representing spatial depth information for the particular pair of image data for each of the supplied image data pairs ((Δ

X_L1, Δ

X<sub>Rn+1−

i</sub>)); and

(c) said disparity-detection device outputs the determined disparity values to a coherence-detection device (KD) in which the output disparity value (KE) is determined for every shift of image data and represents the associated spatial depth information.

8. An image processing device according to claim 7, wherein

(a) the image data picking means supplies image data as half images, line-nested, during two sequential half image periods;

and

(b) which includes;

(i) a memory device (MEM) for interim storage of actual disparity values determined by the coherence-detection device (KD) during a first half image period; and

(ii) means for supplying the interim-stored disparity values to said coherence-detection device (KD) during the subsequent, second half-image period, in addition to the disparity values ascertained by the disparity-detection device (DD), from which the actual disparity values for the full image may be determined.

9. An image processing device according to claim 7, which includes:

(a) a preprocessing device for receiving said supplied image data of the first image and of the second image; and

(b) said preprocessing device produces different types of image data from the image data of the first image and of the second image and outputs said image data to displacement devices and to at least one disparity-detection device (DD) for the particular type of image data.

10. An image processing device according to claim 7, wherein said image picking device supplies the image data of the first and second image serially and synchronized to each other.

11. An image processing device according to claim 10, wherein said displacement devices shift the image data point by at least a fraction of an image point with respect to each other.

12. An image processing device according to claim 11, wherein:

(a) said image picking device supplies said image data as analog image signals; and

(b) said displacement devices comprise analog delay-line network chains, and shifts of the image elements by adjusting corresponding transit time.

13. An image processing device according to claim 11, wherein:

(a) said image picking device supplies said image data as digital image signals; and

(b) said displacement devices comprise clocked shift registers.

14. An image processing device according to claim 11, wherein:

(a) said image picking device supplies said image data as digital image signals; and

(b) said displacement devices comprise filters that interpolate the image data values between neighboring image elements.

15. An image processing device according to claim 7, wherein at least one disparity element determines the particular disparity value by a gradient-based processing.

16. An image processing device according to claim 7, which includes a display device to which the actual disparity values (KE) are input and displayed as a disparity map representing the depth information.

17. A method of image processing having displacement steps (VSEL, VSER;

Δ

X_L1, . . . , Δ

X_Ln, Δ

X_R1, . . . , Δ

X_Rn) for mutual shifting of the image data of a first supplied image and of a second supplied image taken at a different recording angle, comprising the following steps in operative order;

(a) picking off said image data in parallel from displacement features (VSEL, VSER);

(b) supplying said image data in pairs ((Δ

X_L1, Δ

X_Rn), (Δ

X_L2, Δ

X<sub>Rn−

1</sub>), . . . (Δ

X<sub>Ln−

1</sub>, Δ

X_R2), (Δ

X_Ln, Δ

X_R1)) to an outlet-connected disparity-detection device (DD;

EST₁, . . . , EST_n);

(c) determining in said disparity-detection device a disparity value representing spatial depth information for the particular pair of image data for each of the supplied image data pairs ((Δ

X_L1, Δ

X<sub>Rn+1−

i</sub>));

(d) outputting determinied disparity values to a coherence-detection device (KD); and

(e) determining in said coherence-detection device actual disparity values (KE) for every shift of image data representative of associated spatial depth information.

18. A method of image processing according to claim 17, wherein:

(a) said image data supplying step includes supplying image data as half images, line-nested, during two sequential half image periods; and

(b) the method includes the steps of;

(i) storaging in a memory device (MEM) for an interim period said actual disparity values determined by the coherence-detection device (KD) during a first half image period; and

(ii) supplying the interim-stored disparity values to said coherence-detection device (KD) during the subsequent, second half-image period, in addition to the disparity values ascertained by said disparity-detection device (DD), from which the actual disparity values for the full image may be determined.

19. A method of image processing according to claim 17, which includes:

(a) the step of providing a preprocessing device for receiving said supplied image data of the first image and of the second image; and

(b) producing from said preprocessing device different types of image data from the image data of the first image and of the second image and outputting said image data to displacement devices and to at least one disparity-detection device (DD) for the particular type of image data.

20. An image processing computer program product, comprising:

(a) a computer useable medium having computer readable program code embodied thereon for performing displacement steps (VSEL, VSER;

Δ

X_L1, . . . , Δ

X_Ln, Δ

X_R1, . . . , Δ

X_Rn) for mutual shifting of the image data of a first supplied image and of a second supplied image taken at a different recording angle; and

(b) said computer readable program code causes a computer to;

(i) determine a disparity value representing spatial depth information for the particular pair of image data for selected image data pairs ((Δ

X_L1, Δ

X<sub>Rn+1−

i</sub>)) supplied to said computer; and

(ii) determine for every shift of image data actual disparity values (KE) representative of associated spatial depth information.