Wide dynamic range network camera

• US 7,492,391 B1
• Filed: 07/14/2004
• Issued: 02/17/2009
• Est. Priority Date: 07/14/2003
• Status: Active Grant

• Current Assignee: Arecont Vision Costar, LLC (Costar Technologies, Inc.)
• Original Assignee: Arecont Vision LLC (Costar Technologies, Inc.)
• Inventors: Kaplinsky, Michael
• Primary Examiner(s): Henn; Timothy J
• Application Number:
  US10/890,870
• Time in Patent Office: 1,679 Days
• Field of Search: 34821199-2113, 348/216.1, 348/218.1, 348/239, 348/296, 348/297, 348/308
• US Class Current: 348/218.1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/76 by influencing the image signals
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/46 by combining or binning pixels
• CPC Class Codes: H04N 25/57 Control of the dynamic range

Abstract

A network camera comprising an image sensor, an image processing unit, a buffer memory and a network interface. The image processing unit is configured to divide the entire image readout by the image sensor into a plurality of overexposed regions and a plurality of non-overexposed regions, select a subset of the overexposed regions containing most oversaturated pixels, control said image sensor to readout only said selected subset of the overexposed regions with adjusted optical integration time, apply multiplicative scaling to readout pixels of said selected subset of the overexposed regions to generate scaled pixels, and replace pixels in the entire image corresponding to said selected subset of the overexposed regions with respective said scaled pixels to increase dynamic range of the entire image.

Claims

1. A method for generating wide dynamic range image in video cameras having an image sensor capable of readout of externally specified partial frames, comprising the following steps:

configuring said image sensor to expose and readout a base frame;

applying a low-pass filter to dark areas of the base frame;

compressing base frame pixel range from 0 to TH1 into range 0 to TH2 and stretching pixel range from TH1 to ADCmax into range from TH2 to ADCmax, wherein TH1 is a parameter where pixels above TH1 are oversaturated and wherein ADCmax is the maximum bit value of ADC output and wherein TH2 is a parameter where TH2<

TH1;

determining rectangular image areas (windows) enclosing largest oversaturated base frame areas;

storing the base frame in an image buffer memory;

configuring said image sensor to sequentially expose and readout said windows;

compressing the pixel range in each readout window from t_w*TH1/t₀ ADCmax into range TH2 to ADCmax, wherein said t_w is an optical integration time of a current window and said t₀ is an optical integration time of said base frame; and

replacing in said image buffer memory base frame pixels with luminance values greater than TH2 with corresponding pixels from said windows.

2. The method of claim 1, wherein said configuring of said image sensor to expose and readout complete base frame comprises setting image sensor optical integration time, wherein said optical integration time is calculated using luminance statistics from a previously acquired base frame.

3. The method of claim 1, wherein said applying of low-pass filter comprises replacing all dark pixels with weighted average of nine pixels centered at a current dark pixel, wherein dark pixels have luminance values below low luminance threshold T0, and wherein T0 is a predetermined parameter.

4. The method of claim 1, wherein said compressing and said stretching of said pixel ranges of said base frame is performed by means of the following transformation:

$\begin{array}{c}{Y}_{\mathrm{base}}^{\mathrm{new}}=<br><br>Y_{\mathrm{base}}·<br><br>\frac{\mathrm{TH}<br><br><br><br>2}{\mathrm{TH}<br><br><br><br>1},\mathrm{for}<br><br><br><br>\mathrm{base}<br><br><br><br>\mathrm{frame}<br><br><br><br>\mathrm{pixels}<br><br><br><br>\mathrm{with}<br><br><br><br>Y_{\mathrm{base}}<<br><br>\mathrm{TH}<br><br><br><br>1<br><br><br><br>\mathrm{and}\\ Y_{\mathrm{base}}^{\mathrm{new}}=<br><br>\left(Y_{\mathrm{base}}-\mathrm{TH}<br><br><br><br>1\right)·<br><br>\frac{\mathrm{ADC}<br><br><br><br>\max -\mathrm{TH}<br><br><br><br>2}{\mathrm{ADC}<br><br><br><br>\max -\mathrm{TH}<br><br><br><br>1}+\mathrm{TH}<br><br><br><br>2,\mathrm{for}<br><br><br><br>Y_{\mathrm{base}}><br><br>\mathrm{TH}<br><br><br><br>1,\end{array}$

where Y_base is a pixel value in said base frame.

5. The method of claim 1, wherein said determining rectangular image areas (windows) enclosing largest oversaturated base frame areas comprises:

dividing the entire image into small rectangular sub-frames, wherein sub-frames comprise one or more image pixels;

identifying sub-frames with high luminance values, wherein sub-frame luminance is high when the average of the luminance values of pixels comprising said sub-frame exceeds TH, wherein TH is a predetermined parameter;

grouping said sub-frames into clusters, wherein said clusters are defined as two or more sub-frames with high luminance values are positioned so that all their centers can be connected together by a continuous curve passing only within the sub-frames of the cluster; and

selecting each of said clusters and determining a minimum size rectangular window enclosing all pixels of the cluster.

6. The method of claim 1, wherein said configuring of said image sensor to sequentially expose and readout said windows comprises setting an image sensor optical integration time for exposure of each window, wherein said image sensor optical integration time is calculated using luminance statistics from a window nearest to the current window, wherein said nearest window was readout during the generation of previous composite frame.

7. The method of claim 5, wherein only two, three, or four windows enclosing largest oversaturated base frame areas are used for generation of said wide dynamic range image.

8. A method for generating wide dynamic range images in a video camera having an image sensor, the method comprising:

controlling said image sensor to expose and readout a base frame;

dividing the base frame into a plurality of sub-frames;

identifying sub-frames having average luminance value greater than a predetermined threshold, as over exposed sub-frames;

grouping the overexposed sub-frames into a plurality of clusters, wherein each cluster is defined as two or more overexposed sub-frames positioned so that their centers can be connected together by a continuous curve passing within the sub-frames of the each cluster;

selecting a portion of the plurality of clusters;

for each cluster in the selected portion, determining a minimum size rectangular window enclosing the entirety of the each cluster;

exposing each of said windows;

applying multiplicative scaling to image pixels in each window; and

replacing pixels of base frame with scaled pixels from one or more windows, wherein said base frame and said windows are exposed at different values of said optical integration time.

9. The method of claim 8, wherein said controlling of said image sensor comprises controlling image sensor optical integration time, wherein said optical integration time is calculated using luminance statistics from a previously acquired base frame.

10. The method of claim 8, wherein said replacing pixels of base frame comprises replacing all dark pixels with weighted average of nine pixels centered at a current dark pixel, wherein dark pixels have luminance values below a low luminance threshold.

11. The method of claim 8, wherein said controlling of said image sensor comprises setting an image sensor optical integration time for exposure of each window, wherein said image sensor optical integration time is calculated using luminance statistics from a window nearest to the current window, and wherein said nearest window was readout during the generation of previous composite frame.

12. A video camera comprising:

an image sensor for exposing and reading out a base frame;

a memory buffer for buffering said base frame; and

an image processing unit configured to divide the base frame into a plurality of sub-frames;

identify sub-frames having average luminance value greater than a predetermined threshold, as overexposed sub-frames;

group the overexposed sub-frames into a plurality of clusters, wherein each cluster is defined as two or more overexposed sub-frames positioned so that their centers can be connected together by a continuous curve passing within the sub-frames of the each cluster;

select a portion of the plurality of clusters, for each cluster in the selected portion, determining a minimum size rectangular window enclosing the entirety of the each cluster;

exposed each of said windows;

apply multiplicative scaling to image pixels in each window; and

replace pixels of base frame with scaled pixels from one or more windows, wherein said base frame and said windows are exposed at different values of said optical integration time.

13. The video camera of claim 12, further comprising a network interface for transmitting image data to a computer network.

14. The video camera of claim 12, wherein said image processing unit comprises a plurality of comparators, wherein said image processing unit is configured to determine said overexposed pixels using said comparators, and wherein said overexposed pixels are pixels that exceed a given threshold value.

15. The video camera of claim 13, wherein said network interface is Ethernet network interface.

16. The video camera of claim 12, further comprising an image processor equipped to perform image compression.