Method and apparatus for determining ambient conditions from an image sequence, such as fog, haze or shadows

• US 6,037,976 A
• Filed: 10/30/1996
• Issued: 03/14/2000
• Est. Priority Date: 10/31/1995
• Status: Expired due to Term

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Wixson, Lambert Ernest
• Primary Examiner(s): Garber, Wendy
• Assistant Examiner(s): HARRINGTON, ALICIA M
• Application Number:
  US08/742,433
• Time in Patent Office: 1,231 Days
• Field of Search: 348/297, 348/143, 348/122, 348/229, 348/129, 348/152, 348/239, 348/602, 382/174
• US Class Current: 348/122
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/41 based on statistical description of texture
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

A method and apparatus for determining certain ambient conditions in a scene by analyzing a sequence of images that represent the scene. The apparatus uses only image information to determine scene illumination, or the presence of shadows, fog, smoke, or haze by comparing properties of detected objects, averaged over a finite video sequence, against properties of the reference image of the scene as that scene would appear without any objects present. Such a reference image is constructed in a manner similar to time-averaging successive camera images.

Claims

1. A method for detecting ambient conditions of a scene represented by a sequence of images comprising the steps of:

generating a reference image containing background information regarding the scene;

comparing, pixel-by-pixel, each of said images in said sequence of images to said reference image;

classifying, in response to each comparison, the pixels of said images as either background or non-background;

comparing a brightness measure of each non-background pixel, computed over a neighborhood of pixels local to that pixel, against a threshold; and

processing each non-background pixel to determine the ambient conditions of the scene by

classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels; and

processing the mean energy difference of the dark and bright pixels to determine if the scene contains shadows.

2. The method of claim 1 wherein the processing step further comprises:

classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

sum the number of bright and dark pixels over a number of images; and

process the sums of bright and dark pixels to determine if the scene is well-lit or poorly-lit.

3. The method of claim 2 wherein the processing step further comprises the steps of:

classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels;

determining an absolute magnitude of the mean energy difference of all bright non-background pixels and an absolute magnitude of the mean energy difference for all dark non-background pixels; and

processing the absolute magnitude of the mean energy difference of the dark and bright pixels to determine if the scene contains fog or haze.

4. The method of claim 1 wherein the threshold is the brightness measure obtained over a corresponding neighborhood of pixels in the reference image.

5. The method of claim 1 wherein the processing step further comprises the steps of:

classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels;

determining an absolute magnitude of the mean energy difference of all bright non-background pixels and an absolute magnitude of the mean energy difference for all dark non-background pixels; and

process the absolute magnitude of the mean energy difference of the dark and bright pixels to determine if the scene contains fog or haze.

6. Apparatus for detecting ambient conditions of a scene represented by a sequence of images comprising:

a reference image generator for generating a reference image containing background information regarding the scene;

a comparator for comparing, pixel-by-pixel, each of said images in said sequence of images to said reference image;

an illumination processor for classifying, in response to each comparison, the pixels of said images as either background or non-background, for comparing a brightness measure of each non-background pixel, computed over a neighborhood of pixels local to that pixel, against a threshold, and for processing each non-background pixel to determine the ambient conditions of the scene, wherein said illumination processor comprises;

a comparator for comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

means for determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels; and

means for processing the mean energy difference of the dark and bright pixels to determine if the scene contains shadows.

7. The apparatus of claim 6 wherein the illumination processor further comprises:

means for classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

means for summing the number of bright and dark pixels over a number of images; and

means for processing the sums of bright and dark pixels to determine if the scene is well-lit or poorly-lit.

8. The apparatus of claim 7 wherein the illumination processor further comprises:

means for classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

a comparator for comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

means for determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels;

means for determining an absolute magnitude of the mean energy difference of all bright non-background pixels and an absolute magnitude of the mean energy difference for all dark non-background pixels; and

means for processing the absolute magnitude of the mean energy difference of the dark and bright pixels to determine if the scene contains fog or haze.

9. The apparatus of claim 6 wherein the threshold is the brightness measure obtained over a corresponding neighborhood of pixels in the reference image.

10. The apparatus of claim 6 wherein the illumination processor further comprises:

means for classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

means for comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

means for determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels;

means for determining an absolute magnitude of the mean energy difference of all bright non-background pixels and an absolute magnitude of the mean energy difference for all dark non-background pixels; and

means for processing the absolute magnitude of the mean energy difference of the dark and bright pixels to determine if the scene contains fog or haze.

11. A method for detecting ambient conditions of a scene represented by a sequence of images comprising the steps of:

generating a reference image containing background information regarding the scene;

comparing, pixel-by-pixel, each of said images in said sequence of images to said reference image;

classifying, in response to each comparison, the pixels of said images as either background or non-background;

comparing a brightness measure of each non-background pixel, computed over a neighborhood of pixels local to that pixel, against a threshold; and

processing each non-background pixel to determine the ambient conditions of the scene by

classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels;

determining an absolute magnitude of the mean energy difference of all bright non-background pixels and an absolute magnitude of the mean energy difference for all dark non-background pixels; and

processing the absolute magnitude of the mean energy difference of the dark and bright pixels to determine if the scene contains fog or haze.

12. The method of claim 11 wherein the processing step further comprises:

classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

sum the number of bright and dark pixels over a number of images; and

process the sums of bright and dark pixels to determine if the scene is well-lit or poorly-lit.

13. The method of claim 12 wherein the processing step further comprises the steps of:

classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels; and

processing the mean energy difference of the dark and bright pixels to determine if the scene contains shadows.

14. The method of claim 11 wherein the threshold is the brightness measure obtained over a corresponding neighborhood of pixels in the reference image.

15. The method of claim 11 wherein the processing step further comprises the steps of:

classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels;

processing the mean energy difference of the dark and bright pixels to determine if the scene contains shadows.

16. Apparatus for detecting ambient conditions of a scene represented by a sequence of images comprising:

a reference image generator for generating a reference image containing background information regarding the scene;

a comparator for comparing, pixel-by-pixel, each of said images in said sequence of images to said reference image;

an illumination processor for classifying, in response to each comparison, the pixels of said images as either background or non-background, for comparing a brightness measure of each non-background pixel, computed over a neighborhood of pixels local to that pixel, against a threshold, and for processing each non-background pixel to determine the ambient conditions of the scene, wherein said illumination processor further comprises;

a comparator for comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

means for determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels; and

means for processing the mean energy difference of the dark and bright pixels to determine if the scene contains fog or haze.

17. The apparatus of claim 16 wherein the illumination processor further comprises:

means for classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

means for summing the number of bright and dark pixels over a number of images; and

means for processing the sums of bright and dark pixels to determine if the scene is well-lit or poorly-lit.

18. The apparatus of claim 17 wherein the illumination processor further comprises:

means for classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

a comparator for comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

means for determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels;

means for determining an absolute magnitude of the mean energy difference of all bright non-background pixels and an absolute magnitude of the mean energy difference for all dark non-background pixels; and

means for processing the absolute magnitude of the mean energy difference of the dark and bright pixels to determine if the scene contains shadows.

19. The apparatus of claim 16 wherein the threshold is the brightness measure obtained over a corresponding neighborhood of pixels in the reference image.

20. The apparatus of claim 16 wherein the illumination processor further comprises:

means for classifying, in response to said comparison, each of said non-background pixels as either bright or dark;

a comparator for comparing an energy value of each pixel in said image to an energy value in each pixel of said reference image;

means for determining a mean energy difference of all bright non-background pixels and a mean energy difference for all dark non-background pixels;

means for determining an absolute magnitude of the mean energy difference of all bright non-background pixels and an absolute magnitude of the mean energy difference for all dark non-background pixels; and

means for processing the absolute magnitude of the mean energy difference of the dark and bright pixels to determine if the scene contains shadows.