Method and apparatus for estimating feature values in a region of a sequence of images

• US 6,681,058 B1
• Filed: 03/06/2000
• Issued: 01/20/2004
• Est. Priority Date: 04/15/1999
• Status: Expired due to Term

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Hanna, Keith, Kumar, Rakesh
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US09/518,872
• Time in Patent Office: 1,415 Days
• Field of Search: 382/294, 382/295, 382/284, 382/275, 382/261, 382/265, 382/282, 382/170, 382/190, 358/464, 358/450, 348/412--41, 348/671, 348/672, 348/678
• US Class Current: 382/294
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic

Abstract

A method and apparatus are disclosed that estimate the brightness or other feature values of unchanging or slowly changing regions of an image in a sequence of video images even when the regions is obscured by objects over large portions of the video sequence. The apparatus and method generate a histogram for each image region position over a plurality of image frames in the sequence. The mode, or most frequently occurring value, of the image region as indicated by the histogram is selected as representing the unchanging portion of the image. The mode values of all of the regions are then assembled to form a composite image of the unchanging or slowly changing feature values. According to one method, the histogram is generated using a recursive filter. In order to process images that exhibit some motion from frame to frame, the images in the video sequence may be aligned before generating the histogram. If the camera produces artifacts such as variations in the image caused by an automatic gain control (AGC) function, each image in the sequence of video images may be filtered either temporally or spatially before performing the histogramming operation to remove these artifacts. To reduce processing time, the image processing may be spaced in time such that only every n^th image is processed. Alternatively, each region of an image sequence may be processed at random irregular intervals in order to obtain the histogram. In one embodiment of the invention, the histogram is applied over relatively small groups of frames in order to generate a noise reduced image.

Claims

1. A method for estimating values of features of one or more image regions in a video sequence including a plurality of video image frames when the one or more of the features are obscured by objects over portions of the video sequence, the method comprising the steps of:

generating a histogram for each image region over the a plurality of image frames in the sequence;

calculating at each image region a mode value as indicated by the histogram;

replacing each image region by its mode value to produce an image of the estimated feature values of the sequence of images; and

excluding, from the histogram generating step, regions of the plurality of images in which there is a prior indication that the features are obscured.

2. A method according to claim 1, further including the step of detecting motion among corresponding regions in the plurality of video image frames to provide the prior indication that the features are obscured.

6. A method according to claim 1, wherein the image sequence includes a number of frames, F, wherein the step of generating the histogram collects statistics at each image region for each frame in every n frames of the image sequence and, using the statistics to index into a look-up table, generates the histogram for every nth image frame, where n is an integer less than F.

7. A method according to claim 1, wherein the image sequence includes number of frames, F, wherein the step of generating the histogram processes only every nth image frame in the video sequence where n is an integer less than F.

8. A method according to claim 1, wherein the step of generating the histogram processes image frames in the sequence at irregular intervals.

9. A method according to claim 1, wherein, the image sequence exhibits motion, the step of replacing each image region by its mode generates a sequence of successive estimated feature value images, and the step of generating the histogram operates over a number of frames sufficient to preserve at least a portion of the motion in the sequence of estimated feature value images.

3. A method for estimating values of features of one or more image regions in a video sequence including a plurality of video image frames when the one or more of the features are obscured by objects over portions of the video sequence, the method comprising the step of:

generating a histogram for each image region over the plurality of image frames in the sequence by recursively filtering each image region over the plurality of image frames to generate the histogram;

calculating at each image region a mode value indicated be the histogram; and

replacing each image region by its mode values to produce an image of the estimated feature values of the sequence of images.

4. A method according to claim 3, further comprising the step of aligning the images in the video sequence before generating the histogram.

5. A method according to claim 3, further comprising the step of filtering each image frame in the video sequence before generating the histogram.

10. A method for estimating values of features of one or more image regions in a video sequence including a plurality of video image frames when the one or more of the features are obscured by objects over portions of the video sequence, the method comprising the steps of:

aligning each image frame in the video sequence to a previously processed image frame of the plurality of video image frames;

filtering each image frame In the video sequence to normalize values of the features In the plurality of video image frames in the video sequence;

generating a histogram for each image region over the a plurality of image frames in the sequence, including, for each region In each image frame in the video sequence, the steps of;

determining the feature values for the region in the image frame;

determining if the features in the region are obscured;

if the features in the region are not obscured in the image frame, incrementing cells of a histogram corresponding to the determined feature values;

calculating, at each image region, a mode value as indicated by the histogram;

replacing each image region by its mode value to produce an image of the estimated feature values of the sequence of images.

11. A method according to claim 10, wherein the step of determining if the features in the region are obscured includes the step of detecting motion among corresponding regions in the plurality of video image frames to an indication that features are obscured in regions that exhibit motion.

12. A method according to claim 10, wherein the step of generating the histogram recursively filters each image region over the plurality of image frames to generate the histogram.

13. Apparatus which estimates values of features of one or more image regions in a video sequence including a plurality of video image frames when the one or more of the features are obscured by objects over portions of the video sequence, the apparatus comprising:

a source of input images;

a histogram memory to store a histogram for each image region over the plurality of image frames in the sequence;

a histogram processor which;

recursively filters the input images to generate the histogram for each image region over the plurality of image frames in the sequence; and

calculates a mode value at each image region, as indicated by the histogram to provide an image of the estimated feature values of the sequence of images.

14. Apparatus according to claim 13, wherein the histogram processor includes a recursive filter.

15. Apparatus according to claim 13, wherein the histogram processor includes an image alignment process that spatially aligns the images in the video sequence before generating the histogram.

16. Apparatus according to claim 15, further comprising a filter which filters each image frame in the video sequence before generating the histogram.

17. A carrier including computer instructions that cause a computer to estimate feature values of one or more image regions in a video sequence including a plurality of video image frames when the one or more of the features are obscured by objects over portions or the video sequence, the computer instructions causing the computer to perform the steps of:

generating a histogram for each image region over the plurality of video image frames in the video sequence;

exduding, from the histogram generating step, regions of the plurality of images in which there is a prior indication that the features are obscured;

calculating at each image region a mode value as indicated by the histogram; and

replacing each image region by its mode value to produce an image of the estimated feature values of the sequence of images.

18. A carrier according to claim 17, wherein the computer instructions that cause the computer to exclude regions of the plurality of images in which there is a prior indication that the features are obscured, further cause the computer to detect motion among corresponding regions in the plurality of video image frames to provide the prior indication that the features are obscured.

19. A carrier including a computer instructions that cause a computer to estimate values of features of one or more image regions in a video sequence including a plurality of video image frames when the one or more of the features are obscured by objects over portions of the video sequence, the computer instructions causing the computer to perform the steps of:

generating a histogram for each image region over the plurality of image frames in the sequence by recursively filtering each image region over the plurality of image frames to generate the histogram;

calculating at each image region a mode value as indicated by the histogram; and

replacing each image region by its mode value to produce an image of the estimated feature values of the sequence of images.

20. A carrier according to claim 19, further including instructions that cause the computer to align the images in the video sequence before the instructions that cause the computer to generate the histogram.

21. A carrier according to claim 19, further including Instructions that cause the computer to filter each image frame in the video sequence before the instructions that cause the computer to generate the histogram.