System and method for multi-resolution background adaptation

• US 6,061,088 A
• Filed: 01/20/1998
• Issued: 05/09/2000
• Est. Priority Date: 01/20/1998
• Status: Expired due to Term

• Current Assignee: FLIR Commercial Systems Incorporated (Teledyne Technologies Incorporated)
• Original Assignee: NCR Corporation
• Inventors: Perez-Jacome, Jorge E., Moed, Michael C., Crabtree, Ralph N., Khosravi, Mehdi
• Primary Examiner(s): Le, Vu
• Application Number:
  US09/009,167
• Time in Patent Office: 840 Days
• Field of Search: 348/169, 348/584, 348/586, 348/578, 348/700, 348/576, 348/575, 382/103, 382/282
• US Class Current: 348/169
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A multi-resolution background adaptation system and method wherein image information for a video frame is analyzed to update a background image in as great a resolution as possible. The video frame is partitioned into a plurality of cell levels, each cell level comprising at least one cell. Each cell is further partitioned into children cells for several hierarchical levels. Background image adaptation begins at the frame level and proceeds to lower cell levels of smaller cells when the lower cell levels are not substantially occluded by objects.

Claims

1. A system for generating background image information of a scene comprising:

(a) a video camera suitable for viewing the scene and generating video signals representing image information for the scene;

(b) a frame grabber connected to the video camera for generating a stream of video frames, each video frame comprising image information for the scene at an instant of time; and

(c) a processor connected to the frame grabber, the processor being programmed to update image information for a background image for the scene during a background update cycle based upon image information for a video frame, by;

(i) selecting a region of the video frame;

(ii) analyzing image information for smaller subregions within the region to determine whether any subregion is substantially occluded by an object;

(iii) updating the background image based upon image information for the region if it is determined that any subregion within the region is substantially occluded by an object; and

(iv) repeating steps (ii) and (iii) for each of the subregions when it is determined that the none of the subregions is substantially occluded by an object.

2. The system of claim 1, wherein the processor is further programmed to update the background image based upon image information for the region by:

(v) determining whether image information for the region indicates that the region is sufficiently occluded by an object; and

(vi) if it is determined that the region is sufficiently occluded by an object, then determining whether motion has occurred, has not occurred or is pending within the region with respect to image information for the region retained at a prior background update cycle.

3. The system of claim 2, wherein the processor is programmed to update the background image with image information for the region if it is determined that motion has not occurred within the region.

4. The system of claim 2, wherein the processor is programmed to maintain unchanged image information in the background image corresponding to the region if it is determined that motion has occurred within the region.

5. The system of claim 2, wherein the processor is programmed to gradually update the background image with image information for the region if it is determined that motion is pending within the region.

6. The system of claim 2, wherein the processor is programmed to update the background image when it is determined that the region is not sufficiently occluded by an object, by:

(vii) determining a measure of difference between image information for the region in the video frame with image information for a corresponding region in the background image for a prior background update cycle;

(viii) updating the background image with image information for the region if the measure of difference is less than a predetermined threshold; and

(ix) gradually updating the background image with the image information for the region if the measure of difference is greater than the predetermined threshold.

7. The system of claim 2, wherein the processor is programmed to store image information in a reference frame for regions in which during a background update cycle it is determined that motion has occurred or has not occurred, and wherein the processor is programmed to determine whether motion has occurred, has not occurred, or is pending within the region based upon the reference frame by:

(x) comparing image information for the region of the video frame with image information for a corresponding region in the reference frame;

(xi) determining that motion has occurred within the region when there is a significant difference between image information for the region in the video frame and image information for the corresponding region in the reference frame;

(xii) determining that motion has not occurred within the region when there is not a significant difference between image information for the region in the current video frame and image information for the corresponding region in the reference frame, and a period of time has elapsed since the reference frame was previously updated with image information for the region, which period of time is greater than a predetermined threshold; and

(xiii) determining that motion is pending within the region when there is not a significant difference between image information for the region in the video frame and image information for the corresponding region in the reference frame, and the period of time which has elapsed since the reference frame was previously updated with image information for the region is not greater than the predetermined threshold.

8. The system of claim 1, and further comprising a memory, and wherein the processor is programmed to store image information representing the background image in the memory at each background update cycle.

9. The system of claim 1, wherein the processor is programmed to perform steps (i)-(iv) to update the background image for the scene at an update period of a predetermined number of video frames.

10. A system for generating background image information of a scene comprising:

(a) a video camera suitable for viewing a scene and generating video signals representing image information for the scene;

(b) a frame grabber connected to the video camera for generating a stream of video frames, each video frame comprising image information for the scene at an instant of time; and

(c) a processor connected to the frame grabber, the processor being programmed to update image information for a background image for the scene during a background update cycle based upon image information for a video frame, by;

(i) partitioning the video frame into a plurality of cell levels each comprising at least one cell, wherein the cells are arranged in a hierarchy from a highest cell level corresponding to an entire video frame and lowest cell level, each cell in a cell level except for the lowest cell level having a plurality of children cells at a next lower cell level which are subdivisions of it;

(ii) analyzing image information for each children cell of a cell to determine whether any children cell is substantially occluded by an object;

(iii) updating a background image based upon image information for the cell if it is determined than at least one children cell of the cell is substantially occluded by an object;

(iv) repeating steps (ii) and (iii) for each of the children cells when it is determined that none of the children cells is substantially occluded by an object.

11. The system of claim 10, wherein the processor is programmed to perform steps (ii)-(iii) beginning with the highest cell level and progressively for lower cell levels when it is determined that a lower cell level is not substantially occluded by an object.

12. The system of claim 10, wherein the processor is programmed to determine whether motion has occurred, has not occurred, or is pending within the cell with respect to image information for the cell retained at a prior background update cycle when it is determined that at least one children cell is substantially occluded by an object.

13. The system of claim 12, wherein the processor is programmed to update the background image with image information for the cell if it is determined that motion has not occurred within the cell.

14. The system of claim 12, wherein the processor is programmed to maintain unchanged image information in the background image for the cell if it is determined that motion has occurred within the cell.

15. The system of claim 12, wherein the processor is programmed to gradually update the background image with image information for the cell if it is determined that motion is pending within the cell.

16. The system of claim 12, wherein the processor is programmed to update the background image when it is determined that at least one of the children cells of the cell is substantially occluded by an object and the cell itself is not at least partially occluded by an object, by:

(v) determining a measure of difference between image information representing the cell in the video frame and image information for a corresponding cell in the background image;

(vi) updating the background image with image information for the cell if the measure of difference is less than a predetermined threshold; and

(vii) gradually updating the background image with the image information for the cell if the measure of difference is greater than the predetermined threshold.

17. The system of claim 12, wherein the processor is programmed to store image information in a reference frame for cells in which during a background update cycle it is determined that motion has occurred or has not occurred, and wherein the processor is programmed to determine whether motion has occurred within the cell, by:

(viii) comparing image information for the cell for the video frame with image information for a corresponding cell in the reference frame;

(ix) determining that motion has occurred within the cell when there is a significant difference between the image information for the cell in the video frame and image information for the corresponding cell in the reference frame;

(x) determining that motion has not occurred within the cell when there is not a significant difference between image information for the cell in the current video frame and image information for the corresponding cell in the reference frame, and a period of time has elapsed since the reference frame was previously updated with image information for the cell, which period of time is greater than a predetermined threshold since the cell was last processed, and storing image information for the cell in the reference frame; and

(xi) determining that motion is pending within the cell when there is not a significant difference between image information for the cell in the current video frame and image information for the corresponding cell in the reference image frame, and the period of time which has elapsed since the reference frame was previously updated with image information for the cell is not greater than the predetermined threshold since the cell was last processed.

18. The system of claim 10, and further comprising a memory, and wherein the processor is programmed to store image information representing the background image in the memory at each background update cycle.

19. The system of claim 10, wherein the processor is programmed to perform steps (i)-(iv) to update the background image for the scene at an update period of a predetermined number of video frames.

20. A method for updating image information for a background image of a scene monitored by a video camera from which is generated a stream of video frames, each comprising image information of the scene at an instant of time, the background image being updated during a background update cycle based upon image information for a video frame, comprising steps of:

(a) selecting a region of the video frame;

(b) analyzing image information for at least one smaller subregion within the region to determine whether the at least one smaller subregion is substantially occluded by an object;

(c) updating a background image based upon image information for the region if it is determined that the at least one subregion within the region is substantially occluded by an object; and

(d) repeating steps (b) and (c) for each of a plurality of smaller subregions within the region when it is determined that the at least one subregion is not substantially occluded by an object.

21. The method of claim 20, wherein the step of updating the background image based upon image information for the region comprises steps of:

(e) determining whether image information for the region indicates that the region is sufficiently occluded by an object;

(f) if it is determined that the region is sufficiently occluded by an object, then determining whether motion has occurred, has not occurred or is pending within the region with respect to image information for the region retained at a prior background update cycle.

22. The method of claim 21, wherein the step of updating the background image comprises updating the background image with image information for the region if it is determined that motion has not occurred within the region.

23. The method of claim 21, wherein the step of updating the background image comprises maintaining unchanged image information in the background image corresponding to the region if it is determined that motion has occurred within the region.

24. The method of claim 21, wherein the step of updating the background image comprises gradually updating the background image with image information for the region if it is determined that motion is pending within the region.

25. The method of claim 21, wherein the step of updating the background image when it is determined that the region is not sufficiently occluded by an object comprises:

(g) determining a measure of difference between image information for the region in the video frame and image information for a corresponding region in the background image;

(h) updating the background image with image information for the region if the measure of difference is less than a predetermined threshold; and

(i) gradually updating the background image with the image information for the region if the measure of difference is greater than the predetermined threshold.

26. The method of claim 21, and further comprising the step of storing image information in a reference frame for regions in which during a background update cycle it is determined that motion has occurred or has not occurred, and wherein the step of determining whether motion has occurred within the region comprises steps of:

j) comparing image information for the region of the video frame with image information for the corresponding region in the reference frame;

(k) determining that motion has occurred within the region when there is a significant difference between image information for the region in the current video frame and image information for the corresponding region in the reference frame;

(l) determining that motion has not occurred within the region when there is not a significant difference between image information for the region in the current video frame and image information for the corresponding region in the reference frame, and a period of time has elapsed since the reference frame was previously updated with image information for the region, which period of time is greater than a predetermined threshold; and

(m) determining that motion is pending within the region when there is not a significant difference between image information for the region in the current video frame and image information for the corresponding region in the reference frame, and the period of time which has elapsed since the reference frame was previously updated with image information for the region is not greater than the predetermined threshold.

27. The method of claim 20, and further comprising a step of storing image information for the background image at each background update cycle.

28. The method of claim 20, wherein steps (a)-(d) are performed to update the background image for the scene at an update period of a predetermined number of video frames.

29. A computer-readable medium storing executable instructions which cause a computer to generate image information for a background image of a scene during a background update cycle based upon a video frame representing image information for the scene at an instant of time, by:

(a) selecting a region of the video frame;

(b) analyzing image information for smaller subregions within the region to determine whether any subregion is substantially occluded by an object;

(c) updating a background image based upon image information for the region if it is determined that any subregion of the region is substantially occluded by an object; and

(d) repeating steps (b) and (c) for each of the subregions when it is determined that none of the subregions is substantially occluded by an object.

30. The computer-readable medium of claim 29, wherein the executable instructions for updating the background image comprises executable instructions for:

(e) determining whether image information for the region indicates that the region is sufficiently occluded by an object;

(f) if it is determined that the region is sufficiently occluded by an object, then determining whether motion has occurred, has not occurred, or is pending within the region with respect to image information for the region retained at a prior background update cycle.

31. The computer-readable medium of claim 30, wherein the executable instructions for updating the background image if it is determined that motion has not occurred within the region comprises executable instructions for updating the background image with image information for the region.

32. The computer-readable medium of claim 30, wherein the executable instructions for updating the background image if it is determined that motion has occurred within the region comprises executable instructions for maintaining unchanged image information in the background image corresponding to the region.

33. The computer-readable medium of claim 30, wherein the executable instructions for updating the background image if it is determined that motion is pending within the region comprises executable instructions gradually updating the background image with image information for the region.

34. The computer-readable medium of claim 30, wherein the executable instructions for updating the background image when it is determined that the region is not sufficiently occluded by an object comprises executable instructions for:

(g) determining a measure of difference between image information for the region in the video frame and image information for a corresponding region in the background image;

(h) updating the background image with image information for the region if the measure of difference is less than a predetermined threshold; and

(i) gradually updating the background image with image information for the region if the measure of difference is greater than the predetermined threshold.

35. The computer-readable medium of claim 30, and further comprising executable instructions for storing image information in a reference frame for regions in which during a background update cycle it is determined that motion has occurred or has not occurred, and wherein the executable instructions for performing the step of determining whether motion has occurred, has not occurred or is pending within a region comprises executable instructions for:

(j) comparing image information for the region of the video frame with image information for the corresponding region in the reference frame;

(k) determining that motion has occurred within the region when there is a significant difference between image information for the region in the video frame and image information for the corresponding region in the reference;

(l) determining that motion has not occurred within the region when there is not a significant difference between image information for the region in the video frame and image information for the corresponding region in the reference frame, and a period of time has elapsed since the reference frame was previously updated with image information for the region, which period of time is greater than a predetermined threshold; and

(m) determining that motion is pending within the region when there is not a significant difference between image information for the region in the video frame and image information for the corresponding region in the reference frame, and the period of time which has elapsed since the reference frame was previously updated with image information for the region is not greater than the predetermined threshold.

36. The computer-readable medium of claim 29, and further comprising executable instructions for causing image information for the background image to be stored in a memory at each background update cycle.

37. The computer-readable medium of claim 29, and further comprising executable instructions for causing steps (a)-(d) to be performed to update the background image for the scene at an update period of a predetermined number of video frames.