System and method for detecting interactions of people and vehicles

• US 6,570,608 B1
• Filed: 08/24/1999
• Issued: 05/27/2003
• Est. Priority Date: 09/30/1998
• Status: Expired due to Term

• Current Assignee: Texas Instruments, Inc.
• Original Assignee: Texas Instruments, Inc.
• Inventors: Tserng, Christopher
• Primary Examiner(s): Philippe, Gims S.
• Application Number:
  US09/379,715
• Time in Patent Office: 1,372 Days
• Field of Search: H04/N.718, 348/142, 348/143, 348/144, 348/148, 348/149, 348/154, 348/155, 348/161, 348/169, 348/170, 348/171, 348/172, 348/586, 348/700, 348/699, 382/100, 382/192, 382/203, 382/283, 340/541
• US Class Current: 348/143
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

A video surveillance system that implements object detection and event recognition employing smart monitoring algorithms to analyze a video stream and recognize the interaction of people with cars. The system forms a reference image consisting of the background of the scene viewed by the video camera. The system forms a foreground difference image between the current image and an updated reference image including any stationary cars. The system detects any objects other than any stationary car in the scene. The updated reference image is formed by forming a background difference image between the current image and the reference image. Any detected object is examined to determine if it is a car object. Upon initial detection of a stationary car object, the system forms a reference car image of a portion of the current image corresponding to the position of the stationary car. The system then forms the updated reference image by substituting this reference car image for a corresponding portion of the reference image. A number of special techniques preserve object integrity when objects overlap cars. The video surveillance system notes events when objects interact with cars such as an ENTER CAR event and an EXIT CAR event.

Claims

1. A method for detecting interactions between people and vehicles from a video stream of a sequence of images of a scene comprising the steps of:

forming a reference image consisting of a background of the scene;

forming an updated reference image including any stationary car objects including

forming a background difference image between a current image and said reference image,

detecting objects in said difference image,

determining if any detected object in said difference image is a car object,

determining whether any car objects are stationary,

upon initial determination that a detected car object becomes stationary

forming a reference car image of a portion of said current image corresponding to a position of any stationary car object, and

substituting said reference car image for a corresponding portion of said reference image, thereby forming said updated reference image;

forming a foreground difference image between a current image and said updated reference image; and

detecting any blobs in said foreground difference image thereby detecting objects other than any stationary car objects in the scene.

2. The method of claim 1, wherein:

said step of determining whether any detected object in said difference image is a car object includes determining a detected object is a car object if said detected object has a height greater than 3.5 feet and a width of greater than 11.0 feet.

3. The method of claim 1, wherein:

said step of determining whether any detected object in said difference image is a car object includes determining a detected object is a car object if said detected object has greater than a minimum size and a height/width aspect ratio of less than a predetermined aspect ratio.

4. The method of claim 3, wherein:

said minimum size includes a height greater than 3.5 feet and a width of greater than 11.0 feet; and

said predetermined aspect ratio 1 s 0.7.

5. The method of claim 1, further comprising:

tracking a position of each detected object in an immediately prior image to a corresponding detected object of a current image; and

if an object of said immediately prior image overlaps a car object of said immediately prior image for a first image when said car object is determined to be stationary, then said step of forming an updated reference image includes forming a fake background in said updated reference image having black pixels at all pixel locations of said overlapping object in said immediately prior image.

6. The method of claim 5, further comprising:

determining if a stationary car object in an immediately prior image begins to move in a current image;

upon determination that a car object begins to move in said current image, checking for validity any detected object by

determining if any pixels of each detected object includes black pixels corresponding to said fake background, and

declaring said detected object as false and not tracking any object in an immediately prior image to said detected object of a current image if said determined number of black pixels corresponding to fake background is greater than 70% of all pixels of said detected object.

7. The method of claim 1, further comprising the steps of:

determining whether a previously stationary car object begins to move by

comparing a stationary car object in a prior image with a corresponding car object in a current image,

calculating a prior centroid of said previously stationary car object in the prior image,

calculating a current centroid of said corresponding car object in the current image, and

determining said previously stationary car object has begun to move if said prior centroid and said current centroid differ by more than a predetermined amount.

8. The method of claim 7, wherein:

said predetermined amount is 6.5 pixels.

9. The method of claim 1, further comprising the steps of:

upon detection of a stationary car object forming a car background image of a portion of said reference image corresponding to the position of said reference car image;

determining whether a previously stationary car object begins to move;

upon determining that a previously stationary car object has begun to move

removing detected objects overlapping the now moving car object, and

substituting said reference car image for a corresponding portion of said updated reference image, thereby restoring said reference image.

10. The method of claim 1, further comprising the steps of:

determining whether a previously stationary car object begins to move;

upon determining that a previously stationary car object has begun to move

determining if any detected object corresponds to an object overlapping a car object in an immediately prior image, and

checking for validity any such detected overlapping object.

11. The method of claim 10, further comprising:

tracking a position of each detected object in an immediately prior image to a corresponding detected object of a current image; and

said step of checking for validity includes

determining whether a track of objects in prior images corresponding to any detected overlapping object is less than a predetermined number of images,

declaring said detected overlapping object as false and not tracking any object in an immediately prior image to said detected overlapping object of a current image if said track of corresponding objects in prior images is less than said predetermined number of images.

12. The method of claim 11, further comprising:

if said track of objects in prior images corresponding to any detected overlapping object is greater than or equal to said predetermined number of images

reviewing corresponding objects along said track of said object back track until a prior object is found having a width of less than 3 feet which is not changing substantially between images,

storing said prior object,

when said car object moves whereby said detected overlapping object no longer overlaps said car object comparing said prior object to said corresponding object in said current image, and

confirming tracking of said prior object to said object in said current image if said prior object substantially matches said object in said current image.

13. A method for detecting interactions between people and vehicles from a video stream of a sequence of images of a scene comprising the steps of:

forming a reference image consisting of a background of the scene;

forming a background difference image between a current image and said reference image;

detecting objects in said background difference image;

determining if any detected object in said background difference image is a car object;

determining whether any car objects are stationary;

forming an updated reference image including any stationary car objects;

forming a foreground difference image between a current image and said updated reference image;

detecting any blobs in said foreground difference image thereby detecting objects other than any stationary car objects in the scene; and

checking for validity any objects newly detected in said current image by

determining a number of pixels of any newly detected object in said foreground difference image matching corresponding pixels in said car difference image, and

declaring said newly detected object as false and not tracking any object in an immediately prior image to said newly detected object of a current image if said determined number of matching pixels is less than half of all pixels of said detected newly moving overlapping object.

14. A method for detecting interactions between people and vehicles from a video stream of a sequence of images of a scene comprising the steps of:

forming a reference image consisting of a background of the scene;

forming a background difference image between a current image and said reference image;

detecting objects in said background difference image;

determining if any detected object in said background difference image is a car object;

determining whether any car objects are stationary;

forming an updated reference image including any stationary car objects;

forming a foreground difference image between a current image and said updated reference image;

detecting any blobs in said foreground difference image thereby detecting objects other than any stationary car objects in the scene; and

determining an ENTER CAR event whenever any detected object overlaps a position of any stationary cars in one image in the video stream and a corresponding detected object is not present in a next following image in the video stream.

15. A method for detecting interactions between people and vehicles from a video stream of a sequence of images of a scene comprising the steps of:

forming a reference image consisting of a background of the scene;

forming a background difference image between a current image and said reference image;

detecting objects in said background difference image;

determining if any detected object in said background difference image is a car object;

determining whether any car objects are stationary;

forming an updated reference image including any stationary car objects;

forming a foreground difference image between a current image and said updated reference image;

detecting any blobs in said foreground difference image thereby detecting objects other than any stationary car objects in the scene; and

determining an EXIT CAR event whenever a detected object overlaps a position of any stationary car in one image in the video stream and a corresponding detected object is not present in an immediately prior image in the video stream.