Target detection and tracking from video streams

• US 7,801,330 B2
• Filed: 01/31/2007
• Issued: 09/21/2010
• Est. Priority Date: 06/24/2005
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: ObjectVideo Incorporated (Alarm.com Holdings, Inc.)
• Inventors: Yu, Li, Zhang, Zhong, Venetianer, Peter L., Lipton, Alan J., Yin, Weihong, Hu, Yongtong
• Primary Examiner(s): Carter; Aaron W
• Application Number:
  US11/700,007
• Publication Number:
  US 20070127774A1
• Time in Patent Office: 1,329 Days
• Field of Search: 382/103, 382/104, 382/107, 382/197, 382/228, 348143-160, 348169-172, 345/419
• US Class Current: 382/103
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction

Abstract

A technique for video processing includes: receiving video from a scene; detecting moving pixels in the video; detecting line segments or motion blocks in the video based on the detected moving pixels; identifying targets in the video based on the detected line segments or motion blocks; tracking targets in the video based on the identified targets; and managing the tracked targets in the video.

Claims

1. A method for video processing, comprising:

determining a set of calibration parameters for a video camera, comprising;

selecting a calibration sample in a video frame,

estimating coarse calibration parameters,

iteratively estimating fine calibration parameters based on the estimated coarse calibration parameters,

determining mean matching error based on the estimated fine calibration parameters,

comparing the mean matching error to a predetermined threshold,

based on the comparison, estimating a verification target,

selecting a calibration verification sample in the video frame, the calibration verification sample being different from the selected calibration sample,

comparing the estimated verification target with the selected calibration verification sample, and

one of accepting the calibration parameters or determining a new set of calibration parameters;

receiving video of a scene from the video camera;

detecting moving pixels in the video;

detecting line segments or motion blocks in the video based on the detected moving pixels;

identifying targets in the video based on the detected line segments or motion blocks;

tracking targets in the video based on the identified targets; and

managing the tracked targets in the video;

wherein a computer or application-specific hardware performs said receiving video, detecting moving pixels, detecting line segments or motion blocks, identifying targets, tracking targets, and managing the tracked targets.

3. The method of claim 1, wherein said detecting line segments comprises:

counting edge pixels; and

identifying a line segment based on the edge pixels by identifying a start point, predicting next search directions, identifying a next line pixel, and providing a line segment.

4. The method of claim 1, wherein said detecting motion blocks comprises:

determining coarse motion blocks; and

determining fine motion blocks.

5. The method of claim 1, wherein said tracking targets comprises:

(A) updating existing targets by;

predicting a target,

assigning a line segment to the predicted target, and

updating the target;

(B) detecting new targets by;

performing line segment clustering,

performing cluster verification based on the line segment clustering, and

generating a new target based on the cluster verification;

(C) refining at least one of the updated or new targets by;

agglomerating remaining line segments to nearest targets,

re-estimating the targets, and

updating the targets;

(D) merging or splitting the updated targets, wherein merging the updated targets is performed by;

obtaining a target pair,

comparing the obtained targets, and

merging the target pair if parameters of the target pair are similar; and

wherein splitting the updated targets is performed by;

obtaining a target,

comparing the obtained target to a pre-defined target,

splitting the obtained target into sub-targets if the obtained target is not similar to the pre-defined target,

comparing each sub-target to the pre-defined target, and assigning target identities to each sub-target similar to the pre-defined target;

(E) merging or splitting the new targets by;

generating a foreground mask,

detecting foreground objects based on the foreground mask, and

analyzing the foreground objects to obtain new targets by comparing the foreground objects to a multiple target size threshold, a minimum single target size threshold, or a maximum single target size threshold; and

(F) cleaning the merged or split targets by;

(a) obtaining a target,

(b) verifying the obtained target being detected in a current frame,

(c) verifying the obtained target not to be moving out of a field of view of a video camera in a previous frame, and one of

(d) maintaining the obtained target or removing the obtained target based at least on one of the step (b) or step (c).

6. The method of claim 1, wherein said tracking targets comprises:

identifying a motion state for each target; and

identifying a transition from the motion state of each target, wherein the motion state for each target is one of a moving state, a stopped state, or a stationary state.

7. The method of claim 1, wherein said tracking targets comprises:

tracking each target based on a motion state history for each target.

8. The method of claim 1, wherein said identifying targets comprises:

performing change detection on the video to obtain a change mask and a background image;

performing motion detection on the video to obtain a motion mask;

generating a foreground edge detection region mask based on the change mask and the motion mask by;

performing a logical operation on the change mask and the motion mask to obtain a change-or-motion mask;

performing edge detection on the background image to obtain a background edge mask;

performing a logical operation on the background edge mask to obtain a background non-edge mask; and

performing a logical operation on the change-or-motion mask and the background non-edge mask to obtain the foreground edge detection region mask;

performing foreground edge detection based on the foreground edge detection region mask to obtain detected foreground edges;

performing line segment detection based on the detected foreground edges to obtain detected line segments;

performing convex region detection based on the detected line segments to obtain detected convex regions; and

generating a foreground region mask based on the detected convex regions and the change-or-motion mask.

9. The method of claim 8, wherein said performing convex region detection comprises:

generating convex regions from line segments;

merging change connecting convex regions by;

identifying a smallest unprocessed convex region,

identifying valid neighboring convex regions of the smallest unprocessed convex region,

identifying connecting line segments between the valid neighboring convex regions and the smallest unprocessed convex region,

identifying change connecting line segments based on the connecting line segments,

identifying a closest change connecting convex region based on the change connecting line segments, and

merging the closest change connecting convex region and the smallest unprocessed convex region; and

merging spatial connecting convex regions.

10. The method of claim 1, further comprising:

receiving an access code entry;

identifying a video tripwire in the scene;

detecting when one or more identified targets cross the video tripwire;

matching each detected target with each received access code entry; and

generating an alarm when the detected target does not match the received access code entry.

11. The method of claim 1, further comprising:

counting the identified targets; and

monitoring and adjust at least one of a lighting, air-conditioning, heating or ventilating system based on the counted targets.

12. The method of claim 1, further comprising:

detecting at least one of a stationary or moving identified targets in the scene;

counting the detected stationary targets;

estimating a dwell time based on the counted stationary targets;

recording the estimated dwell time; and

recording the dwell time in a log.

13. The method of claim 1, further comprising:

using a video camera to obtain the video of the scene.

14. The method of claim 1, further comprising:

using a graphical user interface to provide images of targets and facilitate a selection of a calibration sample and a calibration verification sample by a user in a video frame, the calibration verification sample being different from the selected calibration sample.

15. The method of claim 1, further comprising:

using a camera calibration module to facilitate a selection of a calibration sample and a calibration verification sample in a video frame, the calibration verification sample being different from the calibration sample.

2. A non-transitory computer-readable medium comprising software for video processing, the medium comprising:

instructions for determining a set of calibration parameters for a video camera, comprising;

instructions for receiving a calibration sample from a video frame,

instructions for estimating coarse calibration parameters,

instructions for iteratively estimating fine calibration parameters based on the estimated coarse calibration parameters,

instructions for determining mean matching error based on the estimated fine calibration parameters,

instructions for comparing the mean matching error to a predetermined threshold,

instructions for estimating a verification target based on the comparison,

instructions for receiving a calibration verification sample from the video frame, the calibration verification sample being different from the calibration sample,

instructions for comparing the estimated verification target with the determined calibration verification sample, and

instructions for one of accepting the calibration parameters or determining a new set of calibration parameters;

instructions for receiving video of a scene from the video camera;

instructions for detecting moving pixels in the video;

instructions for detecting line segments or motion blocks in the video based on the detected moving pixels;

instructions for identifying targets in the video based on the detected line segments or motion blocks;

instructions for tracking targets in the video based on the identified targets; and

instructions for managing the tracked targets in the video.