Target detection and tracking from video streams

• US 20070127774A1
• Filed: 01/31/2007
• Published: 06/07/2007
• Est. Priority Date: 06/24/2005
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: ObjectVideo Incorporated (Alarm.com Holdings, Inc.)
• Inventors: Yu, Li, Zhang, Zhong, Yin, Weihong, Hu, Yongtong, Lipton, Alan, Venetianer, Peter
• Granted Patent: US 7,801,330 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction

Abstract

A technique for video processing includes: receiving video from a scene; detecting moving pixels in the video; detecting line segments or motion blocks in the video based on the detected moving pixels; identifying targets in the video based on the detected line segments or motion blocks; tracking targets in the video based on the identified targets; and managing the tracked targets in the video.

Claims

1. An apparatus for video processing, the apparatus is configured to perform operations comprising:

receiving video of a scene;

detecting moving pixels in the video;

detecting line segments or motion blocks in the video based on the detected moving pixels;

identifying targets in the video based on the detected line segments or motion blocks;

tracking targets in the video based on the identified targets; and

managing the tracked targets in the video.

2. An apparatus as in claim 1, wherein detecting moving pixels comprises:

separating foreground in the video from background in the video; and

detecting edges in the video.

3. An apparatus as in claim 1, wherein detecting each line segment comprises:

counting edge pixels; and

identifying a line segment based on the edge pixels.

4. An apparatus as in claim 3, wherein identifying the line segment comprises:

identifying a start point;

predicting next search directions;

identifying a next line pixel; and

providing a line segment.

5. An apparatus as in claim 1, wherein detecting the motion blocks comprises:

determining coarse motion blocks; and

determining fine motion blocks.

6. An apparatus as in claim 1, wherein tracking targets comprises:

updating existing targets;

detecting new targets;

refining at least one of the updated or new targets;

merging or splitting the targets; and

cleaning the merged or split targets.

7. An apparatus as in claim 6, wherein updating existing targets comprises:

predicting a target;

assigning a line segment to the predicted target; and

updating the target.

8. An apparatus as in claim 6, wherein detecting new targets comprises:

performing line segment clustering;

performing cluster verification based on the line segment clustering; and

generating a new target based on the cluster verification.

9. An apparatus as in claim 6, wherein refining at least one of the updated or new targets comprises:

agglomerating remaining line segments to nearest targets;

re-estimating the targets; and

updating the targets.

10. An apparatus as in claim 6, wherein merging the targets comprises:

obtaining a target pair;

comparing the obtained targets; and

merging the target pair if parameters of the target pair are similar.

11. An apparatus as in claim 6, wherein splitting the targets comprises:

obtaining a target;

comparing the obtained target to a pre-defined target;

splitting the obtained target into sub-targets if the obtained target is not similar to the pre-defined target;

comparing each sub-target to the pre-defined target; and

assigning target identities to each sub-target similar to the pre-defined target.

12. An apparatus as in claim 6, wherein merging or splitting the targets comprises:

generating a foreground mask;

detecting foreground objects based on the foreground mask; and

analyzing the foreground objects to obtain new targets.

13. An apparatus as in claim 12, wherein analyzing the foreground objects comprises at least one of:

comparing the foreground objects to a multiple target size threshold, a minimum single target size threshold, or a maximum single target size threshold.

14. An apparatus as in claim 6, wherein cleaning the targets comprises:

(a) obtaining a target;

(b) verifying the obtained target being detected in a current frame;

(c) verifying the obtained target not to be moving out of a field of view of a video camera in a previous frame; and

one of

(d) maintaining the obtained target or removing the obtained target based at least on one of the step (b) or step (c).

15. An apparatus as in claim 1, wherein tracking targets comprises:

identifying a motion state for each target; and

identifying a transition from the motion state of each target.

16. An apparatus as in claim 15, wherein the motion state for each target is one of a moving state, a stopped state, or a stationary state.

17. An apparatus as in claim 1, wherein tracking targets comprises:

tracking each target based on a motion state history for each target.

18. An apparatus as in claim 1, wherein identifying targets comprises:

performing change detection on the video to obtain a change mask;

performing motion detection on the video to obtain a motion mask;

generating a foreground edge detection region mask based on the change mask and the motion mask;

performing foreground edge detection based on the foreground edge detection region mask to obtain detected foreground edges;

performing line segment detection based on the detected foreground edges to obtain detected line segments;

performing convex region detection based on the detected line segments to obtain detected convex regions; and

generating a foreground region mask based on the detected convex regions.

19. An apparatus as in claim 18, wherein performing the change detection further comprises obtaining a background image, and wherein generating a foreground edge detection region mask comprises:

performing a logical operation on the change mask and the motion mask to obtain a change-or-motion mask;

performing edge detection on the background image to obtain a background edge mask;

performing a logical operation on the background edge mask to obtain a background non-edge mask; and

performing a logical operation on the change-or-motion mask and the background non-edge mask to obtain the foreground edge detection region mask.

20. An apparatus as in claim 19, wherein generating the foreground region mask is based on the detected convex regions and the change-or-motion mask.

21. An apparatus as in claim 18, wherein performing convex region detection comprises:

generating convex regions from line segments;

merging change connecting convex regions; and

merging spatial connecting convex regions.

22. An apparatus as in claim 21, wherein merging change connecting convex regions comprises:

identifying a smallest unprocessed convex region;

identifying valid neighboring convex regions of the smallest unprocessed convex region;

identifying connecting line segments between the valid neighboring convex regions and the smallest unprocessed convex region;

identifying change connecting line segments based on the connecting line segments;

identifying a closest change connecting convex region based on the change connecting line segments; and

merging the closest change connecting convex region and the smallest unprocessed convex region.

23. An apparatus as in claim 1, wherein the apparatus comprises application-specific hardware to emulate a computer and/or software for video processing, the application-specific hardware to perform the operations of claim 1.

24. An apparatus as in claim 1, wherein the apparatus comprises a computer-readable medium comprising software for video processing, which when executed by a computer, causes the computer to perform the operations of claim 1.

25. An apparatus as in claim 1, wherein the apparatus is embodied in a video camera.

26. An apparatus as in claim 1, wherein the apparatus is embodied in a point of sale system.

27. An apparatus as in claim 1, wherein the apparatus is embodied in an access control system.

28. An apparatus as in claim 27, wherein the apparatus is embodied in a tailgating detection.

29. An apparatus as in claim 28, wherein the operations further comprise:

receiving an access code entry;

identifying a video tripwire in the scene;

detecting when one or more identified targets cross the video tripwire; and

matching each detected target with each received access code entry.

30. An apparatus as in claim 29, further comprising:

generating an alarm when the detected target does not match the received access code entry.

31. An apparatus as in claim 1, wherein the apparatus is embodied in at least one of a building management or an energy management system.

32. An apparatus as in claim 31, wherein the operations further comprise:

counting identified targets.

33. An apparatus as in claim 32, wherein the operations further comprise:

monitoring at least one of a lighting, air-conditioning, heating or ventilating system based on the counted targets.

34. An apparatus as in claim 1, wherein the apparatus is embodied in a smart sensor.

35. An apparatus as in claim 1, wherein the apparatus is embodied in a target counting system.

36. An apparatus as in claim 35, wherein the operations further comprise:

detecting at least one of a stationary or moving identified targets in the scene;

counting the detected stationary targets;

estimating a dwell time based on the counted stationary targets; and

recording the estimated dwell time.

37. An apparatus as in claim 36, wherein the operations further comprise:

recording the dwell time in a log.

38. An apparatus as in claim 1, wherein the operations further comprise:

calibrating a video camera from which the video of the scene is received.

39. An apparatus as in claim 38, further comprising:

determining a model of the target;

determining a calibration sample; and

determining calibration parameters based at least on the target model and calibration sample.

40. An apparatus as in claim 39, wherein determining a calibration sample comprises:

providing images of targets on a graphical user interface; and

selecting the calibration sample in the image by a user.

41. An apparatus as in claim 39, wherein determining calibration parameters comprises:

estimating a set of calibration parameters;

estimating a verification target based on the estimated calibration parameters;

determining a calibration verification sample;

comparing the estimated verification target with the calibration verification sample; and

based on the comparison, one of accepting the calibration parameters or estimating a new set of calibration parameters.

42. An apparatus as in claim 39, wherein estimating a set of calibration parameters comprises:

estimating coarse calibration parameters;

iteratively estimating fine calibration parameters based on the estimated coarse calibration parameters;

determining mean matching error based on the estimated fine calibration parameters;

comparing the mean matching error to a predetermined threshold; and

based on the comparison, one of estimating the verification target or requesting a new calibration sample.

43. An apparatus as in claim 39, wherein determining a calibration verification sample comprises:

providing images of targets on a graphical user interface; and

selecting the calibration verification sample in the image by a user, the calibration verification sample being different from the selected calibration sample.

44. An apparatus for video processing, the apparatus is configured to perform operations comprising:

calibrating a video camera;

monitoring an area of interest with the video camera;

receiving video of the area of interest from the video camera;

detecting moving pixels in the video;

detecting line segments or motion blocks in the video based on the detected moving pixels;

identifying targets in the video based on the detected line segments or motion blocks;

tracking targets in the video based on the identified targets; and

managing the tracked targets in the video.

45. An apparatus as in claim 44, further comprising:

determining a model of the target;

determining a calibration sample in a video frame; and

determining calibration parameters based at least on the selected target model and calibration sample.

46. An apparatus as in claim 45, wherein determining calibration parameters comprises:

estimating a set of calibration parameters;

estimating a verification target based on the estimated calibration parameters;

determining a calibration verification sample in the video frame, the calibration verification sample being different from the calibration sample;

comparing the estimated verification target with the determined calibration verification sample; and

one of accepting the calibration parameters or estimating a new set of calibration parameters.

47. An apparatus as in claim 46, wherein estimating a set of calibration parameters comprises:

estimating coarse calibration parameters;

iteratively estimating fine calibration parameters based on the estimated coarse calibration parameters;

determining mean matching error based on the estimated fine calibration parameters;

comparing the mean matching error to a predetermined threshold; and

based on the comparison, one of estimating the verification target or requesting a new calibration sample.

48. A computer system to perform a method for video processing, the method comprising:

receiving video of a scene;

detecting moving pixels in the video;

detecting line segments or motion blocks in the video based on the detected moving pixels;

identifying targets in the video based on the detected line segments or motion blocks;

tracking targets in the video based on the identified targets; and

managing the tracked targets in the video.

49. A system as in claim 48, the method further comprising:

determining a model of the target;

determining a calibration sample in a video frame; and

determining calibration parameters based at least on the selected target model and calibration sample.

50. A system as in claim 49, wherein determining calibration parameters comprises:

estimating a set of calibration parameters;

estimating a verification target based on the estimated calibration parameters;

determining a calibration verification sample in the video frame, the calibration verification sample being different from the calibration sample;

comparing the estimated verification target with the determined calibration verification sample; and

one of accepting the calibration parameters or estimating a new set of calibration parameters.

51. A system as in claim 50, wherein estimating a set of calibration parameters comprises:

estimating coarse calibration parameters;

iteratively estimating fine calibration parameters based on the estimated coarse calibration parameters;

determining mean matching error based on the estimated fine calibration parameters;

comparing the mean matching error to a predetermined threshold; and

based on the comparison, one of estimating the verification target or requesting a new calibration sample.

52. A method for video processing, comprising:

receiving video of a scene;

detecting moving pixels in the video;

detecting line segments or motion blocks in the video based on the detected moving pixels;

identifying targets in the video based on the detected line segments or motion blocks;

tracking targets in the video based on the identified targets; and

managing the tracked targets in the video.

53. A method as in claim 52, further comprising:

determining a model of the target;

determining a calibration sample in a video frame; and

determining calibration parameters based at least on the selected target model and calibration sample.

54. A method as in claim 53, wherein determining calibration parameters comprises:

estimating a set of calibration parameters;

estimating a verification target based on the estimated calibration parameters;

determining a calibration verification sample in the video frame, the calibration verification sample being different from the calibration sample;

comparing the estimated verification target with the determined calibration verification sample; and

one of accepting the calibration parameters or estimating a new set of calibration parameters.

55. A method as in claim 54, wherein estimating a set of calibration parameters comprises:

estimating coarse calibration parameters;

iteratively estimating fine calibration parameters based on the estimated coarse calibration parameters;

determining mean matching error based on the estimated fine calibration parameters;

comparing the mean matching error to a predetermined threshold; and

based on the comparison, one of estimating the verification target or requesting a new calibration sample.

56. A video processing apparatus, comprising:

means for receiving video of a scene;

means for detecting moving pixels in the video;

means for detecting line segments or motion blocks in the video based on the detected moving pixels;

means for identifying targets in the video based on the detected line segments or motion blocks;

means for tracking targets in the video based on the identified targets; and

means for managing the tracked targets in the video.