Real-time video analysis for security surveillance

• US 9,928,708 B2
• Filed: 12/14/2015
• Issued: 03/27/2018
• Est. Priority Date: 12/12/2014
• Status: Expired due to Fees

• Current Assignee: Shanghai Hanshi Information Technology Company Limited
• Original Assignee: HawXeye, Inc
• Inventors: Lin, Andy, Neblett, Kyle, Savvides, Marios, Singh, Karanhaar, Bhagavatula, Chandrasekhar
• Primary Examiner(s): BELLA, MATTHEW C
• Assistant Examiner(s): Shin, Soo
• Application Number:
  US14/968,714
• Publication Number:
  US 20160171852A1
• Time in Patent Office: 834 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19663 Surveillance related processing done local to the camera

Abstract

A camera and method identify moving objects of interest in a field of view of the camera. The method includes: capturing two or more images successively over a time period, each image being associated with different times during the time period; obtaining binary image from each successive pair of images, the binary image comprising a binary value at each pixel indicating whether a change in pixel values of at least a predetermined magnitude has occurred at that pixel between the time associated with the first image of the success pair of images and time associated with the second image of the successive pair of images; deriving one or more motion boxes each encapsulating one or more nearby pixels in binary image; processing motion boxes of each binary image to obtain refined motion boxes; and classifying refined motion boxes each into categories representative of one moving object of interest.

Claims

1. In a camera, a method for identifying moving objects of interest in a field of view of the camera, comprising:

capturing two or more images successively over a time period, each image being associated with a different time during the time period;

obtaining a binary image from each successive pair of images, the binary image comprising a binary value at each pixel indicating whether or not a change in pixel values of at least a predetermined magnitude has occurred at that pixel between the time associated with the first image of the successive pair of images and time associated with the second image of the successive pair of images;

deriving one or more motion boxes each encapsulating one or more nearby pixels in the binary image;

processing the motion boxes of each binary image to obtain refined motion boxes, wherein the motion boxes are processed using a plurality of modular filters each receiving one or more motion boxes as input and providing one or more processed motion boxes as output, the processed motion boxes of one of the modular filters being suitable to be provided as input to another one of the modular filters, the modular filters comprising one or more tracking filters, each tracking filter tracking motion boxes of the binary images through the time period, and the tracking filter rejects motion boxes that show a displacement over the time period that is less than a predetermined value; and

classifying portions of the successive pair of images corresponding to the refined motion boxes each into a category representative of one of the moving objects of interest.

2. The method of claim 1, further comprising converting pixel values of the captured images to grey scale intensities.

3. The method of claim 1, wherein the successive pair of images are down-sampled prior to obtaining the binary image.

4. The method of claim 1, further comprising performing an image dilation step on the binary image.

5. The method of claim 4, wherein the image dilation step comprises a Manhattan transformation step.

6. The method of claim 4, wherein the image dilation step comprises a radial dilation step.

7. The method of claim 1, further comprises a connectivity analysis step for identifying the nearby pixels.

8. The method of claim 1, wherein the modular filters comprise one or more area limiting filters.

9. The method of claim 1, wherein the modular filters comprise one or more grouping filters.

10. The method of claim 9, wherein the motion boxes are stretched prior to applying the grouping filters.

11. The method of claim 1, wherein the modular filters comprise one or more aspect ratio filters.

12. The method of claim 1, wherein the tracking filter rejects motion boxes that do not persist over a pre-determined number of frames over the time period.

13. The method of claim 1, wherein the tracking filter rejects motion boxes that shows a total distance traversed over the time period that is less than a predetermined value.

14. The method of claim 1, wherein the refined motion boxes are classified according to a linear classification scheme.

15. The method of claim 1, wherein the refined motion boxes are classified according to a latent Dirichlet allocation scheme.

16. The method of claim 1, wherein the refined motion boxes are classified using a neural network implemented in a memory module of the camera.

17. The method of claim 16, wherein the neural network is previously-trained prior to implementation in the memory module of the camera.

18. A camera comprising:

an optical and actuation system;

a communication interface;

non-volatile storage;

a processor; and

a memory module, wherein the memory module, upon initialization, loads from the non-volatile storage instructions to be executed by the processor, the instructions comprising instructions for carrying out a method for identifying moving objects of interest in a field of view of the camera, the method comprising;

capturing two or more images successively over a time period, each image being associated with a different time during the time period;

obtaining a binary image from each successive pair of images, the binary image comprising a binary value at each pixel indicating whether or not a change in pixel values of at least a predetermined magnitude has occurred at that pixel between the time associated with the first image of the successive pair of images and time associated with the second image of the successive pair of images;

deriving one or more motion boxes each encapsulating one or more nearby pixels in the binary image;

processing the motion boxes of each binary image to obtain refined motion boxes, wherein the motion boxes are processed using a plurality of modular filters implemented as instructions executable by the processor, each modular filter receiving one or more motion boxes as input and providing one or more processed motion boxes as output, the processed motion boxes of one of the modular filters being suitable to be provided as input to another one of the modular filters, the modular filters comprising one or more tracking filters, each tracking filter tracking motion boxes of the binary images through the time period, and the tracking filter rejects motion boxes that shows a total distance traversed over the time period that is less than a predetermined value; and

classifying portions of the successive pair of images corresponding to the refined motion boxes each into a category representative of one of the moving objects of interest.

19. The camera of claim 18, wherein the memory module further loads for execution by the processor instructions for carrying out control methods for operating the optical and actuation system, the communication interface and the non-volatile storage.

20. The camera of claim 18, further comprising an infra-red illumination system controlled by the processor.

21. The camera of claim 18, wherein the instructions further comprises instructions for converting pixel values of the captured images to grey scale intensities.

22. The camera of claim 18, wherein the successive pair of images are down-sampled prior to obtaining the binary image.

23. The camera of claim 18, wherein the instructions further comprises instructions for performing an image dilation step on the binary image.

24. The camera of claim 23, wherein the image dilation step comprises a Manhattan transformation step.

25. The camera of claim 23, wherein the image dilation step comprises a radial dilation step.

26. The camera of claim 18, wherein the instructions further comprises instructions for carrying out a connectivity analysis step for identifying the nearby pixels.

27. The camera of claim 18, wherein the modular filters comprise one or more area limiting filters.

28. The camera of claim 18, wherein the modular filters comprise one or more grouping filters.

29. The camera of claim 28, wherein the motion boxes are stretched prior to applying the grouping filters.

30. The camera of claim 18, wherein the modular filters comprise one or more aspect ratio filters.

31. The camera of claim 18, wherein the tracking filter rejects motion boxes that do not persist over a predetermined number of frames over the time period.

32. The camera of claim 18, wherein the tracking filter rejects motion boxes that show a displacement over the time period that is less than a predetermined value.

33. The camera of claim 18, wherein the refined motion boxes are classified according to a linear classification scheme.

34. The camera of claim 18, wherein the refined motion boxes are classified according to a latent Dirichlet allocation scheme.

35. The camera of claim 18, wherein the refined motion boxes are classified using a neural network implemented in a memory module of the camera.

36. The camera of claim 35, wherein the neural network is previously-trained prior to implementation in the memory module of the camera.