Searching recorded video

• US 9,171,075 B2
• Filed: 12/30/2010
• Issued: 10/27/2015
• Est. Priority Date: 12/30/2010
• Status: Active Grant

• Current Assignee: Pelco (Schneider Electric SE)
• Original Assignee: Pelco (Schneider Electric SE)
• Inventors: Millar, Greg, Aghdasi, Farzin, Wang, Lei
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): YANG, WEI WEN
• Application Number:
  US12/982,601
• Publication Number:
  US 20120170803A1
• Time in Patent Office: 1,762 Days
• Field of Search: 382/103, 382/104, 382/159, 382/199
• US Class Current: 1/1
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06V 10/273 removing elements interfering with the pattern to be recognised
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Embodiments of the disclosure provide for systems and methods for creating metadata associated with a video data. The metadata can include data about objects viewed within a video scene and/or events that occur within the video scene. Some embodiments allow users to search for specific objects and/or events by searching the recorded metadata. In some embodiments, metadata is created by receiving a video frame and developing a background model for the video frame. Foreground object(s) can then be identified in the video frame using the background model. Once these objects are identified they can be classified and/or an event associated with the foreground object may be detected. The event and the classification of the foreground object can then be recorded as metadata.

Claims

1. A method for creating metadata associated with a video frame, the method comprising:

receiving a video frame;

developing a background model for the video frame;

separating a foreground object from the video frame using the background model and a ground plane model;

detecting and removing reflections from the video frame responsive to a ratio, of a number of pixels associated with the foreground to a number of pixels associated with the video frame, exceeding a threshold, wherein detecting the reflections comprises

generating a difference image by subtracting an intensity of the video frame from a background image,

generating a histogram of the difference image, and

determining that a reflection is present responsive to a maximum of the histogram being shifted from zero by more than a threshold amount;

classifying the foreground object into a classification; and

recording the classification of the foreground object in metadata.

2. The method according to claim 1, wherein background model comprises a short-term background model.

3. The method according to claim 2, wherein background model further comprises a long-term background model used in conjunction with the short-term background model.

4. The method according to claim 1, wherein background model comprises a long-term background model.

5. The method according to claim 1 further comprising detecting and removing shadows from an image frame.

6. The method according to claim 1 further comprising:

detecting an event associated with the foreground object; and

recording the event in metadata.

7. The method according to claim 6, wherein the foreground object is classified by one or more of object size or object type.

8. The method according to claim 6, wherein the detected event is an event selected from the group consisting of the foreground object entering a frame, the foreground object leaving the frame, camera sabotage, the foreground object holding still, the foreground object merging with another object, the foreground object splitting into two objects, the foreground object entering a zone, the foreground object leaving a zone, the foreground object crossing a tripwire, the foreground object being removed, the foreground object being abandoned, object counting, object loitering, and object timer.

9. The method according to claim 1, wherein the method occurs at a video camera.

10. The method according to claim 1, wherein the foreground object is further classified by object type.

11. The method according to claim 1, wherein the foreground object is further classified by object velocity.

12. The method according to claim 1, wherein the background model is constructed using a probability Gaussian model.

13. The method according to claim 1, wherein separating the foreground object from the video frame comprises subtracting the background model from the video frame to return a foreground object.

14. The method according to claim 1, wherein the classification data associated with the foreground object are recorded as metadata separate from the video data.

32. The method of claim 1, wherein separating the foreground object from the video frame using the background model and the ground plane model comprises:

determining a normalized size of a grouped set of pixels in the video frame using the ground plane model; and

classifying the grouped set of pixels as the foreground object if the normalized size of the grouped set of pixels falls within a predetermined range.

33. The method of claim 1, wherein the classification characterizes the foreground object based on an object color, an aspect ratio, and a position.

34. The method of claim 32, wherein the normalized size of the grouped of pixels is determined based on a position of the grouped set of pixels relative to a vanishing point associated with the ground plane model.

15. A method for creating metadata about a video frame, the method comprising:

receiving a video frame;

developing a background model for the video frame;

separating a foreground object from the video frame using the background model and a ground plane model;

detecting and removing reflections from the video frame responsive to ratio, of a number of pixels associated with the foreground to a number of pixel associated with the video frame, exceeding a threshold, wherein detecting the reflections comprises

generating a difference image by subtracting an intensity of the video frame from a background image,

generating a histogram of the difference image, and

determining that a reflection is present responsive to a maximum of the histogram being shifted from zero by more than a threshold amount;

identifying the foreground object as a previously identified foreground object;

classifying the foreground object into a classification based on characteristics of the foreground object; and

recording the classification of the object in metadata.

16. The method according to claim 15, wherein the video frame comprises a plurality of video frames.

17. The method according to claim 15, wherein the identifying the foreground object includes tracking the foreground object from previous video frames.

18. The method according to claim 15 further comprising determining an event of the foreground object.

19. The method according to claim 18, wherein the event is determined based at least in part on the classification of the foreground object.

20. The method according to claim 18, wherein the event is determined based at least in part on a change in the classification of the foreground object from a classification of the foreground object in a previous frame.

21. The method according to claim 18 further comprising detecting and removing shadows from an image frame.

22. A video camera system comprising:

a video camera for capturing video data of a scene and outputting the video data as digital data;

a processor communicatively coupled to the camera, wherein the processor is configured to

receive video data from the video camera,

develop a background model for the video data,

separate a foreground object from the video data using the background model and a ground plane model,

detect and remove reflections from the video frame responsive to ratio, of a number of pixels associated with the foreground to a number of pixel associated with the video frame, exceeding a threshold, wherein to detect the reflections the processor is configured to

generate a difference image by subtracting an intensity of the video frame from a background image,

generate a histogram of the difference image, and

determine that a reflection is present responsive to a maximum of the histogram being shifted from zero by more than a threshold amount,

identify the foreground object'"'"'s location within the video scene of the video data,

determine the foreground object'"'"'s color,

classify the foreground object,

record the foreground object'"'"'s location and color in metadata; and

an output port communicatively coupled to the processor and configured to output the metadata.

23. The video camera system according to claim 22, wherein background model comprises a short-term background model.

24. The video camera system according to claim 23, wherein background model further comprises a long-term background model used in conjunction with the short-term background model.

25. The video camera system according to claim 22, wherein background model comprises a long-term background model.

26. The video camera system according to claim 22, wherein the output port is configured to output the video data separate from the metadata.

27. The video camera system according to claim 22, wherein the processor is further configured to compress the video data according to a video compression algorithm.

28. The video camera system according to claim 22, wherein the processor is further configured to determine the speed of the foreground object and record the speed of the foreground object in metadata.

29. The video camera system according to claim 22, wherein the processor is further configured to determine an aspect ratio of the foreground object and record the aspect ratio of the foreground object in metadata, and wherein the processor is further configured to classify the foreground object based further on the aspect ratio of the foreground object.

30. A video processing system comprising:

a video input port configured to receive video data;

a processor communicatively coupled to the video input port and configured to;

develop a background model for the video data,

separate a foreground object from the video data using the background model and a ground plane model,

detect and remove reflections from the video frame responsive to ratio, of a number of pixels associated with the foreground to a number of pixel associated with the video frame, exceeding a threshold, wherein to detect the reflections the processor is configured to

generate a difference image by subtracting an intensity of the video frame from a background image,

generate a histogram of the difference image, and

determine that a reflection is present responsive to a maximum of the histogram being shifted from zero by more than a threshold amount,

determine a relative location of the foreground object,

determine a relative color of the foreground object, and

classify the foreground object; and

a video output port communicatively coupled to the processor and configured to output the relative location and the relative color of the foreground object as metadata.

31. The video processing system according to claim 30, wherein the output port is configured to output the video data separate from the metadata.