Pipeline architecture for analyzing multiple video streams

• US 20050232462A1
• Filed: 10/13/2004
• Published: 10/20/2005
• Est. Priority Date: 03/16/2004
• Status: Active Grant

• Current Assignee: Identiv, Inc.
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Russell, Stephen G., Vallone, Robert P., Freeman, J. Andrew
• Granted Patent: US 7,697,026 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A pipeline architecture for analyzing multiple streams of video is embodied, in part, in a layer of application program interfaces (APIs) to each stage of processing. Buffer queuing is used between some stages, which helps moderate the load on the CPU(s). Through the layer of APIs, innumerable video analysis applications can access and analyze video data flowing through the pipeline, and can annotate portions of the video data (e.g., frames and groups of frames), based on the analyses performed, with information that describes the frame or group. These annotated frames and groups flow through the pipeline to subsequent stages of processing, at which increasingly complex analyses can be performed. At each stage, portions of the video data that are of little or no interest are removed from the video data. Ultimately, “events” are constructed and stored in a database, from which cross-event and historical analyses may be performed and associations with, and among, events may be made.

Claims

1. A method of processing video data that represents one or more video streams, the method comprising:

at a first stage of a multiple stage pipeline, allowing one or more first analyzers to perform a first set of on-the-fly analysis operations on the video data;

based on the first set of on-the-fly analysis operations, filtering the video data to produce filtered video data, wherein the filtered video data contains a subset of the video data; and

at a second stage of the multiple stage pipeline, allowing one or more second analyzers to perform a second set of on-the-fly analysis operations on the filtered video data;

wherein the second set of analysis operations are more complex than the first set of analysis operations.

2. The method of claim 1, wherein the one or more video streams includes one or more streams of surveillance video.

3. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 1.

4. A system for processing video data that represents one or more video streams, the system comprising:

at a first stage of a multiple stage pipeline, means for allowing one or more first analyzers to perform a first set of on-the-fly analysis operations on the video data;

means for filtering the video data, based on the first set of on-the-fly analysis operations, to produce filtered video data, wherein the filtered video data contains fewer frames than the video data; and

at a second stage of the multiple stage pipeline, means for allowing one or more second analyzers to perform a second set of on-the-fly analysis operations on the filtered video data;

wherein the second set of analysis operations are more complex than the first set of analysis operations.

5. A method of processing video data that represents one or more video streams, the method comprising:

receiving video data that represents one or more video streams at a first stage of processing;

at the first stage of processing, providing an interface designed to allow

(a) analysis of frames that correspond to each stream, and

(b) reduction of the video data by removing a portion of the video data that, at the first stage, was determined to not contain content of interest; and

sending remaining video data to a second stage of processing for further analysis.

6. The method of claim 5, wherein the one or more video streams include a stream of surveillance video.

7. The method of claim 5, wherein the step of sending the remaining video data to the second stage includes,

queuing the video data in a buffer.

8. The method of claim 5, further comprising:

at the first stage of processing, providing multiple processing threads, wherein each thread is configured to process only one video stream from one camera.

9. The method of claim 5,

wherein the step of receiving video data includes receiving video data that represents multiple streams;

wherein the step of providing an interface includes providing an interface that is designed to allow analysis of frames that correspond to more than one of the multiple streams and reduction of the video data for more than one of the multiple streams; and

wherein the step of sending includes sending video data that represents more than one of the multiple streams to a second stage of processing for further analysis.

10. The method of claim 9, wherein the step of sending the video data to the second stage includes,

queuing the video data that represents each stream in corresponding buffers.

11. The method of claim 10, further comprising the step of:

at the second stage of processing, providing multiple processing threads, wherein each of the multiple processing threads is configured to read information stored in any one or more of the buffers.

12. The method of claim 10, wherein the step of providing an interface includes providing the interface to one or more analysis applications to allow access to the buffers to read the respective video data, and wherein the interface is designed to allow, for each buffer, analysis of a frame by each of the one or more analysis applications before allowing any of the analysis applications to analyze a subsequent frame in the video data that represents the same stream.

13. The method of claim 5, further comprising:

at the first stage of processing, providing an interface designed to allow

(c) annotation of a frame that corresponds to a stream, with first analytic information about the frame, wherein the first analytic information is generated based on the analysis, at the first stage, of the frame; and

wherein the step of sending includes sending the video data, including the first analytic information, to the second stage of processing for further analysis.

14. The method of claim 13, wherein the step of sending the remaining video data to the second stage includes,

queuing the remaining video data, including the first analytic information, in a buffer that is accessible from the second stage.

15. The method of claim 13, wherein the step of providing includes providing, at the first stage of processing, an interface designed to allow

analysis and annotation by a plurality of video analysis applications, and

generation, by each of the plurality of applications, separate first analytic information that corresponds to the frame.

16. The method of claim 13, further comprising:

at the second stage of processing, providing an interface designed to allow

(a) analysis of the first analytic information,

(b) further reduction of the video data by removing frames that, at the second stage, were determined to not contain content of interest,

(c) annotation of a frame with second analytic information about the frame, wherein the analytic information is generated based on the analysis, at the second stage, of at least the first analytic information; and

sending remaining video data, including the second analytic information, to a third stage of processing for further analysis.

17. The method of claim 13, further comprising:

at the second stage of processing, providing an interface designed to allow

(a) analysis of a group of frames,

(b) generation of event information that describes the group of frames, wherein the event information is based on analysis, at the second stage, of the frames in the group; and

storing the event information in a data repository.

18. The method of claim 5, further comprising:

at the second stage of processing, providing an interface designed to allow

(a) analysis of frames that correspond to each video data,

(b) further reduction of the video data by removing a portion of the video data that, at the second stage, was determined to not contain content of interest, and

(c) annotation of a frame with analytic information about the frame, wherein the analytic information is generated based on the analysis, at the second stage, of the frame; and

sending remaining video data, including the analytic information, to a third stage of processing for further analysis.

19. The method of claim 18, wherein the step of sending the remaining video data to the third stage includes,

queuing the video data, including the analytic information, in a buffer.

20. The method of claim 18, wherein the step of providing includes providing, at the second stage of processing, an interface designed to allow

analysis and annotation by a plurality of video analysis applications, and

generation, by each of the plurality of applications, separate analytic information that corresponds to the frame.

21. The method of claim 18, further comprising:

at the third stage of processing, providing an interface designed to allow

(a) analysis of a group of frames,

(b) generation of event information that describes the group of frames, wherein the event information is based on analysis, at the third stage, of the frames in the group; and

storing the event information in a data repository.

22. The method of claim 21, wherein the step of providing includes providing an interface that is designed to allow analysis of a group of frames from a single stream.

23. The method of claim 21, wherein the step of providing includes providing an interface that is designed to allow analysis of a group of frames from multiple streams, wherein each of the multiple streams originates from a different camera.

24. The method of claim 18, further comprising:

at the third stage of processing, providing an interface designed to allow

(a) analysis of analytic information about frames in a group of frames,

(b) generation of event information that describes the group of frames, wherein the event information is based at least in part on the analytic information about frames in the group; and

storing the event information in a data repository.

25. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 5.

26. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 6.

27. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 7.

28. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 8.

29. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 9.

30. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 10.

31. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 11.

32. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 12.

33. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 13.

34. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 14.

35. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 15.

36. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 16.

37. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 17.

38. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 18.

39. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 19.

40. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 20.

41. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 21.

42. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 22.

43. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 23.

44. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 24.

45. A method for analyzing video data that represents one or more video streams, the method comprising computer-implemented steps of:

at a first stage of processing in a multiple stage pipeline,

reading video data that represents one or more video streams;

analyzing frame data that corresponds to each of the streams; and

for one or more of the streams, annotating at least one corresponding frame with analytic information about the frame, wherein the analytic information is generated based on analyzing the corresponding frame; and

outputting the analytic information for providing to a second stage of processing for further analysis of the frame.

46. The method of claim 45, wherein the step of analyzing frame data includes analyzing one or more frames that correspond to a stream.

47. The method of claim 45, wherein the step of analyzing frame data includes analyzing information about one or more frames that correspond to a stream.

48. The method of claim 45, wherein the further analysis is more computationally complex than the analyzing at the first stage.

49. The method of claim 45, wherein the one or more video streams includes one or more streams of surveillance video.

50. The method of claim 45, further comprising the computer-implemented step of:

dividing one or more of the streams into groups of frames, wherein the frames in a group of frames have a similar characteristic;

annotating each group of frames with annotation information to identify which frames are contained in the group of frames; and

outputting the annotation information for providing to the second stage of processing for further analysis of the group of frames.

51. The method of claim 50, wherein the annotation information identifies the start time of the group of frames.

52. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 45.