Pipeline architecture for analyzing multiple video streams

• US 7,697,026 B2
• Filed: 10/13/2004
• Issued: 04/13/2010
• Est. Priority Date: 03/16/2004
• Status: Expired due to Fees

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Russell, Stephen G., Vallone, Robert P., Freeman, J. Andrew
• Primary Examiner(s): Rao; Andy S.
• Assistant Examiner(s): Pe; Geepy
• Application Number:
  US10/965,687
• Publication Number:
  US 20050232462A1
• Time in Patent Office: 2,008 Days
• Field of Search: 348143-161, 348/61, 382/103
• US Class Current: 348/143
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A pipeline architecture for analyzing multiple streams of video is embodied, in part, in a layer of application program interfaces (APIs) to each stage of processing. Buffer queuing is used between some stages, which helps moderate the load on the CPU(s). Through the layer of APIs, innumerable video analysis applications can access and analyze video data flowing through the pipeline, and can annotate portions of the video data (e.g., frames and groups of frames), based on the analyses performed, with information that describes the frame or group. These annotated frames and groups flow through the pipeline to subsequent stages of processing, at which increasingly complex analyses can be performed. At each stage, portions of the video data that are of little or no interest are removed from the video data. Ultimately, “events” are constructed and stored in a database, from which cross-event and historical analyses may be performed and associations with, and among, events may be made.

Claims

1. A method of processing video data that represents one or more video streams, the method comprising:

at a first stage of a multiple stage pipeline, providing a first application programming interface (API) through which one or more first analyzers are allowed to perform a first set of real-time analysis operations on the video data;

based on the first set of real-time analysis operations, filtering the video data to produce filtered video data, wherein the filtered video data contains a subset of the video data; and

at a second stage of the multiple stage pipeline, providing a second API through which one or more second analyzers are allowed to perform a second set of analysis operations on the filtered video data;

wherein the second set of analysis operations are more complex than the first set of analysis operations;

wherein the one or more first analyzers are one or more first analysis applications that are external to the multiple stage pipeline and are plugged into the first stage of the multiple stage pipeline via the first API;

wherein the one or more second analyzers are one or more second analysis applications that are external to the multiple stage pipeline and are plugged into the second stage of the multiple stage pipeline via the second API.

2. The method of claim 1, wherein the one or more video streams includes one or more streams of surveillance video.

3. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 1.

4. A system for processing video data that represents one or more video streams, the system comprising:

at a first stage of a multiple stage pipeline, a first application programming interface (API) means for allowing one or more first analyzers to perform a first set of real-time analysis operations on the video data;

means for filtering the video data, based on the first set of real-time analysis operations, to produce filtered video data, wherein the filtered video data contains fewer frames than the video data; and

at a second stage of the multiple stage pipeline, a second API means for allowing one or more second analyzers to perform a second set of analysis operations on the filtered video data;

wherein the second set of analysis operations are more complex than the first set of analysis operations;

wherein the one or more first analyzers are one or more first analysis applications that are external to the multiple stage pipeline and are plugged into the first stage of the multiple stage pipeline via the first API;

wherein the one or more second analyzers are one or more second analysis applications that are external to the multiple stage pipeline and are plugged into the second stage of the multiple stage pipeline via the second API.

5. A method of processing video data that represents one or more video streams, the method comprising:

receiving the video data at a first stage of processing of a multiple stage pipeline;

at the first stage of processing, providing an application programming interface (API) designed to allow, by one or more analysis applications that are external to the multiple stage pipeline and are plugged into the first stage of the multiple stage pipeline via the API;

(a) real-time analysis of frames that correspond to each video stream of the one or more video streams, and

(b) reduction of the video data by removing a plurality of frames from the video data that, at the first stage, was determined to not contain content of interest; and

sending remaining video data to a second stage of processing for further analysis.

6. The method of claim 5, wherein the one or more video streams include a stream of surveillance video.

7. The method of claim 5, wherein the step of sending the remaining video data to the second stage includes,

queuing the video data in a buffer.

8. The method of claim 5, further comprising:

at the first stage of processing, providing multiple processing threads, wherein each thread is configured to process only one video stream from one camera.

9. The method of claim 5,

wherein the step of receiving video data includes receiving video data that represents multiple streams;

wherein the step of providing an interface includes providing an interface that is designed to allow analysis of frames that correspond to more than one of the multiple streams and reduction of the video data for more than one of the multiple streams; and

wherein the step of sending includes sending video data that represents more than one of the multiple streams to a second stage of processing for further analysis.

10. The method of claim 9, wherein the step of sending the video data to the second stage includes,

queuing the video data that represents each stream in corresponding buffers.

11. The method of claim 10, further comprising the step of:

at the second stage of processing, providing multiple processing threads, wherein each of the multiple processing threads is configured to read information stored in any one or more of the buffers.

12. The method of claim 10, wherein the step of providing an interface includes providing the interface to one or more analysis applications to allow access to the buffers to read the respective video data, and wherein the interface is designed to allow, for each buffer, analysis of a frame by each of the one or more analysis applications before allowing any of the analysis applications to analyze a subsequent frame in the video data that represents the same stream.

13. The method of claim 5, further comprising:

at the first stage of processing, providing an interface designed to allow

(c) annotation of a frame that corresponds to a stream, with first analytic information about the frame, wherein the first analytic information is generated based on the analysis, at the first stage, of the frame; and

wherein the step of sending includes sending the video data, including the first analytic information, to the second stage of processing for further analysis.

14. The method of claim 13, wherein the step of sending the remaining video data to the second stage includes,

queuing the remaining video data, including the first analytic information, in a buffer that is accessible from the second stage.

15. The method of claim 13, wherein the step of providing includes providing, at the first stage of processing, an interface designed to allow

analysis and annotation by a plurality of video analysis applications, and

generation, by each of the plurality of applications, separate first analytic information that corresponds to the frame.

16. The method of claim 13, further comprising:

at the second stage of processing, providing an interface designed to allow

(a) analysis of the first analytic information,

(b) further reduction of the video data by removing frames that, at the second stage, were determined to not contain content of interest,

(c) annotation of a frame with second analytic information about the frame, wherein the analytic information is generated based on the analysis, at the second stage, of at least the first analytic information; and

sending remaining video data, including the second analytic information, to a third stage of processing for further analysis.

17. The method of claim 13, further comprising:

at the second stage of processing, providing an interface designed to allow

(a) analysis of a group of frames,

(b) generation of event information that describes the group of frames, wherein the event information is based on analysis, at the second stage, of the frames in the group; and

storing the event information in a data repository.

18. The method of claim 5, further comprising:

at the second stage of processing, providing an interface designed to allow

(a) analysis of frames that correspond to each video data,

(b) further reduction of the video data by removing a portion of the video data that, at the second stage, was determined to not contain content of interest, and

(c) annotation of a frame with analytic information about the frame, wherein the analytic information is generated based on the analysis, at the second stage, of the frame; and

sending remaining video data, including the analytic information, to a third stage of processing for further analysis.

19. The method of claim 18, wherein the step of sending the remaining video data to the third stage includes,

queuing the video data, including the analytic information, in a buffer.

20. The method of claim 18, wherein the step of providing includes providing, at the second stage of processing, an interface designed to allow

analysis and annotation by a plurality of video analysis applications, and

generation, by each of the plurality of applications, separate analytic information that corresponds to the frame.

21. The method of claim 18, further comprising:

at the third stage of processing, providing an interface designed to allow

(a) analysis of a group of frames,

(b) generation of event information that describes the group of frames, wherein the event information is based on analysis, at the third stage, of the frames in the group; and

storing the event information in a data repository.

22. The method of claim 21, wherein the step of providing includes providing an interface that is designed to allow analysis of a group of frames from a single stream.

23. The method of claim 21, wherein the step of providing includes providing an interface that is designed to allow analysis of a group of frames from multiple streams, wherein each of the multiple streams originates from a different camera.

24. The method of claim 18, further comprising:

at the third stage of processing, providing an interface designed to allow

(a) analysis of analytic information about frames in a group of frames,

(b) generation of event information that describes the group of frames, wherein the event information is based at least in part on the analytic information about frames in the group; and

storing the event information in a data repository.

25. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 5.

26. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 6.

27. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 7.

28. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 8.

29. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 9.

30. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 10.

31. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 11.

32. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 12.

33. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 13.

34. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 14.

35. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 15.

36. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 16.

37. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 17.

38. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 18.

39. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 19.

40. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 20.

41. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 21.

42. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 22.

43. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 23.

44. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 24.

45. A method for analyzing video data that represents one or more video streams, the method comprising computer-implemented steps of:

at a first stage of processing in a multiple stage pipeline,

reading, by a video analyzer application that is external to the multiple stage pipeline and that is plugged into the first stage of the multiple stage pipeline via an application programming interface (API), video data that represents one or more video streams;

analyzing, by the video analyzer application, frame data that corresponds to each of the one or more video streams; and

for one or more of the video streams, annotating, by the video analyzer application, at least one corresponding frame with metadata, wherein the metadata includes analytic information about the content of the frame, wherein the analytic information is generated based on analyzing the corresponding frame; and

outputting, by the video analyzer application, the metadata to allow the multiple stage pipeline to provide the metadata to a second stage of processing in the multiple stage pipeline for further analysis of the frame by one or more other video analyzer applications that are external to the multiple stage pipeline and that are plugged into the second stage of the multiple stage pipeline.

46. The method of claim 45, wherein the step of analyzing frame data includes analyzing one or more frames that correspond to a stream.

47. The method of claim 45, wherein the step of analyzing frame data includes analyzing information about one or more frames that correspond to a stream.

48. The method of claim 45, wherein the further analysis is more computationally complex than the analyzing at the first stage.

49. The method of claim 45, wherein the one or more video streams includes one or more streams of surveillance video.

50. The method of claim 45, further comprising the computer-implemented step of:

dividing one or more of the streams into groups of frames, wherein the frames in a group of frames have a similar characteristic;

annotating each group of frames with annotation information to identify which frames are contained in the group of frames; and

outputting the annotation information for providing to the second stage of processing for further analysis of the group of frames.

51. The method of claim 50, wherein the annotation information identifies the start time of the group of frames.

52. A machine-readable storage medium encoded with a computer program, the machine readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 45.

53. The method of claim 45, wherein:

the analytic information describes what was found to be interesting about the frame; and

an application program at the second stage of processing is allowed access to the analytic information to determine, based at least in part on the analytic information, what type of analysis to perform on the frame.

54. A method of processing video data that represents one or more video streams, the method comprising:

receiving the video data at a particular stage of processing of a multiple stage pipeline;

at the particular stage of processing, providing an application programming interface (API) designed to allow, by one or more analysis applications that are external to the multiple stage pipeline and are plugged into the particular stage of the multiple stage pipeline via the API;

(a) analysis of a plurality of frames that correspond to each video stream of the one or more video streams,

(b) reduction of the video data by removing one or more of the plurality of frames from the video data that, at the particular stage, were determined to not contain content of interest, and

(c) annotation of one or more frames, of the plurality of frames, with metadata that includes analytic information about the content of the one or more frames; and

sending remaining video data to a subsequent stage of the multiple stage pipeline for further analysis.