Feed-customized processing of multiple video streams in a pipeline architecture

• US 20060062430A1
• Filed: 10/13/2004
• Published: 03/23/2006
• Est. Priority Date: 03/16/2004
• Status: Active Grant

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Haupt, Gordon T., Russell, Stephen G., Vallone, Robert P., Fleischer, Stephen D., Freeman, J. Andrew, Kirkman, Thomas W.
• Granted Patent: US 7,663,661 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A pipeline architecture for analyzing multiple streams of video is embodied, in part, in a layer of application program interfaces (APIs) to each stage of processing. Buffer queuing is used between some stages, which helps moderate the load on the CPU(s). Through the layer of APIs, innumerable video analysis applications can access and analyze video data flowing through the pipeline, and can annotate portions of the video data (e.g., frames and groups of frames), based on the analyses performed, with information that describes the frame or group. These annotated frames and groups flow through the pipeline to subsequent stages of processing, at which increasingly complex analyses can be performed. At each stage, portions of the video data that are of little or no interest are removed from the video data. Ultimately, “events” are constructed and stored in a database, from which cross-event and historical analyses may be performed and associations with, and among, events may be made.

Claims

1. A method for analyzing video data that represents one or more video streams, the method comprising:

providing an interface designed to allow, at a stage of a multiple stage pipeline,

performing, by a first analyzer application and as first video data passes through the stage of the pipeline, a first set of analysis operations on the first video data to examine the content of the first video data, wherein the first video data is associated with a video stream from a first camera; and

performing, by a second analyzer application and as second video data passes through the stage of the pipeline, a second set of analysis operations on the second video data to examine the content of the second video data, wherein the second video data is associated with a video stream from a second camera.

2. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations concurrently with performing, by the second analyzer application, the second set of analysis operations.

3. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application and as the first video data passes through the stage of the pipeline, a first set of on-the-fly analysis operations on the first video data to examine the content of the first video data;

and

performing, by the second analyzer application and as the second video data passes through the stage of the pipeline, a second set of on-the-fly analysis operations on the second video data to examine the content of the second video data.

4. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing the second set of analysis operations by a second analyzer application that is different than the first analyzer application that performs the first set of analysis operations.

5. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to examine for a first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to examine for a second type of content that is different than the first type of content.

6. The method of claim 5, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the content of the first video data;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the content of the second video data.

7. The method of claim 6, wherein the first threshold is associated with the content of the first video data.

8. The method of claim 6, wherein the first threshold is associated with a particular camera that captured a video stream that corresponds to the first video information.

9. The method of claim 6, wherein the first threshold is stored in a database.

10. The method of claim 5, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to generate analytic information about the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to generate analytic information about the second type of content.

11. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the first type of content, wherein the second threshold is a different threshold than the first threshold.

12. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to generate analytic information about the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to generate analytic information about the first type of content, wherein the second threshold is a different threshold than the first threshold.

13. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to examine the content of the first video data;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to examine the content of the second video data, wherein the second set of analysis operation are different than the first set of analysis operations.

14. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application and as first video data that corresponds to one or more first video streams pass through the stage of the pipeline, a first set of analysis operations on the first video data to examine the content of the first video data, wherein the plurality of first video streams are from a first camera cluster;

and

performing, by the second analyzer application and as second video data that corresponds to a second one or more video streams pass through the stage of the pipeline, a second set of analysis operations on the second video data to examine the content of the second video data, wherein the plurality of second video streams are from a second camera cluster.

15. The method of claim 14, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations concurrently with performing, by the second analyzer application, the second set of analysis operations.

16. The method of claim 14, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application and as the first video data that corresponds to the plurality of first video streams pass through the stage of the pipeline, a first set of on-the-fly analysis operations on the first video data to examine the content of the first video data;

and

performing, by the second analyzer application and as the second video data that corresponds to the plurality of second video streams pass through the stage of the pipeline, a second set of on-the-fly analysis operations on the second video data to examine the content of the second video data.

17. The method of claim 14, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing the second set of analysis operations by a second analyzer application that is different than the first analyzer application that performs the first set of analysis operations.

18. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the first type of content, wherein the second threshold is a different threshold than the first threshold.

19. The method of claim 1, wherein the step of providing includes providing the interface at a quick frame stage of the multiple stage pipeline.

20. The method of claim 1, wherein the step of providing includes providing the interface at a deep frame stage of the multiple stage pipeline.

21. The method of claim 1, wherein the step of providing includes providing the interface at a cluster stage of the multiple stage pipeline.

22. The method of claim 1, wherein the first and second video data are frames of video.

23. The method of claim 1, wherein the first and second video data are groups of frames of video.

24. The method of claim 1, wherein the first and second video data are analytic information about the content of associated video.

25. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 1.

26. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 2.

27. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 3.

28. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 4.

29. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 5.

30. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 6.

31. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 7.

32. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 8.

33. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 9.

34. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 10.

35. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 11.

36. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 12.

37. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 13.

38. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 14.

39. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 15.

40. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 16.

41. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 17.

42. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 18.

43. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 19.

44. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 20.

45. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 21.

46. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 22.

47. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 23.

48. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 24.

49. A system for analyzing video data that represents one or more video streams, the system comprising:

at a stage of a multiple stage pipeline,

means for performing, by a first analyzer application and as first video data passes through the stage of the pipeline, a first set of analysis operations on the first video data to examine the content of the first video data, wherein the first video data is associated with a video stream from a first camera; and

means for performing, by a second analyzer application and as second video data passes through the stage of the pipeline, a second set of analysis operations on the second video data to examine the content of the second video data, wherein the second video data is associated with a video stream from a second camera.

50. A method for analyzing video data that represents one or more video streams, the method comprising:

at a stage of a multiple stage pipeline,

performing, by a first analyzer application and as first video data passes through the stage of the pipeline, a first set of analysis operations on the first video data to examine the content of the first video data, wherein the first video data is associated with a video stream from a first camera; and

performing, by a second analyzer application and as a second video data passes through the stage of the pipeline, a second set of analysis operations on the second video data to examine the content of the second video data, wherein the second video data is associated with a video stream from a second camera.

51. The method of claim 50, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations concurrently with performing, by the second analyzer application, the second set of analysis operations.

52. The method of claim 50, at the stage of the multiple stage pipeline,

performing, by the first analyzer application and as the first video data passes through the stage of the pipeline, a first set of on-the-fly analysis operations on the first video data to examine the content of the first video data;

and

performing, by the second analyzer application and as the second video data passes through the stage of the pipeline, a second set of on-the-fly analysis operations on the second video data to examine the content of the second video data.

53. The method of claim 50, at the stage of the multiple stage pipeline,

performing the second set of analysis operations by a second analyzer application that is different than the first analyzer application that performs the first set of analysis operations.

54. The method of claim 50, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to examine for a first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to examine for a second type of content that is different than the first type of content.

55. The method of claim 54, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the content of the first video data;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the content of the second video data.

56. The method of claim 55, wherein the first threshold is associated with the content of the first video data.

57. The method of claim 55, wherein the first threshold is associated with a particular camera that captured a video stream that corresponds to the first video information.

58. The method of claim 55, wherein the first threshold is stored in a database.

59. The method of claim 54, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to generate analytic information about the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to generate analytic information about the second type of content.

60. The method of claim 50, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the first type of content, wherein the second threshold is a different threshold than the first threshold.

61. The method of claim 50, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to generate analytic information about the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to generate analytic information about the first type of content, wherein the second threshold is a different threshold than the first threshold.

62. The method of claim 50, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to examine the content of the first video data;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to examine the content of the second video data, wherein the second set of analysis operation are different than the first set of analysis operations.

63. The method of claim 50, at the stage of the multiple stage pipeline,

performing, by the first analyzer application and as first video data that corresponds to one or more first video streams pass through the stage of the pipeline, a first set of analysis operations on the first video data to examine the content of the first video data, wherein the plurality of first video streams are from a first camera cluster;

and

performing, by the second analyzer application and as second video data that corresponds to one or more second video streams pass through the stage of the pipeline, a second set of analysis operations on the second video data to examine the content of the second video data, wherein the plurality of second video streams are from a second camera cluster.

64. The method of claim 63, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations concurrently with performing, by the second analyzer application, the second set of analysis operations.

65. The method of claim 63, at the stage of the multiple stage pipeline,

performing, by the first analyzer application and as the first video data pass through the stage of the pipeline, a first set of on-the-fly analysis operations on the first video data to examine the content of the first video data;

and

performing, by the second analyzer application and as the second video data pass through the stage of the pipeline, a second set of on-the-fly analysis operations on the second video data to examine the content of the second video data.

66. The method of claim 63, at the stage of the multiple stage pipeline,

performing the second set of analysis operations by a second analyzer application that is different than the first analyzer application that performs the first set of analysis operations.

67. The method of claim 50, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the first type of content, wherein the second threshold is a different threshold than the first threshold.

68. The method of claim 50, wherein the first and second sets of analysis operations are performed at a quick frame stage of the multiple stage pipeline.

69. The method of claim 50, wherein the first and second sets of analysis operations are performed at a deep frame stage of the multiple stage pipeline.

70. The method of claim 50, wherein the first and second sets of analysis operations are performed at a cluster stage of the multiple stage pipeline.

71. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 50.