Feed-customized processing of multiple video streams in a pipeline architecture

• US 7,663,661 B2
• Filed: 10/13/2004
• Issued: 02/16/2010
• Est. Priority Date: 03/16/2004
• Status: Expired due to Fees

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Haupt, Gordon T., Russell, Stephen G., Vallone, Robert P., Fleischer, Stephen D., Freeman, J. Andrew, Kirkman, Thomas W.
• Primary Examiner(s): AN, SHAWN S
• Application Number:
  US10/964,977
• Publication Number:
  US 20060062430A1
• Time in Patent Office: 1,952 Days
• Field of Search: 348/143, 348/153, 348/159, 348/161, 348/154, 348/155, 348/699, 382/103, 382/165, 382/203
• US Class Current: 348/143
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A pipeline architecture for analyzing multiple streams of video is embodied, in part, in a layer of application program interfaces (APIs) to each stage of processing. Buffer queuing is used between some stages, which helps moderate the load on the CPU(s). Through the layer of APIs, innumerable video analysis applications can access and analyze video data flowing through the pipeline, and can annotate portions of the video data (e.g., frames and groups of frames), based on the analyses performed, with information that describes the frame or group. These annotated frames and groups flow through the pipeline to subsequent stages of processing, at which increasingly complex analyses can be performed. At each stage, portions of the video data that are of little or no interest are removed from the video data. Ultimately, “events” are constructed and stored in a database, from which cross-event and historical analyses may be performed and associations with, and among, events may be made.

Claims

1. A volatile or non-volatile storage device storing instructions which, when executed by one or more processors, cause the performance of, at a stage of a multiple stage pipeline;

receiving, at a particular device, from a first analyzer application, a first call to read first video data, wherein the first video data is associated with a first video stream from a first camera;

in response to receiving the first call, providing the first video data to the first analyzer application to allow the first analyzer application to perform, as the first video data passes through the stage of the pipeline, a first set of one or more analysis operations on the first video data to examine the content of the first video data;

receiving, at the particular device, from a second video analyzer application, a second call to read second video data, wherein the second video data is associated with a second video stream from a second camera that is different than the first camera;

in response to receiving the second call, providing the second video data to the second analyzer application to allow the second analyzer application to perform, as the second video data passes through the stage of the pipeline, a second set of one or more analysis operations on the second video data to examine the content of the second video data, wherein the second set of one or more analysis operations are performed concurrently with the first set of one or more analysis operations, wherein the second set of one or more analysis operations differ from the first set of one or more analysis operations.

2. The volatile or non-volatile storage device of claim 1, wherein:

providing the first video data includes providing the first video data to the first analyzer application to allow the first analyzer application to perform a first set of one or more on-the-fly analysis operations on the first video data; and

providing the second video data includes providing the second video data to the second analyzer application to allow the second analyzer application to perform a second set of one or more on-the-fly analysis operations on the second video data.

3. The volatile or non-volatile storage device of claim 1,

wherein the second analyzer application is different than the first analyzer application.

4. The volatile or non-volatile storage device of claim 1, wherein:

providing the first video data includes providing the first video data to the first analyzer application to allow the first analyzer application to perform, the first set of one or more analysis operations on the first video data to examine for a first type of content; and

providing the second video data includes providing the second video data to the second analyzer application to allow the second analyzer application to perform the second set of one or more analysis operations on the second video data to examine for a second type of content that is different than the first type of content.

5. The volatile or non-volatile storage device of claim 4, wherein:

providing the first video data includes providing the first video data to the first analyzer application to allow the first analyzer application to perform, the first set of one or more analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the content of the first video data; and

providing the second video data includes providing the second video data to the second analyzer application to allow the second analyzer application to perform the second set of one or more analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the content of the second video data, wherein the second threshold is different than the first threshold.

6. The volatile or non-volatile storage device of claim 5, wherein the first threshold is associated with the content of the first video data.

7. The volatile or non-volatile storage device of claim 5, wherein the first threshold is associated with a particular camera that captured the first video stream that corresponds to the first video.

8. The volatile or non-volatile storage device of claim 5, wherein the first threshold is stored in a database.

9. The volatile or non-volatile storage device of claim 4, wherein:

providing the first video data includes providing the first video data to the first analyzer application to allow the first analyzer application to perform the first set of one or more analysis operations on the first video data to determine, based on a first threshold, whether or not to generate analytic information about the first type of content; and

providing the second video data includes providing the second video data to the second analyzer application to allow the second analyzer application to perform the second set of one or more analysis operations on the second video data to determine, based on a second threshold, whether or not to generate analytic information about the second type of content.

10. The volatile or non-volatile storage device of claim 1, wherein:

providing the first video data includes providing the first video data to the first analyzer application to allow the first analyzer application to perform, the first set of one or more analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the content of the first video data; and

providing the second video data includes providing the second video data to the second analyzer application to allow the second analyzer application to perform the second set of one or more analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the content of the second video data, wherein the second threshold is a different threshold than the first threshold.

11. The volatile or non-volatile storage device of claim 1, wherein:

providing the first video data includes providing the first video data to the first analyzer application to allow the first analyzer application to perform, the first set of one or more analysis operations on the first video data to determine, based on a first threshold, whether or not to generate analytic information about the content of the first video data; and

providing the second video data includes providing the second video data to the second analyzer application to allow the second analyzer application to perform, the second set of one or more analysis operations on the second video data to determine, based on a second threshold, whether or not to generate analytic information about the content of the second video data, wherein the second threshold is a different threshold than the first threshold.

12. The volatile or non-volatile storage device of claim 1, wherein:

the instructions include additional instructions which, when executed by one or more processors, cause the performance of providing third video data to the first analyzer application to allow the first analyzer application to perform, as the third video data that corresponds to a first plurality of video streams pass through the stage of the multiple stage pipeline, a third set of one or more analysis operations on the third video data to examine the content of the third video data, wherein the first plurality of video streams are from a first camera cluster; and

providing fourth video data to the second analyzer application to allow the second analyzer application to perform, as the fourth video data that corresponds to a second plurality of video streams pass through the stage of the multiple stage pipeline, a fourth set of one or more analysis operations on the fourth video data to examine the content of the fourth video data, wherein the second plurality of video streams are from a second camera cluster.

13. The volatile or non-volatile storage device of claim 12, wherein:

providing fourth video data includes providing the fourth video data to the second analyzer application to allow the second analyzer application to perform the third set of one or more analysis operations concurrently with the first analyzer application performing, the third set of one or more analysis operations.

14. The volatile or non-volatile storage device of claim 12, wherein:

providing third video data includes providing the third video data to the first analyzer application to allow the first analyzer application to perform as the third video data that corresponds to the first plurality of video streams pass through the stage of the multiple stage pipeline, a first set of one or more on-the-fly analysis operations on the third video data to examine the content of the third video data; and

providing fourth video data includes providing the fourth video data to the second analyzer application to allow the second analyzer application to perform as the fourth video data that corresponds to the second plurality of video streams pass through the stage of the multiple stage pipeline, a second set of one or more on-the-fly analysis operations on the fourth video data to examine the content of the fourth video data.

15. The volatile or non-volatile storage device of claim 12, wherein:

the fourth set of one or more analysis operations is different than the third set of one or more analysis operations.

16. The volatile or non-volatile storage device of claim 1, wherein the stage is a quick frame stage of the multiple stage pipeline.

17. The volatile or non-volatile storage device of claim 1, wherein the stage is a deep frame stage of the multiple stage pipeline.

18. The volatile or non-volatile storage device of claim 1, wherein the stage is a cluster stage of the multiple stage pipeline.

19. The volatile or non-volatile storage device of claim 1, wherein the first and second video data are frames of video.

20. The volatile or non-volatile storage device of claim 1, wherein the first and second video data are groups of frames of video.

21. The volatile or non-volatile storage device of claim 1,

wherein:

the first and second video data are analytic information about the content of frames of video; and

the analytic information is (i) about the content of the frames, (ii) based on a prior analysis of the frames, and (iii) different than the frames.

22. A system for analyzing video data that represents one or more video streams, the system comprising:

at a stage of a multiple stage pipeline,

means for performing, by a first analyzer application and as first video data passes through the stage of the pipeline, a first set of one or more analysis operations on the first video data to examine the content of the first video data, wherein the first video data is associated with a first video stream from a first camera; and

means for performing, by a second analyzer application and as second video data passes through the stage of the pipeline, a second set of one or more analysis operations on the second video data to examine the content of the second video data, wherein the second video data is associated with a second video stream from a second camera that is different than the first camera;

wherein the multiple stage pipeline includes a particular machine through which the first and second video passes and with which both the first and second analyzer applications interact to perform the respective first or second set of one or more analysis operations;

wherein the second set of one or more analysis operations differ from the first set of one or more analysis operations.

23. A method performed by one or more processors for analyzing video data that represents one or more video streams, the method comprising:

at a stage of a multiple stage pipeline,

performing, by a first analyzer application and as first video data passes through the stage of the pipeline, a first set of analysis operations on the first video data to examine the content of the first video data, wherein the first video data is associated with a first video stream from a first camera; and

performing, by a second analyzer application and as a second video data passes through the stage of the pipeline, a second set of analysis operations on the second video data to examine the content of the second video data, wherein the second video data is associated with a second video stream from a second camera that is different than the first camera, wherein the second set of one or more analysis operation are performed concurrently with the first set of one or more analysis operations, wherein the second set of one or more analysis operations differ from the first set of one or more analysis operations;

wherein the multiple stage pipeline includes a particular machine through which the first and second video passes and with which both the first and second analyzer applications interact to perform the respective first or second set of one or more analysis operations;

wherein the method is performed by the one or more processors.

24. The method of claim 23, at the stage of the multiple stage pipeline,

performing, by the first analyzer application and as the first video data passes through the stage of the pipeline, a first set of on-the-fly analysis operations on the first video data to examine the content of the first video data;

and

performing, by the second analyzer application and as the second video data passes through the stage of the pipeline, a second set of on-the-fly analysis operations on the second video data to examine the content of the second video data.

25. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 24.

26. The method of claim 23, at the stage of the multiple stage pipeline,

performing the second set of analysis operations by a second analyzer application that is different than the first analyzer application that performs the first set of analysis operations.

27. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 26.

28. The method of claim 23, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to examine for a first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to examine for a second type of content that is different than the first type of content.

29. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 28.

30. The method of claim 28, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the content of the first video data;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the content of the second video data.

31. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 30.

32. The method of claim 30, wherein the first threshold is associated with the content of the first video data.

33. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 32.

34. The method of claim 30, wherein the first threshold is associated with a particular camera that captured a video stream that corresponds to the first video information.

35. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 34.

36. The method of claim 30, wherein the first threshold is stored in a database.

37. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 36.

38. The method of claim 28, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to generate analytic information about the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to generate analytic information about the second type of content.

39. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 38.

40. The method of claim 23, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the first type of content, wherein the second threshold is a different threshold than the first threshold.

41. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 40.

42. The method of claim 23, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to generate analytic information about the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to generate analytic information about the first type of content, wherein the second threshold is a different threshold than the first threshold.

43. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 42.

44. The method of claim 23, at the stage of the multiple stage pipeline,

performing, by the first analyzer application and as first video data that corresponds to one or more first video streams pass through the stage of the pipeline, a first set of analysis operations on the first video data to examine the content of the first video data, wherein the plurality of first video streams are from a first camera cluster;

and

performing, by the second analyzer application and as second video data that corresponds to one or more second video streams pass through the stage of the pipeline, a second set of analysis operations on the second video data to examine the content of the second video data, wherein the plurality of second video streams are from a second camera cluster.

45. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 44.

46. The method of claim 44, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations concurrently with performing, by the second analyzer application, the second set of analysis operations.

47. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 46.

48. The method of claim 44, at the stage of the multiple stage pipeline,

performing, by the first analyzer application and as the first video data pass through the stage of the pipeline, a first set of on-the-fly analysis operations on the first video data to examine the content of the first video data;

and

performing, by the second analyzer application and as the second video data pass through the stage of the pipeline, a second set of on-the-fly analysis operations on the second video data to examine the content of the second video data.

49. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 48.

50. The method of claim 44, at the stage of the multiple stage pipeline,

performing the second set of analysis operations by a second analyzer application that is different than the first analyzer application that performs the first set of analysis operations.

51. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 50.

52. The method of claim 23, at the stage of the multiple stage pipeline,

performing, by the first analyzer application, the first set of analysis operations on the first video data to determine, based on a first threshold, whether or not to perform an action in response to examining the first type of content;

and

performing, by the second analyzer application, the second set of analysis operations on the second video data to determine, based on a second threshold, whether or not to perform an action in response to examining the first type of content, wherein the second threshold is a different threshold than the first threshold.

53. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 52.

54. The method of claim 23, wherein the first and second sets of analysis operations are performed at a quick frame stage of the multiple stage pipeline.

55. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 54.

56. The method of claim 23, wherein the first and second sets of analysis operations are performed at a deep frame stage of the multiple stage pipeline.

57. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 56.

58. The method of claim 23, wherein the first and second sets of analysis operations are performed at a cluster stage of the multiple stage pipeline.

59. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 58.

60. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 23.

61. The method of claim 23, wherein the first set of one or more analysis operations are of a different type relative to the second set of one or more analysis operations.

62. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 61.

63. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 61.

64. The method of claim 23, wherein the first set of one or more analysis operations are of a same type relative to the second set of one or more analysis operations.

65. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 64.

66. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 64.