Correlation processing among multiple analyzers of video streams at stages of a pipeline architecture

• US 20050259846A1
• Filed: 10/13/2004
• Published: 11/24/2005
• Est. Priority Date: 03/16/2004
• Status: Active Grant

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Haupt, Gordon T., Russell, Stephen G., Vallone, Robert P., Fleischer, Stephen D., Freeman, J. Andrew
• Granted Patent: US 7,664,183 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding

Abstract

A pipeline architecture for analyzing multiple streams of video is embodied, in part, in a layer of application program interfaces (APIs) to each stage of processing. Buffer queuing is used between some stages, which helps moderate the load on the CPU(s). Through the layer of APIs, innumerable video analysis applications can access and analyze video data flowing through the pipeline, and can annotate portions of the video data (e.g., frames and groups of frames), based on the analyses performed, with information that describes the frame or group. These annotated frames and groups flow through the pipeline to subsequent stages of processing, at which increasingly complex analyses can be performed. At each stage, portions of the video data that are of little or no interest are removed from the video data. Ultimately, “events” are constructed and stored in a database, from which cross-event and historical analyses may be performed and associations with, and among, events may be made.

Claims

1. A method for analyzing video data that represents one or more video streams, the method comprising:

providing an interface designed to allow, at a stage of a multiple stage pipeline, performing, by a first analyzer application, a first set of analysis operations on first video data that corresponds to a video stream, to examine the content of the first video data;

performing, by a second analyzer application that is different than the first analyzer application, a second set of analysis operations on second video data that corresponds to a video stream, to examine the content of the second video data; and

evaluating a condition that is based on the first set of analysis operations and the second set of analysis operations.

2. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a subsequent stage of the pipeline.

3. The method of claim 2, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

determining, further based on an occurrence that is external to the video data, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a subsequent stage of the pipeline.

4. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline, performing the second set of analysis operations on the second video data, which is the same video data as the first video data.

5. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline, performing the second set of analysis operations on the second video data, which is different than the first video data.

6. The method of claim 5, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing the second set of analysis operations on the second video data, which correspond to the same stream as the first video data.

7. The method of claim 5, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing the second set of analysis operations on the second video data, which correspond to a different stream than the first video data.

8. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

performing the first set of analysis operations to examine a first type of content;

and

performing the second set of analysis operations to examine a second type of content that is different than the first type of content.

9. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

applying a first weight to results of the first set of analysis operations, producing first weighted results;

applying a second weight to results of the second set of analysis operations, producing second weighted results; and

evaluating a condition that is based on the first weighted results and the second weighted results.

10. The method of claim 9, wherein the step of providing includes providing an interface designed to allow, at the stage of the multiple stage pipeline,

determining, based on the evaluating that is based on the first weighted results and the second weighted results, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a subsequent stage of the pipeline.

11. The method of claim 1, wherein the step of providing includes providing the interface at a quick frame stage of the multiple stage pipeline.

12. The method of claim 11, wherein the step of providing includes providing an interface designed to allow, at the quick frame stage,

determining, based on the evaluating, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a deep frame stage of the multiple stage pipeline.

13. The method of claim 1, wherein the step of providing includes providing the interface at a deep frame stage of the multiple stage pipeline.

14. The method of claim 13, wherein the step of providing includes providing an interface designed to allow, at the deep frame stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a cluster stage of the pipeline.

15. The method of claim 1, wherein the step of providing includes providing the interface at a cluster stage of the multiple stage pipeline.

16. The method of claim 1, wherein the step of providing includes providing an interface designed to allow, at the cluster stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a data repository.

17. The method of claim 1, wherein the first video data represents one or more first frames of video and the second video data represents one or more second frames of video.

18. The method of claim 1, wherein the first video data represents one or more first groups of frames of video and the second video data represents one or more second groups of frames of video.

19. The method of claim 1, wherein the first video data represents analytic information about the content of one or more first frames of video and the second video data represents analytic information about the content of one or more second frames of video.

20. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 1.

21. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 2.

22. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 3.

23. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 4.

24. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 5.

25. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 6.

26. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 7.

27. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 8.

28. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 9.

29. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 10.

30. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 11.

31. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 12.

32. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 13.

33. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 14.

34. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 15.

35. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 16.

36. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 17.

37. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 18.

38. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 19.

39. A system for analyzing video data that represents one or more video streams, the system comprising:

at a stage of a multiple stage pipeline,

means for performing, by a first analyzer application, a first set of analysis operations on first video data that corresponds to a video stream, to examine the content of the first video data;

means for performing, by a second analyzer application that is different than the first analyzer application, a second set of analysis operations on second video data that corresponds to a video stream, to examine the content of the second video data; and

means for evaluating a condition that is based on the first set of analysis operations and the second set of analysis operations.

40. The system of claim 39, further comprising,

at the stage of the multiple stage pipeline,

means for determining, based on the evaluating, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a subsequent stage of the pipeline.

41. A method for analyzing video data that represents one or more video streams, the method comprising:

at a stage of a multiple stage pipeline,

performing, by a first analyzer application, a first set of analysis operations on first video data that corresponds to a video stream, to examine the content of the first video data;

performing, by a second analyzer application that is different than the first analyzer application, a second set of analysis operations second video data that corresponds to a video stream, to examine the content of the second video data; and

evaluating a condition that is based on the first set of analysis operations and the second set of analysis operations.

42. The method of claim 41, at the stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a subsequent stage of the pipeline.

43. The method of claim 42, at the stage of the multiple stage pipeline,

determining, further based on an occurrence that is external to the video data, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a subsequent stage of the pipeline

44. The method of claim 41, at the stage of the multiple stage pipeline,

performing the second set of analysis operations on the second video data, which is the same as the first video data.

45. The method of claim 41, at the stage of the multiple stage pipeline,

performing the second set of analysis operations on the second video data, which is different than the first video data.

46. The method of claim 45, at the stage of the multiple stage pipeline,

performing the second set of analysis operations on the second video data, which correspond to the same stream as the first video data.

47. The method of claim 45, at the stage of the multiple stage pipeline, performing the second set of analysis operations on the second video data, which correspond to a different stream than the first video data.

48. The method of claim 41, at the stage of the multiple stage pipeline,

performing the first set of analysis operations to examine a first type of content;

and

performing the second set of analysis operations to examine a second type of content that is different than the first type of content.

49. The method of claim 41, at the stage of the multiple stage pipeline,

applying a first weight to results of the first set of analysis operations, producing first weighted results;

applying a second weight to results of the second set of analysis operations, producing second weighted results; and

evaluating a condition that is based on the first weighted results and the second weighted results.

50. The method of claim 49, at the stage of the multiple stage pipeline,

determining, based on the evaluating that is based on the first weighted results and the second weighted results, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a subsequent stage of the pipeline.

51. The method of claim 41, wherein the first and second sets of analysis operations and the step of evaluating are performed at a quick frame stage of the multiple stage pipeline.

52. The method of claim 51, at the quick frame stage,

determining, based on the evaluating, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a deep frame stage of the multiple stage pipeline.

53. The method of claim 41, wherein the first and second sets of analysis operations and the step of evaluating are performed at a deep frame stage of the multiple stage pipeline.

54. The method of claim 53, at the deep frame stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a cluster stage of the pipeline.

55. The method of claim 41, wherein the first and second sets of analysis operations and the step of evaluating are performed at a cluster stage of the multiple stage pipeline.

56. The method of claim 41, at the cluster stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a data repository.

57. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 41.

58. A machine-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 42.

59. A method for analyzing video data that represents one or more video streams, the method comprising:

at a stage of a multiple stage pipeline,

in response to

(a) performance of, by a first analyzer application, a first set of analysis operations on first video data that corresponds to a video stream, to examine the content of the first video data, and

(b) performance of, by a second analyzer application that is different than the first analyzer application, a second set of analysis operations on second video data that corresponds to a video stream, to examine the content of the second video data,

evaluating a condition that is based on results of the first set of analysis operations and results of the second set of analysis operations.

60. The method of claim 59, at the stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a subsequent stage of the pipeline.

61. The method of claim 59, at the stage of the multiple stage pipeline,

applying a first weight to results of the first set of analysis operations, producing first weighted results;

applying a second weight to results of the second set of analysis operations, producing second weighted results; and

evaluating a condition that is based on the first weighted results and the second weighted results.

62. The method of claim 61, at the stage of the multiple stage pipeline,

determining, based on the evaluating that is based on the first weighted results and the second weighted results, whether or not to generate information about the content of the video data on which the analysis operations were performed, for sending to a subsequent stage of the pipeline.