Correlation processing among multiple analyzers of video streams at stages of a pipeline architecture

• US 7,664,183 B2
• Filed: 10/13/2004
• Issued: 02/16/2010
• Est. Priority Date: 03/16/2004
• Status: Expired due to Fees

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Haupt, Gordon T., Russell, Stephen G., Vallone, Robert P., Fleischer, Stephen D., Freeman, J. Andrew
• Primary Examiner(s): AN, SHAWN S
• Application Number:
  US10/965,675
• Publication Number:
  US 20050259846A1
• Time in Patent Office: 1,952 Days
• Field of Search: 348/143, 348/153, 348/159, 348/161, 348/154, 348/155, 348/699, 382/103, 382/165, 382/203, 375/240.26
• US Class Current: 375/240.26
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding

Abstract

A pipeline architecture for analyzing multiple streams of video is embodied, in part, in a layer of application program interfaces (APIs) to each stage of processing. Buffer queuing is used between some stages, which helps moderate the load on the CPU(s). Through the layer of APIs, innumerable video analysis applications can access and analyze video data flowing through the pipeline, and can annotate portions of the video data (e.g., frames and groups of frames), based on the analysis performed, with information that describes the frame or group. These annotated frames and groups flow through the pipeline to subsequent stages of processing, at which increasingly complex analysis can be performed. At each stage, portions of the video data that are of little or no interest are removed from the video data. Ultimately, “events” are constructed and stored in a database, from which cross-event and historical analysis may be performed and associations with, and among, events may be made.

Claims

1. A volatile or non-volatile storage device storing instructions which, when executed by one or more processors, cause, at a stage of a multiple stage pipeline:

receiving, from a first video analyzer application a first call to read first video data that corresponds to a first video stream;

in response to the first call, providing the first video data to the first video analyzer application to allow the first video analyzer application to perform a first set of one or more analysis operations on the first video data to examine the content of the first video data and to generate a first result;

receiving, from a second video analyzer application that is different than the first video analyzer application, a second call to read second video data that corresponds to a second video stream; and

in response to the second call, providing the second video data to the second video analyzer application to allow;

the second video analyzer application to perform a second set of one or more analysis operations on the second video data to examine the content of the second video data and to generate a second result, and

a third video analyzer application to evaluate a condition that is based on both the first result and the second result.

2. The volatile or non-volatile storage device of claim 1, wherein providing the second video data includes providing the second video data to the second video analyzer application to allow the third video analyzer application to determine, based on the evaluating, whether or not to generate information about the content of the first or second video data on which the first set or second set of one or more analysis operations were performed, for sending to a subsequent stage of the multiple stage pipeline.

3. The volatile or non-volatile storage device of claim 2, wherein providing the second video data includes providing the second video data to the second video analyzer application to allow the third video analyzer application to determine, further based on an occurrence that is external to the first and second video data, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to the subsequent stage of the multiple stage pipeline.

4. The volatile or non-volatile storage device of claim 1, wherein the second video data is the same video data as the first video data.

5. The volatile or non-volatile storage device of claim 1, wherein the second video data is different than the first video data.

6. The volatile or non-volatile storage device of claim 5, wherein the second video stream is the same stream as the first video stream.

7. The volatile or non-volatile storage device of claim 5, wherein the second video stream is a different stream than the first video stream.

8. The volatile or non-volatile storage device of claim 1, wherein the content of the first video data is of a first type and the content of the second video data is of a second type that is different than the first type.

9. The volatile or non-volatile storage device of claim 1, wherein providing the second video data includes providing the second video data to the second video analyzer application to allow:

a fourth video analyzer application to apply a first weight to the first result, producing first weighted results;

the fourth video analyzer application to apply a second weight to the second result, producing second weighted results; and

the third video analyzer application to evaluate a condition that is based on the first weighted results and the second weighted results.

10. The volatile or non-volatile storage device of claim 9, wherein providing the second video data includes providing the second video data to the second video analyzer application to allow the third video analyzer application to determine, based on the evaluating that is based on the first weighted results and the second weighted results, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to the subsequent stage of the multiple stage pipeline.

11. The volatile or non-volatile storage device of claim 1, wherein the stage is a quick frame stage of the multiple stage pipeline.

12. The volatile or non-volatile storage device of claim 11, wherein providing the second video data includes providing the second video data to the second video analyzer application to allow the third video analyzer application to determine, based on the evaluating, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a deep frame stage of the multiple stage pipeline.

13. The volatile or non-volatile storage device of claim 1, wherein the stage is a deep frame stage of the multiple stage pipeline.

14. The volatile or non-volatile storage device of claim 13, wherein providing the second video data includes providing the second video data to the second video analyzer application to allow the third video analyzer application to determine, based on the evaluating, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a cluster stage of the pipeline.

15. The volatile or non-volatile storage device of claim 1, wherein the stage is a cluster stage of the multiple stage pipeline.

16. The volatile or non-volatile storage device of claim 1, wherein providing the second video data includes providing the second video data to the second video analyzer application to allow the third video analyzer application to determine, based on the evaluating, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a data repository.

17. The volatile or non-volatile storage device of claim 1, wherein the first video data represents one or more first frames of video and the second video data represents one or more second frames of video.

18. The volatile or non-volatile storage device of claim 1, wherein the first video data represents one or more first groups of frames of video and the second video data represents one or more second groups of frames of video.

19. The volatile or non-volatile storage device of claim 1, wherein:

the first video data represents first analytic information;

the first analytic is (i) about the content of one or more first frames of video, (ii) based on a prior analysis of the one or more frames, and (iii) different than the one or more frames; and

the second video data represents analytic information about the content of one or more second frames of video.

20. A system for analyzing video data that represents one or more video streams, the system comprising:

at a stage of a multiple stage pipeline,

means for performing, by a first analyzer application, a first set of one or more analysis operations on first video data that corresponds to a first video stream, to examine the content of the first video data;

means for performing, by a second analyzer application that is different than the first analyzer application, a second set of one or more analysis operations on second video data that corresponds to a second video stream, to examine the content of the second video data; and

wherein performing the first set of one or more analysis operations generates a first result and performing the second set of one or more analysis operations generates a second result;

means for evaluating a condition that is based on both the first result and the second result.

21. The system of claim 20, further comprising,

at the stage of the multiple stage pipeline,

means for determining, based on the evaluating, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a subsequent stage of the multiple stage pipeline.

22. A method performed by one or more processors for analyzing video data that represents one or more video streams, the method comprising the steps of:

at a stage of a multiple stage pipeline,

performing, by a first analyzer application, a first set of analysis operations on first video data that corresponds to a video stream, to examine the content of the first video data;

performing, by a second analyzer application that is different than the first analyzer application, a second set of analysis operations on second video data that corresponds to a video stream, to examine the content of the second video data; and

wherein performing the first set of analysis operations generates a first result and performing the second set of analysis operations generates a second result;

evaluating a condition that is based on both the first result and the second result;

wherein the steps are performed by the one or more processors.

23. The method of claim 22, further comprising, at the stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a subsequent stage of the multiple stage pipeline.

24. The method of claim 23, further comprising, at the stage of the multiple stage pipeline,

determining, further based on an occurrence that is external to the first and second video data, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a subsequent stage of the multiple stage pipeline.

25. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 24.

26. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 23.

27. The method of claim 22, wherein the second video data is the same as the first video data.

28. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 27.

29. The method of claim 22, wherein the second video data is different than the first video data.

30. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 29.

31. The method of claim 29, wherein the second video stream is the same stream as the first video stream.

32. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 31.

33. The method of claim 29, wherein the second video data is a different stream than the first video stream.

34. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 33.

35. The method of claim 22, wherein content of the first video data is of a first type and the content of the second video data is of a second type that is different than the first type.

36. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 35.

37. The method of claim 22, further comprising, at the stage of the multiple stage pipeline,

applying a first weight to the first result, producing first weighted results;

applying a second weight to the second result, producing second weighted results; and

evaluating a condition that is based on the first weighted results and the second weighted results.

38. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 37.

39. The method of claim 37 further comprising, at the stage of the multiple stage pipeline,

determining, based on the evaluating that is based on the first weighted results and the second weighted results, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a subsequent stage of the multiple stage pipeline.

40. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 39.

41. The method of claim 22, wherein the first and second sets of one or more analysis operations and the step of evaluating are performed at a quick frame stage of the multiple stage pipeline.

42. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 41.

43. The method of claim 41, further comprising, at the quick frame stage,

determining, based on the evaluating, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a deep frame stage of the multiple stage pipeline.

44. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 43.

45. The method of claim 22, wherein the first and second sets of one or more analysis operations and the step of evaluating are performed at a deep frame stage of the multiple stage pipeline.

46. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 45.

47. The method of claim 45, further comprising, at the deep frame stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a cluster stage of the multiple stage pipeline.

48. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 47.

49. The method of claim 22, wherein the first and second sets of one or more analysis operations and the step of evaluating are performed at a cluster stage of the multiple stage pipeline.

50. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 49.

51. The method of claim 22, at the cluster stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a data repository.

52. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 51.

53. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 22.

54. A method performed by one or more processors for analyzing video data that represents one or more video streams, the method comprising the steps of:

at a stage of a multiple stage pipeline,

in response to

(a) performance of, by a first analyzer application, a first set of one or more analysis operations on first video data that corresponds to a first video stream, to examine the content of the first video data; and

(b) performance of, by a second analyzer application that is different than the first analyzer application, a second set of one or more analysis operations on second video data that corresponds to a video stream, to examine the content of the second video data,

evaluating a condition that is based on first results of the first set of one or more analysis operations and the second results of the second set of one or more analysis operations;

wherein the steps are performed by the one or more computing devices processors.

55. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 54.

56. The method of claim 54, further comprising, at the stage of the multiple stage pipeline,

determining, based on the evaluating, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a subsequent stage of the pipeline.

57. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 56.

58. The method of claim 54, further comprising, at the stage of the multiple stage pipeline,

applying a first weight to the first results of the first set of one or more analysis operations, producing first weighted results;

applying a second weight to the second results of the second set of one or more analysis operations, producing second weighted results; and

evaluating a condition that is based on the first weighted results and the second weighted results.

59. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 58.

60. The method of claim 58, further comprising, at the stage of the multiple stage pipeline,

determining, based on the evaluating that is based on the first weighted results and the second weighted results, whether or not to generate information about the content of the first or second video data on which the first or second set of one or more analysis operations were performed, for sending to a subsequent stage of the multiple stage pipeline.

61. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 60.