Event generation and camera cluster analysis of multiple video streams in a pipeline architecture

• US 7,667,732 B1
• Filed: 10/13/2004
• Issued: 02/23/2010
• Est. Priority Date: 03/16/2004
• Status: Expired due to Fees

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Pappas, Christian, Haupt, Gordon T., Russell, Stephen G., Vallone, Robert P., Fleischer, Stephen D., Freeman, J. Andrew
• Primary Examiner(s): AN, SHAWN S
• Application Number:
  US10/965,676
• Time in Patent Office: 1,959 Days
• Field of Search: 348/143, 348/153, 348/159, 348/161, 348/154, 348/155, 348/699, 382/103, 382/165, 382/203
• US Class Current: 348/143
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream

Abstract

A pipeline architecture for analyzing multiple streams of video is embodied, in part, in a layer of application program interfaces (APIs) to each stage of processing. Buffer queuing is used between some stages, which helps moderate the load on the CPU(s). Through the layer of APIs, innumerable video analysis applications can access and analyze video data flowing through the pipeline, and can annotate portions of the video data (e.g., frames and groups of frames), based on the analyses performed, with information that describes the frame or group. These annotated frames and groups flow through the pipeline to subsequent stages of processing, at which increasingly complex analyses can be performed. At each stage, portions of the video data that are of little or no interest are removed from the video data. Ultimately, “events” are constructed and stored in a database, from which cross-event and historical analyses may be performed and associations with, and among, events may be made.

Claims

1. A volatile or non-volatile storage device storing instructions which, when executed by one or more processors, cause:

receiving, from a first video analyzer application, a first call to read, from a memory, first analytic information that is associated with first video data;

wherein the first video data includes (a) first frames that correspond to a first video stream and (b) the first analytic information;

wherein the first analytic information is (i) about the content of the first frames and describes the content of the first frames (ii) based on prior analysis of the first frames, and (iii) different than the first frames;

in response to the first call, providing the first analytic information to the first video analyzer application to allow the first video analyzer application to perform a first set of one or more analysis operations on the first analytic information;

receiving, from the first video analyzer application, a second call to read, from a second memory, second analytic information that is associated with second video data;

wherein the second video data includes (a) one or more second frames that correspond to a second video stream and (b) the second analytic information;

wherein the second analytic information is (i) about the content of the second frames and describes the content of the second frames, (ii) based on prior analysis of the second frames, and (iii) different than the second frames;

wherein the second frames are different than the first frames;

in response to the second call, providing the second analytic information to the first video analyzer application to allow;

the first video analyzer application to perform a second set of one or more analysis operations on the second analytic information, and

the first video analyzer application to construct event information about the content of the first frames and the second frames based on both the first analytic information and the second analytic information;

wherein the second analytic information differs from the first analytic information.

2. The volatile or non-volatile storage device of claim 1, wherein:

the one or more first frames originated from a first camera at a first time, and

the one or more second frames originated from the first camera at a second time that is different than the first time.

3. The volatile or non-volatile storage device of claim 1, wherein:

the one or more first frames originated from a first camera, and

the one or more second frames originated from a second camera that is different than the first camera.

4. The volatile or non-volatile storage device of claim 1, wherein the instructions include additional instructions which, when executed by the one or more processors, further cause storing the event information in a data repository.

5. The volatile or non-volatile storage device of claim 4, wherein the step of providing the second analytic information includes providing the first analytic information to the first video analyzer application to allow the third video analyzer application to store the event information in one or more database tables.

6. The volatile or non-volatile storage device of claim 4, wherein the event information stored in the data repository is first event information, and the instructions include additional instructions which, when executed by the one or more processors, further cause providing the first event information, from the data repository, to the first video analyzer application to allow the first video analyzer application to:

add second event information to the first event information, wherein the second event information further describes the content of the one or more first and second frames, and

store the first and second event information together as an entry in the data repository.

7. The volatile or non-volatile storage device of claim 6, wherein the step of providing the first event information includes providing the first event information to the first video analyzer application to allow the first video analyzer application to construct the second event information in response to occurrence of an event that is external to the first and second video data.

8. The volatile or non-volatile storage device of claim 1, wherein the instructions include additional instructions which, when executed by the one or more processors, further cause:

providing the one or more first frames to the first video analyzer application to allow the first video analyzer application to perform a third set of one or more analysis operations on the one or more first frames; and

providing the one or more second frames to the first video analyzer application to allow;

the first video analyzer application to perform a fourth set of one or more analysis operations on the one or more second frames; and

the first video analyzer application to construct the event information about the content of the one or more first and second frames based on the fourth and third sets of one or more analysis operations.

9. The volatile or non-volatile storage device of claim 8, wherein:

the one or more first frames originated from a first camera at a first time, and

the one or more second frames originated from the first camera at a second time that is different than the first time.

10. The volatile or non-volatile storage device of claim 8, wherein:

the one or more first frames originated from a first camera, and

the one or more second frames originated from a second camera that is different than the first camera.

11. The volatile or non-volatile storage device of claim 8, wherein:

the instructions include additional instructions which, when executed by the one or more processors, further cause storing the event information in a data repository; and

providing the one or more second frames includes providing the one or more second frames to the first video analyzer application to allow the first video analyzer application to construct event information that includes references to locations in persistent memory at which the one or more first frames and the one or more second frames are stored.

12. The volatile or non-volatile storage device of claim 8, wherein:

the instructions include additional instructions which, when executed by one or more processors, further cause storing the event information in a data repository; and

wherein storing in a data repository includes storing the event information in one or more database tables.

13. The volatile or non-volatile storage device of claim 8, wherein the third set of one or more analysis operations is a first set of one or more on-the-fly analysis operations and the fourth set of one or more analysis operations are a second set of one or more on-the-fly analysis operations.

14. The volatile or non-volatile storage device of claim 8, wherein the one or more first and second frames are queued in a buffer.

15. The volatile or non-volatile storage device of claim 8, wherein the one or more first frames are from a first group of frames and the one or more second frames are from a second group of frames that is different than the first group of frames.

16. The volatile or non-volatile storage device of claim 1, wherein the event information includes pointers to locations in persistent memory at which the one or more first frames and the one or more second frames are stored.

17. The volatile or non-volatile storage device of claim 1, wherein the event information includes a start time and an end time of each of the one or more first frames and the one or more second frames.

18. The volatile or non-volatile storage device of claim 1, wherein the event information that includes references to locations in persistent memory at which a first group of frames that contains the one or more first frames and a second group of frames that contains the one or more second frames are stored.

19. The volatile or non-volatile storage device of claim 18, wherein the first group of frames is a different group than the second group of frames.

20. The volatile or non-volatile storage device of claim 1, wherein the step of providing the second analytic information includes providing the second analytic information to the first video analyzer application to allow the first video analyzer application to construct the event information in response to occurrence of an event that is external to the first and second video data.

21. The volatile or non-volatile storage device of claim 1, wherein the first set of one or more analysis operations is a first set of one or more on-the-fly analysis operations and the second set of one or more analysis operations is a second set of one or more on-the-fly analysis operations.

22. The volatile or non-volatile storage device of claim 1, wherein the first and second analytic information are queued in a buffer.

23. The volatile or non-volatile storage device of claim 1, wherein the one or more first and second frames correspond to surveillance video streams.

24. The volatile or non-volatile storage device of claim 1, wherein the one or more first frames are from a first group of frames and the one or more second frames are from a second group of frames that is different than the first group of frames.

25. A system for analyzing video data that represents one or more video streams, the system comprising:

means for reading, from a memory, first analytic information that is associated with first video data and second analytic information that is associated with second video data;

wherein the first video data includes (a) one or more first frames that correspond to a first video stream and (b) the first analytic information;

wherein the second video data includes (a) one or more second frames that correspond to a second video stream and (b) the second analytic information;

wherein the first analytic information is (i) about the content of the one or more first frames and describes the content of the one or more first frames, (ii) based on prior analysis of the one or more first frames, and (iii) different than the one or more first frames;

wherein the second analytic information is (i) about the content of the one or more second frames and describes the content of the one or more second frames, (ii) based on prior analysis of the one or more second frames, and (iii) different than the one or more second frames;

wherein the one or more second frames are different than the one or more first frames;

means for performing a first set of one or more analysis operations on the first analytic information and performing a second set of one or more analysis operations on the second analytic information;

means for constructing event information about the content of the one or more first and second frames based on both the first analysis information and the second analytic information;

wherein the second analytic information differs from the first analytic information;

means for storing the event information in a data repository.

26. The system of claim 25, further comprising:

means for performing a third set of one or more analysis operations on the one or more first frames and performing a fourth set of one or more analysis operations on the one or more second frames; and

means for constructing the event information about the content of the one or more first frames and the one or more second frames based on the third set of one or more analysis operations and the fourth set of one or more analysis operations.

27. A method performed by one or more processors for analyzing video data that represents one or more video streams, the method comprising the steps of:

reading, from a memory, video data that includes first video data and second video data;

wherein the first video data includes first analytic information about one or more first frames that correspond to a first video stream;

wherein the first analytic information is (i) about the content of the one or more first frames and describes the content of the one or more first frames, (ii) based on prior analysis of the one or more first frames, and (iii) different than the one or more first frames;

wherein the second video data includes second analytic information about one or more second frames that correspond to a second video stream;

wherein the second analytic information is (i) about the content of the one or more second frames and describes the content of the one or more second frames, (ii) based on prior analysis of the one or more second frames, and (iii) different than the one or more second frames;

performing a first set of one or more analysis operations on the first analytic information;

performing a second set of one or more analysis operations on the second analytic information;

constructing event information about the content of the one or more first and second frames based on the first set of one or more analysis operations on the first analytic information and the second set of one or more analysis operations on the second analytic information;

wherein the second analytic information differs from the first analytic information;

wherein the steps are performed by the one or more computing devices processors.

28. The method of claim 27, wherein:

the one or more first frames originated from a first camera at a first time; and

the one or more second frames originated from the first camera at a second time that is different than the first time.

29. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 28.

30. The method of claim 27, wherein:

the one or more first frames originated from a first camera; and

the one or more second frames originated from a second camera that is different than the first camera.

31. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 30.

32. The method of claim 27, wherein the step of constructing includes constructing event information that includes pointers to locations in persistent memory at which the one or more first and second frames are stored.

33. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 32.

34. The method of claim 27, wherein the step of constructing includes constructing event information that includes references to locations in persistent memory at which a first group of frames that contains the one or more first frames and a second group of frames that contains the one or more second frames are stored.

35. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 34.

36. The method of claim 27, wherein the step of constructing includes construction first event information for storage in a data repository, the method further comprising:

retrieving the first event information from the data repository;

adding second event information to the first event information, wherein the second event information further describes the content of the one or more first and second frames; and

outputting the first and second event information for storage as an entry in the data repository.

37. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 36.

38. The method of claim 27, wherein the step of constructing includes constructing event information about the content of the one or more first frames and the one or more second frames in response to occurrence of an event that is external to the video data.

39. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 38.

40. The method of claim 27, wherein the steps of performing include:

performing a first set of one or more on-the-fly analysis operations on the first video data; and

performing a second set of on-the-fly one or more analysis operations on the second video data.

41. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 40.

42. The method of claim 27, wherein the step of reading includes reading from a buffer.

43. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 42.

44. The method of claim 27,

wherein the steps of reading and performing include reading and performing the first and second set of one or more analysis operations on the first and second analytic information that correspond to surveillance video streams;

and

wherein the step of constructing includes constructing event information about the content of the frames that correspond to the surveillance video streams.

45. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 44.

46. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 27.