Deep frame analysis of multiple video streams in a pipeline architecture

• US 7,672,370 B1
• Filed: 10/13/2004
• Issued: 03/02/2010
• Est. Priority Date: 03/16/2004
• Status: Expired due to Fees

• Current Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Original Assignee: 3VR Security Incorporated (Identiv, Inc.)
• Inventors: Pappas, Christian, Haupt, Gordon T., Russell, Stephen G., Vallone, Robert P., Fleischer, Stephen D., Freeman, J. Andrew
• Primary Examiner(s): AN, SHAWN S
• Application Number:
  US10/965,682
• Time in Patent Office: 1,966 Days
• Field of Search: 375/240.01, 348/153, 348/159, 348/161, 348/154, 348/155, 348/699, 382/103, 382/165, 382/203
• US Class Current: 375/240.01
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream

Abstract

A pipeline architecture for analyzing multiple streams of video is embodied, in part, in a layer of application program interfaces (APIs) to each stage of processing. Buffer queuing is used between some stages, which helps moderate the load on the CPU(s). Through the layer of APIs, innumerable video analysis applications can access and analyze video data flowing through the pipeline, and can annotate portions of the video data (e.g., frames and groups of frames), based on the analyses performed, with information that describes the frame or group. These annotated frames and groups flow through the pipeline to subsequent stages of processing, at which increasingly complex analyses can be performed. At each stage, portions of the video data that are of little or no interest are removed from the video data. Ultimately, “events” are constructed and stored in a database, from which cross-event and historical analyses may be performed and associations with, and among, events may be made.

Claims

1. A volatile or non-volatile storage device storing instructions which, when executed by one or more processors, cause:

receiving, from a second video analysis application, a call to read, from a memory, first analytic information that is associated with video data;

wherein the video data includes (a) a frame that corresponds to a video stream and (b) the first analytic information;

wherein the first analytic information is (i) generated by a first video analysis application that is different than the second video analysis application about the content of the frame (ii) based on a prior analysis of the frame by the first video analysis application, and (iii) different than the frame; and

providing the first analytic information to the second video analysis application to allow the second video analysis application to examine the first analytic information to determine whether or not the second video analysis application is to perform a particular set of one or more on-the-fly analysis operations on the frame.

2. The volatile or non-volatile storage device of claim 1, wherein:

providing the first analytic information includes providing the first analytic information to the second video analysis application to allow the second video analysis application to;

annotate the frame with second analytic information about the content of the frame based on the particular set of one or more on-the-fly analysis operations; and

wherein the instructions include additional instructions which, when executed by the one or more processors, further cause sending, to another stage of analytical processing, at least the second analytic information about the content of the frame.

3. The volatile or non-volatile storage device of claim 2, wherein

providing the first analytic information includes providing the first analytic information to the second video analysis application to allow the second video analysis application to annotate the frame with second analytic information that describes the content of the frame further than the first analytic information.

4. The volatile or non-volatile storage device of claim 2, wherein:

providing the first analytic information includes providing the first analytic information to the second video analysis application to allow the second video analysis application to annotate the frame with second analytic information that describes the content of the frame with a greater degree of confidence than the first analytic information.

5. The volatile or non-volatile storage device of claim 2, wherein sending includes sending the frame.

6. The volatile or non-volatile storage device of claim 2, wherein sending includes sending a pointer to a location in persistent memory at which the frame is stored.

7. The volatile or non-volatile storage device of claim 2, wherein:

receiving a call includes receiving, from each of a plurality of video analysis applications a call to read, from the memory, the first analytic information about the content of the frame that corresponds to the video stream;

providing the first analytic information includes providing the first analytic information to each of the plurality of video analysis applications to allow each of the plurality of video analysis applications to;

examine the first analytic information to determine whether or not to perform a set of one or more on-the-fly analysis operations on the frame,

perform the analysis operations on the frame, and

annotate the frame with second analytic information based on the set of one or more on-the-fly analysis operations.

8. The volatile or non-volatile storage device of claim 7, wherein providing the first analytic information includes providing the first analytic information to each of the plurality of video analysis applications to allow each of the plurality of video analysis applications to annotate the frame by generating second analytic information that is separate from second analytic information generated by any other of the plurality of applications.

9. The volatile or non-volatile storage device of claim 2, wherein providing the first analytic information includes providing the first analytic information to the second video analysis application to allow the second video analysis application to perform the particular set of one or more on-the-fly analysis operations on the content of the frame in relation to the content of a different frame that corresponds to the same video stream.

10. The volatile or non-volatile storage device of claim 2, wherein providing the first analytic information includes providing the first analytic information to the second video analysis application to allow the second video analysis application to annotate the frame with second analytic information that includes information about audio content of the frame.

11. The volatile or non-volatile storage device of claim 1, wherein providing the first analytic information includes providing the first analytic information to the second video analysis application to allow the second video analysis application to examine information that is external to the video data to determine whether or not to perform the particular set of one or more on-the-fly analysis operations on the frame.

12. The volatile or non-volatile storage device of claim 1, wherein providing the first analytic information includes providing the first analytic information to the second video analysis application to allow the second video analysis application to determine not to perform the particular set of one or more on-the-fly analysis operations on the frame and to remove video data that corresponds to the frame from the video data that represents the video stream without sending the removed video data to another stage of analytical processing.

13. The volatile or non-volatile storage device of claim 1, wherein the instructions include additional instructions which, when executed by the one or more processors, further cause:

receiving, from a third video analysis application, a second call to read, from the memory, second video data that represents a group of frames that corresponds to a second video stream and corresponding first group annotation information that identifies what frames are contained in the group, and

in response to receiving the second call, providing the first group annotation information to the third video analysis application to allow the third video analysis application to examine the corresponding first group annotation information to determine whether or not to perform a second particular set of one or more on-the-fly analysis operations on the group of frames.

14. The volatile or non-volatile storage device of claim 13, wherein:

providing the first group annotation information includes providing the first analytic information to the third video analysis application to allow the third video analysis application to;

determine to perform the second particular set of one or more on-the-fly analysis operations on the group of frames,

perform the analysis operations on the group of frames;

annotate the group of frames with second group annotation information about the content of the group of frames based on the second particular set of one or more on-the-fly analysis operations; and

wherein the instructions include additional instructions which, when executed by the one or more processors, further cause sending, to another stage of analytical processing, the second group annotation information about the content of the group of frames.

15. The volatile or non-volatile storage device of claim 14, wherein providing the first group annotation information includes providing the first group annotation information to the third video analysis application to allow the third video analysis application to annotate the group of frames with second group annotation information that describes the content of the group of frames further than the first group annotation information.

16. The volatile or non-volatile storage device of claim 14, wherein providing the first group annotation information includes providing the first group annotation information to the third video analysis application to allow the third video analysis application to annotate the group of frames with second group annotation information that describes the content of the group of frames with a greater degree of confidence than the first group annotation information.

17. The volatile or non-volatile storage device of claim 13, wherein-providing the first group annotation information includes providing the first analytic information to the third video analysis application to allow the third video analysis application to determine not to perform the second particular set of one or more on-the-fly analysis operations on the group of frames and to remove video data that represents the group of frames from the video data that represents the video stream without sending the removed video data to another stage of analytical processing.

18. The volatile or non-volatile storage device of claim 1, wherein the first analytic information is queued in a buffer when the call is received.

19. The volatile or non-volatile storage device of claim 1, wherein the instructions include additional instructions which, when executed by the one or more processors, further cause, for each of a plurality of video streams,

receiving, from a third video analysis application, a second call to read, from the memory, second video data that represents frames that correspond to said each video stream and that are queued in a buffer that corresponds to said each video stream,

in response to receiving the second call, providing the second video data to the third video analysis application to allow the second video analysis application to:

examine the second video data to determine whether or not to perform a set of one or more on-the-fly analysis operations on a particular frame of the frames,

perform the set of one or more on-the-fly analysis operations on the particular frame, and

annotate the frame with second analytic information based on the set of one or more on-the-fly analysis operations.

20. The volatile or non-volatile storage device of claim 19, wherein each thread of a plurality of processing threads is configured to read information stored in any one of the buffers.

21. The volatile or non-volatile storage device of claim 19, wherein the instructions include additional instructions which, when executed by the one or more processors, further cause:

for each buffer, providing the second video data to each of one or more video analysis applications to allow said each video analysis application to analyze each video data that represents a frame before allowing any of the one or more video analysis applications to analyze video data that represents a subsequent frame in the corresponding video stream.

22. The volatile or non-volatile storage device of claim 1, wherein the video data corresponds to streams of surveillance video.

23. A system for analyzing video data that represents one or more video streams, the system comprising:

means for reading, from a memory, first analytic information that is associated with the video data,

wherein the video data includes (a) a frame that corresponds to a stream and (b) the first analytic information;

wherein the first analytic information is (i) generated by a first video analysis application about the content of the frame (ii) based on a prior analysis of the frame by the first video analysis application, and (iii) different than the frame; and

means for examining, by a second video analysis application that is different than the first video analysis application, the first analytic information to determine whether or not the second video analysis application is to perform a particular set of one or more on-the-fly analysis operations on the frame.

24. The system of claim 23, further comprising:

means for performing the one or more on-the-fly analysis operations on the frame, and

means for annotating the frame with second analytic information about the content of the frame based on the one or more on-the-fly analysis operations; and

means for sending, to another stage of analytical processing, at least the second analytic information about the content of the frame.

25. A method performed by one or more processors for analyzing video data that represents one or more video streams, the method comprising the steps of:

reading, from a memory, first analytic information that is associated with the video data;

wherein the video data includes (a) a frame that corresponds to a stream and (b) first analytic information;

wherein the first analytic information is (i) generated by a first video analysis application about the content of the frame, (ii) based on a prior analysis of the frame by the first video analysis application, and (iii) different than the frame; and

examining, by a second video analysis application that is different than the first video analysis application, the first analytic information to determine whether or not the second video analysis application is to perform a particular set of one or more on-the-fly analysis operations on the frame;

wherein the steps of reading and examining are performed by the one or more processors.

26. The method of claim 25, further comprising the steps of:

if it is determined to perform the particular set of one or more on-the-fly analysis operations on the frame, then

performing the particular set of one or more on-the-fly analysis operations on the frame, and

annotating the frame with second analytic information about the content of the frame based on the particular set of one or more on-the-fly analysis operations; and

outputting the second analytic information for providing to another stage of processing for further analysis.

27. The method of claim 26, wherein the step of annotating includes annotating the frame with second analytic information that describes the content of the frame further than the first analytic information.

28. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 27.

29. The method of claim 26, wherein the step of annotating includes annotating the frame with second analytic information that describes the content of the frame with a greater degree of confidence than the first analytic information.

30. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 29.

31. The method of claim 26, wherein the step of outputting includes outputting a pointer to a location in persistent memory at which the frame is stored.

32. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 31.

33. The method of claim 26, wherein the step of performing includes performing the particular set of one or more on-the-fly analysis operations on the content of the frame in relation to the content of a different frame that corresponds to the same stream.

34. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 33.

35. The method of claim 26, wherein the step of annotating includes annotating the frame with third analytic information that includes information about audio content of the frame.

36. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 35.

37. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 26.

38. The method of claim 25, wherein the particular set of one or more on-the-fly analysis operations are more computationally complex than the prior analysis.

39. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 38.

40. The method of claim 25, wherein the step of examining includes examining information that is external to the video data to determine whether or not to perform the particular set of one or more on-the-fly analysis operations on the frame.

41. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 40.

42. The method of claim 25,

wherein the step of reading includes reading, from the memory, video data that represents a group of frames that corresponds to a stream and corresponding first group annotation information that identifies what frames are contained in the group, and

wherein the step of examining includes examining the group of frames and the corresponding first group annotation information to determine whether or not to perform a second particular set of on-the-fly analysis operations on the group of frames.

43. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 42.

44. The method of claim 42, further comprising the steps of:

if it is determined to perform the second particular set of one or more on-the-fly analysis operations on the group of frames, then

performing the second particular set of one or more on-the-fly analysis operations on the group of frames;

annotating the group of frames with second group annotation information about the content of the group of frames based on the second particular set of one or more on-the-fly analysis operations; and

outputting the second group annotation information about the content of the group of frames for providing to another stage of processing for further analysis of the group of frames.

45. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 44.

46. The method of claim 44, wherein the step of annotating includes annotating the group of frames with second group annotation information that describes the content of the one or more frames contained in the group of frames further than the first group annotation information.

47. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 46.

48. The method of claim 44, wherein the step of annotating includes annotating the group of frames with second group annotation information that describes the content of the one or more frames contained in the group of frames with a greater degree of confidence than the first group annotation information.

49. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 48.

50. The method of claim 25, wherein the step of reading includes reading, from the memory, the video data, which includes the first analytic information, that is queued in a buffer.

51. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 50.

52. The method of claim 25, wherein the step of reading includes reading, from the memory, video data that includes frames that correspond to the video stream and that are queued in a buffer.

53. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 52.

54. The method of claim 25, wherein the step of reading includes reading, from the memory, the video data, which includes first analytic information that corresponds to frames from a plurality of video streams and that is queued in corresponding buffers.

55. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 54.

56. The method of claim 25, wherein the steps of reading and examining include reading and examining video data that includes first analytic information about frames that correspond to streams of surveillance video.

57. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 56.

58. A volatile or non-volatile storage device storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 25.