AUTOMATIC EXTRACTION OF SECONDARY VIDEO STREAMS

• US 20100165112A1
• Filed: 03/27/2007
• Published: 07/01/2010
• Est. Priority Date: 03/28/2006
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: ObjectVideo Incorporated (Alarm.com Holdings, Inc.)
• Inventors: Myers, Gary W., Cutting, Robert A., Clark, John I.W., Chosak, Andrew J., Lipton, Alan J., Scanlon, W. Andrew
• Granted Patent: US 8,848,053 B2
• Time in Patent Office: Days
• US Class Current: 348/169
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/20 using video object coding

Abstract

The automatic generation (702) of one or more secondary video streams based on an input primary video stream (701) may use analytics performed on the video to provide information on targets, events, and/or areas of interest to permit the one or more secondary video streams (703) to concentrate on one or more targets, events, and/or areas of interest.

Claims

1. A method of automatically generating at least one secondary video stream based on an incoming primary video stream, comprising:

performing video analytics on the primary video stream to generate one or more analysis results, the one or more analysis results including information relating to at least one element selected from the group consisting of a target, an event, and an area of interest; and

automatically extracting at least one secondary video stream from the primary video stream based on the one or more analysis results, wherein a field of view of the secondary video stream is smaller or equal to than a field of view of the primary video stream.

2. The method according to claim 1, wherein said performing video analytics comprises:

applying the primary video stream to a vision module; and

applying results obtained from said vision module to an inference module.

3. The method according to claim 2, wherein said applying the primary video stream to a vision module comprises:

performing target detection.

4. The method according to claim 3, wherein said applying the primary video stream to a vision module further comprises:

performing at least one operation selected from the group consisting of classifying at least one target and tracking at least one target.

5. The method according to claim 2, wherein said applying the primary video stream to a vision module comprises:

performing camera motion compensation.

6. The method according to claim 2, wherein said applying the primary video stream to a vision module comprises:

identifying one or more foreground portions in the primary video stream;

grouping the one or more foreground portions into one or more targets; and

generating at least one video primitive describing at least one characteristic of at least one of said one or more targets.

7. The method according to claim 6, wherein said applying the primary video stream to a vision module further comprises:

tracking said one or more targets; and

classifying said one or more targets.

8. The method according to claim 2, wherein said applying results obtained from said vision module to an inference module comprises:

applying at least one rule to said results obtained from said vision module, the at least one rule selected from the group consisting of;

a target criterion, an event criterion, and an area of interest criterion.

9. The method according to claim 8, wherein said applying results obtained from said vision module to an inference module further comprises:

maintaining a list of one or more currently active targets of interest.

10. The method according to claim 1, wherein said automatically extracting comprises:

extracting a best-shot video stream including one or more targets of interest.

11. The method according to claim 1, wherein said automatically extracting comprises:

receiving configuration information to be used in determining how to extract at least one secondary video stream.

12. The method according to claim 1, wherein said automatically extracting comprises:

operating on a down-sampled version of the primary video stream to extract a secondary video stream selected from the group consisting of;

a secondary video stream extracted from the primary video stream; and

a secondary video stream extracted from the down-sampled version of the primary video stream.

13. A computer-readable medium containing software code that, when executed by a processor, performs the method according to claim 1.

14. An apparatus to automatically generate at least one secondary video stream based on an incoming primary video stream, comprising:

a video analytics module to operate on the primary video stream to generate one or more analysis results, the one or more analysis results including information relating to at least one element selected from the group consisting of a target, an event, and an area of interest; and

a video extraction module to automatically extracting at least one secondary video stream from the primary video stream based on the one or more analysis results, wherein a field of view of the secondary video stream is smaller than or equal to a field of view of the primary video stream.

15. The apparatus according to claim 14, wherein said video analytics module comprises:

a vision module; and

an inference module to receive results from said vision module.

16. The apparatus according to claim 15, wherein said vision module comprises:

a segmentation module;

a blobizer; and

a primitive generation module.

17. The apparatus according to claim 16, wherein said vision module further comprises:

a target'"'"' tracking module; and

a target classification module.

18. The apparatus according to claim 14, wherein at least one of said modules comprises at least one device to implement at least a portion of the functionality of said at least one of said modules, said device selected from the group consisting of a special-purpose hardware device and a programmable hardware device.

19. A method of automatically generating at least one secondary video stream based on an incoming primary video stream, comprising:

performing first video analytics on the primary video stream to generate one or more first analysis results, the one or more first analysis results including information relating to at least one element selected from the group consisting of a target, an event, and an area of interest;

automatically extracting at least one secondary video stream from the primary video stream based on the one or more first analysis results, wherein a field of view of the secondary video stream is smaller than or equal to a field of view of the primary video stream;

performing second video analytics on said at least one secondary video stream to provide one or more second analysis results; and

feeding back said one or more second analysis results such that said automatically extracting at least one secondary video stream is further based on said one or more second analysis results.

20. The method according to claim 19, wherein said performing first video analytics comprises:

applying the primary video stream to a first vision module; and

applying results obtained from said first vision module to a first inference module.

21. The method according to claim 20, wherein said applying the primary video stream at a first vision module comprises:

performing target detection.

22. The method according to claim 21, wherein said applying the primary video stream at a first vision module comprises at least one operation selected from the group consisting of performing target classification and performing target tracking.

23. The method according to claim 20, wherein said applying the primary video stream at a first vision module comprises:

performing camera motion compensation.

24. The method according to claim 20, wherein said applying the primary video stream at a first vision module comprises:

identifying one or more foreground portions in the primary video stream;

grouping the one or more foreground portions into one or more targets; and

generating at least one video primitive describing at least one characteristic of at least one of said one or more targets.

25. The method according to claim 24, wherein said applying the primary video stream at a first vision module further comprises:

tracking said one or more targets; and

classifying said one or more targets.

26. The method according to claim 20, wherein said applying results of said first vision module to said first inference module comprises:

apply at least one rule to said results obtained from said first vision module, the at least one rule selected from the group consisting of;

a target criterion, an event criterion, and an area of interest criterion.

27. The method according to claim 26, wherein said applying results of said first vision module to said first inference module further comprises:

maintaining a list of one or more currently active targets of interest.

28. The method according to claim 19, wherein said performing second video analytics on said at least one secondary video stream comprises:

applying the at least one secondary video stream to a second vision module; and

applying results obtained from said second vision module to a second inference module.

29. The method according to claim 28, wherein said applying the at least one secondary video stream to a second vision module comprises:

estimating scene motion in said at least one secondary video stream;

performing camera motion compensation on the at least one secondary video stream to provide at least one compensated secondary video stream;

identifying one or more foreground portions of said at least one compensated secondary video stream;

identifying at least one location of at least one target in said at least one compensated secondary video stream; and

generating at least one video primitive relating to said at least one target.

30. The method according to claim 29, wherein said identifying at least one location comprises:

performing a template matching operation using information about at least one target.

31. The method according to claim 30, wherein said applying the at least one secondary video stream to a second vision module further comprises detecting at least one image feature in said at least one secondary video stream;

and

wherein said performing a template matching operation uses the at least one image feature.

32. The method according to claim 30, wherein said performing a template matching operation uses at least one stored target model.

33. The method according to claim 32, wherein said performing a template matching operation further comprises providing information to update said at least one stored target model.

34. The method according to claim 19, wherein said automatically extracting comprises:

extracting a best-shot video stream including one or more targets of interest.

35. The method according to claim 19, wherein said automatically extracting comprises:

receiving configuration information to be used in determining how to extract at least one secondary video stream.

36. The method according to claim 19, wherein said automatically extracting comprises:

operating on a down-sampled version of the primary video stream to extract a secondary video stream selected from the group consisting of;

a secondary video stream extracted from the primary video stream; and

a secondary video stream extracted from the down-sampled version of the primary video stream.

37. A computer-readable medium containing software code that, when executed by a processor, causes the processor to execute the method according to claim 19.

38. An apparatus to automatically generate at least one secondary video stream based on an incoming primary video stream, comprising:

a first video analytics module to operate on the primary video stream to generate one or more first analysis results, the one or more first analysis results including information relating to at least one element selected from the group consisting of a target, an event, and an area of interest;

a video extraction module to automatically generate at least one secondary video stream from the primary video stream based on the one or more first analysis results, wherein a field of view of the secondary video stream is smaller than or equal to a field of view of the primary video stream; and

a second video analytics module to operate on said at least one secondary video stream to provide one or more second analysis results and to feed back said one or more second analysis results to the video extraction module;

wherein extraction of the at least one secondary video stream by the video extraction module is further based on said one or more second analysis results.

39. The apparatus according to claim 38, wherein said first video analytics module comprises:

a first vision module; and

a first inference module to receive results from said first vision module.

40. The apparatus according to claim 39, wherein said first vision module comprises:

a first segmentation module;

a blobizer; and

a first primitive generation module.

41. The apparatus according to claim 40, wherein said first vision module further comprises:

a target tracking module; and

a target classification module.

42. The apparatus according to claim 38, wherein said second video analytics module comprises:

a second vision module; and

a second inference module to receive results from said second vision module.

43. The apparatus according to claim 42, wherein said second vision module comprises:

a scene motion estimation module to estimate scene motion in said at least one secondary video stream;

a camera motion compensation module to operate on said at least one secondary video stream to produce at least one compensated secondary video stream;

a second segmentation module to identify at least one foreground portion in at least one compensated secondary video stream;

a target identification module to determine a location of at least one target in at least one compensated secondary video stream based on said at least one foreground portion; and

a second primitive generation module to generate one or more video primitives.

44. The apparatus according to claim 43, wherein said target identification module comprises:

a template matching module.

45. The apparatus according to claim 44, wherein said second vision module further comprises:

an image feature detection module to detect at least one image feature from said at least one secondary video stream and to provide said at least one image feature to said template matching module.

46. The apparatus according to claim 44, wherein said second vision module further comprises:

a storage device to store at least one target model, wherein said storage device is coupled to said template matching module to communicate therewith regarding said at least one target model.

47. The apparatus according to claim 38, wherein at least one of said modules comprises at least one device to implement at least a portion of the functionality of said at least one of said modules, said device selected from the group consisting of:

a special-purpose hardware device and a programmable hardware device.

48. A method of automatically generating at least one secondary video stream based on an incoming primary video stream, comprising:

automatically extracting at least one secondary video stream from the primary video stream, wherein a field of view of the secondary video stream is smaller than or equal to a field of view of the primary video stream;

performing video analytics on said at least one secondary video stream to provide one or more analysis results; and

feeding back said one or more analysis results such that said automatically extracting at least one secondary video stream is based at least in part on said one or more second analysis results.

49. The method according to claim 48, wherein said automatically extracting comprises:

initially automatically extracting one or more secondary video streams from the primary video stream based on a predefined scheme for generating one or more secondary video streams.

50. The method according to claim 49, wherein said initially automatically extracting comprises a technique selected from the group consisting of:

generating said one or more secondary video streams corresponding to predefined areas within the primary video stream; and

generating at least one secondary video stream that is scanned across at least part of the area within the primary video stream.

51. The method according to claim 48, wherein said performing video analytics on said at least one secondary video stream comprises:

applying the at least one secondary video stream to a vision module; and

applying results obtained from said vision module to an inference module.

52. The method according to claim 51, wherein said applying the at least one secondary video stream to a vision module comprises:

estimating scene motion in said at least one secondary video stream;

performing camera motion compensation on the at least one secondary video stream to provide at least one compensated secondary video stream;

identifying one or more foreground portions of said at least one compensated secondary video stream;

identifying at least one location of at least one target in said at least one compensated secondary video stream; and

generating at least one video primitive relating to said at least one target.

53. The method according to claim 52, wherein said identifying at least one location comprises:

performing a template matching operation using information about at least one target.

54. The method according to claim 53, wherein said applying the at least one secondary video stream to a vision module further comprises detecting at least one image feature in said at least one secondary video stream;

and

wherein said performing a template matching operation uses the at least one image feature.

55. The method according to claim 53, wherein said performing a template matching operation uses at least one stored target model.

56. The method according to claim 55, wherein said performing a template matching operation further comprises providing information to update said at least one stored target model.

57. The method according to claim 48, wherein said automatically extracting comprises:

extracting a best-shot video stream including one or more targets of interest.

58. The method according to claim 48, wherein said automatically extracting comprises:

receiving configuration information to be used in determining how to extract at least one secondary video stream.

59. The method according to claim 48, wherein said automatically extracting comprises:

operating on a down-sampled version of the primary video stream to extract a secondary video stream selected from the group consisting of;

a secondary video stream extracted from the primary video stream; and

a secondary video stream extracted from the down-sampled version of the primary video stream.

60. A computer-readable medium containing software code that, when executed by a processor, causes the processor to execute the method according to claim 48.

61. An apparatus to automatically generate at least one secondary video stream based on an incoming primary video stream, comprising:

a video extraction module to automatically generate at least one secondary video stream from the primary video stream, wherein a field of view of the secondary video stream is smaller than or equal to a field of view of the primary video stream; and

a video analytics module to operate on said at least one secondary video stream to provide one or more analysis results and to feed back said one or more second analysis results to the video extraction module;

wherein extraction of the at least one secondary video stream by the video extraction module is further based on said one or more analysis results.

62. The method according to claim 61, wherein said video analytics module comprises:

a vision module; and

an inference module to receive results from said vision module.

63. The apparatus according to claim 62, wherein said vision module comprises:

a scene motion estimation module to estimate scene motion in said at least one secondary video stream;

a camera motion compensation module to operate on said at least one secondary video stream to produce at least one compensated secondary video stream;

a segmentation module to identify at least one foreground portion in at least one compensated secondary video stream;

a target identification module to determine a location of at least one target in at least one compensated secondary video stream based on said at least one foreground portion; and

a primitive generation module to generate one or more video primitives.

64. The apparatus according to claim 63, wherein said target identification module comprises:

a template matching module.

65. The apparatus according to claim 64, wherein said vision module further comprises:

an image feature detection module to detect at least one image feature from said at least one secondary video stream and to provide said at least one image feature to said template matching module.

66. The apparatus according to claim 64, wherein said vision module further comprises:

a storage device to store at least one target model, wherein said storage device is coupled to said template matching module to communicate therewith regarding said at least one target model.

67. The apparatus according to claim 61, wherein at least one of said modules comprises at least one device to implement at least a portion of the functionality of said at least one of said modules, said device selected from the group consisting of:

a special-purpose hardware device and a programmable hardware device.