Apparatus having flexible capabilities for analysis of video information

• US 5,974,235 A
• Filed: 10/31/1996
• Issued: 10/26/1999
• Est. Priority Date: 10/31/1996
• Status: Expired due to Term

• Current Assignee: Sensormatic Electronics LLC (Johnson Controls International plc)
• Original Assignee: Sensormatic Electronics Corporation (Johnson Controls International plc)
• Inventors: Pedersen, Chris Harvey Jr., Nguyen, Lyn, Nunally, Patrick O., Lin-Liu, Sen, Winter, Gerhard Josef, MacCormack, David Ross, Wilson, Charles Park, Nguyen, William Thanh, Klein, Harry Eric, Auyeung, Alex Kamlun
• Primary Examiner(s): PEESO, THOMAS R
• Application Number:
  US08/742,017
• Time in Patent Office: 1,090 Days
• Field of Search: 395/200.61, 395/200.77, 395/200.32, 358/538, 358/539, 358/261.2, 348/394, 348/407, 348/415, 348/416, 382/236, 382/238, 382/246
• US Class Current: 709/202
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G08B 13/19604 involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/19634 Electrical details of the system, e.g. component blocks for carrying out specific functions
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: G08B 13/19667 Details realated to data compression, encryption or encoding, e.g. resolution modes for reducing data volume to lower transmission bandwidth or memory requirements
• CPC Class Codes: G08B 13/19673 Addition of time stamp, i.e. time metadata, to video stream
• CPC Class Codes: G08B 13/1968 Interfaces for setting up or customising the system
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: H04N 5/445 for displaying additional information H04N5/50 takes precedence
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 5/782 on tape
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 9/8042 involving data reduction
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal

Abstract

A flexible video information analysis apparatus stores a video information data base and a plurality of moving image content analysis algorithms for analyzing the video information in the data base. A user can manipulate a mouse to select one of the analysis algorithms. The selected algorithm is used to analyze video information in the data base.

Claims

1. A flexible video information analysis apparatus, comprising:

(a) first storage means for storing a video information data base;

(b) second storage means for storing a plurality of moving image content analysis algorithms for analyzing video information in the data base;

(c) user actuatable means for selecting one of said algorithms; and

(d) analysis means, responsive to said user actuatable means, for analyzing said video information in the data base in accordance with said one of said algorithms selected by said user actuatable means.

2. An apparatus according to claim 1, further comprising:

a plurality of video cameras for generating respective sequences of video signals; and

means for digitizing the respective sequences of video signals generated by said video cameras and for supplying the digitized sequences of video signals to said first storage means.

3. An apparatus according to claim 1, further comprising means for displaying a plurality of icons or alphanumeric labels each representing a respective one of said algorithms for analyzing video information.

4. An apparatus according to claim 3, wherein said user actuatable means comprises means for selecting one of the displayed icons or alphanumeric labels.

5. An apparatus according to claim 4, wherein said means for selecting one of the displayed icons comprises a mouse.

6. An apparatus according to claim 4, wherein said means for displaying and said means for selecting one of the displayed icons comprise a touch-sensitive screen.

7. An apparatus according to claim 1, wherein said user actuatable means includes means for setting a parameter for constraining execution of a selected one of said algorithms.

8. An apparatus according to claim 7, wherein said means for setting said parameter includes location means for selecting a portion of an image plane which corresponds to said video information, said analysis means responding to said location means by analyzing said video information with respect to the portion of the image plane selected by said location means.

9. An apparatus according to claim 1, wherein at least one of said stored algorithms is for detecting moving objects represented by data in said stored video information data base.

10. An apparatus according to claim 1, wherein said analysis means generates analysis data indicative of results of the analysis performed by said analysis means in accordance with said selected one of said algorithms and said first storage means includes means for storing said analysis data generated by said analysis means.

94. An apparatus according to claim 1, wherein said first storage means includes a hard drive.

95. An apparatus according to claim 94, wherein said second storage means includes a second hard drive.

96. An apparatus according to claim 95, wherein said first storage means includes a third hard drive.

97. An apparatus according to claim 1, further comprising a housing in which said first and second storage means and said analysis means are contained.

11. A method of storing and retrieving video data, comprising the steps of:

storing data representing plural images of a dynamic video image stream;

selecting a search algorithm to be used in searching for a desired image among the images represented by the stored data;

setting a parameter for constraining execution of the selected search algorithm; and

searching said stored data in accordance with the selected search algorithm and the set parameter.

12. A method according to claim 11, wherein the selected search algorithm is for detecting moving objects represented by said stored data.

13. A method according to claim 11, wherein the parameter for constraining execution of the selected search algorithm represents a location in an image plane at which the selected search algorithm is to be performed.

14. A method of carrying out an algorithm for detecting features represented by video data stored in a video information data base, comprising the steps of:

setting a parameter for constraining execution of said algorithm; and

executing said algorithm in accordance with the set parameter.

15. A method according to claim 14, wherein said algorithm is an algorithm for detecting at least one moving object represented by the stored video data, and said setting step includes selecting a portion of an image plane which corresponds to the stored video data, said portion being a portion at which said at least one moving object is to be detected by execution of said algorithm.

16. A method according to claim 15, wherein said portion of an image plane is selected by displaying an image which corresponds to an image represented by the stored video data and superimposing a line or polygon on said displayed image, said superimposed line or polygon representing said selected portion of said image plane.

17. A method according to claim 16, wherein said displaying of said image includes retrieving from the video information data base data representing said image to be displayed.

18. A method according to claim 16, wherein said displaying of said image is performed by:

generating with a video camera a video signal representing an image which corresponds to an image represented by said stored video data; and

displaying the video signal generated by said video camera.

19. A method according to claim 14, wherein said algorithm is an algorithm for detecting at least one moving object represented by the stored video data, and said setting step includes indicating at least one of a maximum size and a minimum size of the object to be detected.

20. A method according to claim 19, wherein said at least one of a maximum size and a minimum size is indicated by displaying an image corresponding to an image represented by the stored video data and superimposing on said displayed image an image element indicative of said at least one of a maximum size and a minimum size.

21. A method according to claim 20, wherein a first image element indicative of said maximum size of the object to be detected is displayed in a first color superimposed on said displayed image and a second image element indicative of said minimum size of the object to be detected is displayed in a second color, different from said first color, superimposed on said displayed image.

22. A method according to claim 21, wherein said first and second image elements are displayed simultaneously.

23. A method according to claim 14, wherein said parameter setting step includes displaying a graphical user interface element and adjusting the displayed graphical user interface element by means of a user input device.

24. A method according to claim 23, wherein the displayed graphical user interface element is a slide bar.

25. A method according to claim 23, wherein the displayed graphical user interface element is a dial.

26. A method according to claim 23, wherein the user input device is a mouse.

27. A flexible video information analysis apparatus, comprising:

(a) a video information source for providing video information which includes at least one dynamic sequence of video information frames;

(b) storage means for storing a plurality of algorithms for analyzing the video information provided by the video information source;

(c) user actuatable means for selecting one of said algorithms; and

(d) analysis means responsive to said user actuatable means, for receiving the video information provided by said video information source and for analyzing the received video information provided by the video information source in accordance with said one of said algorithms selected by said user actuatable means.

28. An apparatus according to claim 27, wherein said video information source includes a video camera which generates a dynamic sequence of video information frames.

29. An apparatus according to claim 27, wherein said video information source includes a plurality of video cameras which generate respective sequences of video information frames.

30. An apparatus according to claim 27, further comprising display means for displaying a plurality of icons each representing a respective one of said algorithms for analyzing video information.

31. An apparatus according to claim 30, wherein said user actuatable means comprises means for selecting one of the displayed icons.

32. An apparatus according to claim 31, wherein said means for selecting one of the displayed icons comprises a mouse.

33. An apparatus according to claim 31, wherein said means for displaying and said means for selecting one of the displayed icons comprise a touch-sensitive screen.

34. An apparatus according to claim 27, wherein said user actuatable means includes means for setting a parameter for constraining execution of a selected one of said algorithms.

35. An apparatus according to claim 34, wherein said means for setting said parameter includes location means for selecting a portion of an image plane which corresponds to said video information, said analysis means responding to said location means by analyzing the received video information with respect to the portion of the image plane selected by said location means.

36. An apparatus according to claim 35, wherein said apparatus is operative to display on said display means an image which corresponds to an image represented by said video information, and said location means includes drawing means for superimposing on said image displayed by said display means a line indicative of said selected portion of said image plane.

37. An apparatus according to claim 36, wherein said drawing means includes a mouse.

38. An apparatus according to claim 27, wherein at least one of said stored algorithms is for detecting moving objects represented by said received video information.

39. An apparatus according to claim 38, further comprising object means for selecting at least one of a maximum size and a minimum size of objects to be detected in accordance with said at least one moving object detection algorithm.

40. An apparatus according to claim 39, further comprising display means for displaying an image which corresponds to an image represented by said video information, and wherein said object means includes drawing means for superimposing on said image displayed by said display means an image element indicative of said at least one of a maximum size and a minimum size.

41. An apparatus according to claim 40, wherein said drawing means includes a mouse.

98. An apparatus according to claim 27, further comprising display means for displaying a plurality of character strings each representing a respective one of said algorithms for analyzing video information.

99. An apparatus according to claim 98, wherein said user actuatable means comprises means for selecting one of the displayed character strings.

100. An apparatus according to claim 99, wherein said means for selecting one of the displayed character strings comprises a mouse.

101. An apparatus according to claim 27, wherein said video information source is a video camera, and said analysis means analyzes video information generated by said video camera at substantially the same time that the video information is generated.

102. An apparatus according to claim 27, wherein said video information source is a video data base.

42. A method of analyzing video information, comprising the steps of:

receiving an analog video signal which represents a dynamic sequence of images;

digitizing the received analog video signal to form a sequence of video data frames which represent the dynamic sequence of images;

first analyzing the sequence of video data frames using a first predetermined video data analysis algorithm and generating first analysis data indicative of results of said first analyzing;

storing the sequence of video data frames in a data base;

storing the first analysis data in said data base in association with the stored sequence of video data frames;

retrieving from the data base the stored sequence of video frames;

second analyzing the retrieved sequence of video data frames using a second predetermined video data analysis algorithm and generating second analysis data indicative of results of said second analyzing; and

storing the second analysis data in said data base in association with the stored sequence of video data frames.

43. A method according to claim 42, wherein said first analyzing step is performed before said step of storing the sequence of video data frames in the data base.

44. An apparatus for analyzing video information, comprising:

receiving means for receiving an analog video signal which represents a dynamic sequence of images;

A/D means, connected to said receiving means, for digitizing the analog video signal to form a sequence of video data frames which represent the dynamic sequence of images;

first analysis means, connected to said A/D means, for analyzing the sequence of video data frames using a first predetermined video data analysis algorithm to generate first analysis data;

storage means, connected to said A/D means and said first analysis means, for storing the sequence of video data frames formed by said A/D means and said first analysis data generated by said first analysis means; and

second analysis means, connected to said storage means, for retrieving the sequence of video data frames stored in said storage means and for analyzing the retrieved sequence of video data frames using a second predetermined video data analysis algorithm to generate second analysis data, said second analysis data being stored in said storage means.

45. An apparatus according to claim 44, further comprising display means, connected to said storage means, for displaying at least one image represented by the stored sequence of images.

46. An apparatus according to claim 44, wherein said storage means includes:

compression means for receiving the sequence of video data frames from said A/D means and for applying a compression algorithm to the sequence of video data frames to form compressed video data; and

a storage medium on which said compressed video data is stored.

47. An apparatus according to claim 46, wherein said storage medium is a hard disk.

48. An apparatus according to claim 44, wherein said first analysis means includes a first processing device and said second analysis means includes a second processing device different from said first processing device.

49. A video information analysis apparatus, comprising:

means for receiving video information;

analysis means for analyzing the received video information in accordance with a predetermined analysis algorithm; and

parameter means, actuatable by a user of the apparatus, for setting a parameter to constrain execution of said analysis algorithm, said analysis means being responsive to said parameter means so as to execute said analysis algorithm in accordance with the parameter set by said parameter means.

50. An apparatus according to claim 49, further comprising display means for displaying an image which corresponds to an image represented by said video information, and wherein said parameter means includes drawing means, actuatable by said user of the apparatus, for superimposing on said displayed image a drawing element to indicate the parameter set by said parameter means.

51. An apparatus according to claim 50, wherein said drawing element is a line or polygon indicative of a location in an image plane at which the analysis algorithm is to be performed.

52. An apparatus according to claim 50, wherein said analysis algorithm is for detecting moving objects represented by said video information.

53. An apparatus according to claim 52, wherein said drawing element represents at least one of a maximum size and a minimum size of objects to be detected using said analysis algorithm.

54. An apparatus according to claim 52, wherein said drawing element represents at least one of a maximum velocity and a minimum velocity of objects to be detected by using said analysis algorithm.

55. An apparatus according to claim 52, wherein said drawing means includes a mouse.

56. An apparatus according to claim 49, wherein said parameter means includes display means for displaying a graphical user interface element and user input means for adjusting the graphical user interface element displayed by the display means.

57. An apparatus according to claim 56, wherein the displayed graphical user interface element is a slide bar.

58. An apparatus according to claim 56, wherein the displayed graphical user interface element is a dial.

59. An apparatus according to claim 56, wherein the user input device is a mouse.

60. A method of searching an image information data base, comprising the steps of:

storing a plurality of algorithms for identifying respective features in images represented by the image information in said data base;

selecting two of said stored algorithms for respectively identifying a first feature and a second feature;

selecting a logical operator to form a logical combination of said two selected algorithms; and

searching said data base to identify images which satisfy said logical combination of said two selected algorithms.

61. A method according to claim 60, wherein said image information data base includes data representative of at least one dynamic sequence of video information frames and at least one of said selected algorithms is for detecting a moving object in said at least one sequence of video information frames.

62. Apparatus for analyzing video information comprising:

first video information source means for providing a first stream of video data frames;

second video information source means for providing a second stream of video data frames;

first analysis means for receiving said first stream of video data frames provided by said first video information source means and performing a first moving image content analysis algorithm with respect to the received first stream of video data frames; and

second analysis means for receiving said second stream of video data frames provided by said second video information source means, and performing a second moving image content analysis algorithm with respect to the received second stream of video data frames.

63. Apparatus according to claim 62, wherein said first and second analysis means operate simultaneously in performing said first and second moving image content analysis algorithms.

64. Apparatus according to claim 62, wherein said first analysis means includes a first signal processing integrated circuit and said second analysis means includes a second signal processing integrated circuit separate from said first signal processing integrated circuit.

65. Apparatus according to claim 64, wherein said first signal processing integrated circuit is a digital signal processing integrated circuit and said second signal processing integrated circuit is a general purpose microprocessor.

66. Apparatus according to claim 62, wherein said first video information source means includes at least one video camera, and said video information source means includes a memory device from which said second stream of video data frames is retrieved.

67. Apparatus according to claim 66, wherein said memory device is selected from a group consisting of a hard disk, a magneto-optical disk, and a writable optical disk.

68. Apparatus according to claim 67, further comprising:

compression means for applying a data compression algorithm to said first stream of video data frames to produce compressed video data; and

means for transmitting said compressed video data from said compression means to said hard disk for storage on said hard disk.

69. Apparatus according to claim 68, further comprising means for retrieving said stored compressed video data from said hard disk and for applying a data de-compression algorithm to said retrieved compressed video data to form said second stream of video data frames.

70. Apparatus according to claim 66, wherein said first video information source means is a first video camera and said second video information source means is a second video camera.

71. Apparatus according to claim 70, wherein said first and second analysis means operate simultaneously in performing said first and second moving image content analysis algorithms.

72. Apparatus according to claim 67, wherein said first and second streams of video data are retrieved simultaneously from said hard disk and then respectively supplied to said first and second analysis means.

73. Apparatus for analyzing video information comprising:

a video camera for generating a stream of video data fields;

first analysis means for receiving said stream of video data fields provided by said video camera for performing a first moving image content analysis algorithm with respect to the received stream of video data fields; and

second analysis means for receiving said stream of video fields provided by said video camera, and for performing a second moving image content analysis algorithm with respect to the received stream of video data fields, said second moving image content analysis algorithm being different from said first moving image content analysis algorithm.

74. Apparatus according to claim 73, wherein said first and second analysis means operate simultaneously in performing said first and second moving image content analysis algorithms.

75. A video information analysis apparatus, comprising:

means for generating a first stream of dynamic images and a second stream of dynamic images;

variable-rate image selection means for selecting images for storage from said first stream of images at a first image interval rate, and for selecting images for storage from said second stream of images at a second image interval rate that is different from said first image interval rate;

storage means for storing said images selected for storage from said first stream of images by said image selection means and for storing said images selected for storage from said second stream of images by said image selection means;

playback means for selectively retrieving from said storage means said stored images from said first and second streams of images; and

analysis means for applying a moving image content analysis algorithm to images retrieved by said playback means from a selected one of said first and second streams of images, said analysis means adapting said moving image content analysis algorithm to the respective image interval rate of the selected stream of images.

76. A video information analysis apparatus according to claim 75, wherein said means for generating includes a first video camera for generating said first stream of dynamic images and a second video camera for generating said second stream of dynamic images.

77. A video information analysis apparatus according to claim 75, wherein said means for generating includes a video camera which generates said first stream of dynamic images during a first time period and which generates said second stream of dynamic images during a second time period that is later than said first time period.

78. A video information analysis apparatus according to claim 75, wherein said moving image content analysis algorithm is for detecting moving objects represented by the selected stream of images.

79. A video information analysis apparatus according to claim 78, further comprising display means for displaying images retrieved from said storage means, and wherein said moving image content analysis algorithm is for detecting moving objects at a selected portion of an image plane displayed on said display means.

80. A video information analysis apparatus according to claim 79, further comprising user input means for permitting user of the apparatus to select said portion of the image plane at which moving objects are to be detected by said moving image analysis algorithm.

81. A video information analysis apparatus according to claim 75, wherein said storage means comprises a hard disk, a magneto-optical disk and a writable optical disk.

82. A method of analyzing video information, comprising the steps of:

generating a first stream of dynamic images;

selecting images for storage from said first stream of images at a first image interval rate;

generating a second stream of dynamic images;

selecting images for storage from said second stream of images at a second image interval rate that is different from said first image interval rate;

storing said selected images from said first and second streams of dynamic images;

retrieving the stored images from the selected stream of dynamic images; and

applying a moving image content analysis algorithm to the retrieved images from the selected stream of dynamic images, said algorithm being adapted to the image interval rate of the selected stream of dynamic images.

83. A method according to claim 82, wherein said generating steps are performed by different respective video cameras.

84. A method according to claim 82, wherein said first and second streams are generated during different respective time periods by the same video camera.

85. A method according to claim 82, wherein said moving image content analysis algorithm is for detecting moving objects represented by the selected stream of images.

86. A method according to claim 82, wherein said storing step includes storing said selected images on recording medium selected from the group consisting of a hard disk, a magneto-optical disk and a writable optical disk.

87. A method according to claim 86, wherein said retrieving step includes retrieving said selected images from said recording medium.

88. A method of detecting moving objects represented by a sequence of dynamic images, said sequence of dynamic images being represented by stored video data, at least 75% of said dynamic images having been compression-encoded using an inter-image encoding algorithm such that the video data representing each inter-image encoded image includes image data consisting of a plurality of image data portions each corresponding to a respective region of an image plane, said video data for each inter-image encoded image also including mapping data indicative of the regions of the image plane to which the image data portions of said video data correspond, the method comprising the steps of:

retrieving at least the mapping data included in said stored video data; and

applying a motion detection algorithm to the retrieved mapping data to detect said moving objects.

89. A method according to claim 88, wherein at least 80% of said dynamic images have been compression-encoded using said inter-image encoding algorithm.

90. A method according to claim 89, wherein at least 90% of said dynamic images have been compression-encoded using said inter-image encoding algorithm.

91. A method according to claim 90, wherein at least 95% of said dynamic images have been compression-encoded using said inter-image encoding algorithm.

92. Apparatus for detecting moving objects represented by a sequence of dynamic images, said sequence of dynamic images being represented by stored video data, at least 75% of said dynamic images having been compressed-encoded using an inter-image encoding algorithm such that the video data representing each inter-image encoded image includes image data consisting of a plurality of image data portions each corresponding to a respective region of an image plane, said video data for each inter-image encoded image also including mapping data indicative of the regions of the image plane to which the image data portions of said video data correspond, the apparatus comprising:

means for retrieving at least the mapping data included in said stored video data; and

means for applying a motion detection algorithm to the retrieved mapping data to detect said moving objects.

93. Apparatus according to claim 92, wherein at least 90% of said dynamic images have been compression-encoded using said inter-image encoding algorithm.

103. A flexible video information analysis apparatus, comprising:

(a) a video information source for providing video information which includes at least one dynamic sequence of video information fields;

(b) storage means for storing a plurality of algorithms for analyzing the video information provided by the video information source;

(c) user actuatable means for selecting one of said algorithms; and

(d) analysis means responsive to said user actuatable means, for receiving the video information provided by said video information source and for analyzing the received video information provided by the video information source in accordance with said one of said algorithms selected by said user actuatable means.

104. A method of analyzing video information, comprising the steps of:

receiving an analog video signal which represents a dynamic sequence of images;

digitizing the received analog video signal to form a sequence of video data fields which represent the dynamic sequence of images;

first analyzing the sequence of video data fields using a first predetermined video data analysis algorithm and generating first analysis data indicative of results of said first analyzing;

storing the sequence of video data fields in a data base;

storing the first analysis data in said data base in association with the stored sequence of video data fields;

retrieving from the data base the stored sequence of video data fields;

second analyzing the retrieved sequence of video data fields using a second predetermined video data analysis algorithm and generating second analysis data indicative of results of said second analyzing; and

storing the second analysis data in said data base in association with the stored sequence of video data fields.

105. An apparatus for analyzing video information, comprising:

receiving means for receiving an analog video signal which represents a dynamic sequence of images;

A/D means, connected to said receiving means, for digitizing the analog video signal to form a sequence of video data fields which represent the dynamic sequence of images;

first analysis means, connected to said A/D means, for analyzing the sequence of video data fields using a first predetermined video data analysis algorithm to generate first analysis data;

storage means, connected to said A/D means and said first analysis means, for storing the sequence of video data fields formed by said A/D means and said first analysis data generated by said first analysis means; and

second analysis means, connected to said storage means, for retrieving the sequence of video data fields stored in said storage means and for analyzing the retrieved sequence of video data fields using a second predetermined video data analysis algorithm to generate second analysis data, said second analysis data being stored in said storage means.

106. Apparatus for analyzing video information comprising:

first video information source means for providing a first stream of video data fields;

second video information source means for providing a second stream of video data fields;

first analysis means for receiving said first stream of video data fields provided by said first video information source means and performing a first moving image content analysis algorithm with respect to the received first stream of video data fields; and

second analysis means for receiving said second stream of video data fields provided by said second video information source means, and performing a second moving image content analysis algorithm with respect to the received second stream of video data fields.