Multimedia data analysis in intelligent video information management system

• US 6,035,341 A
• Filed: 04/29/1998
• Issued: 03/07/2000
• Est. Priority Date: 10/31/1996
• Status: Expired due to Term

• Current Assignee: Sensormatic Electronics LLC (Johnson Controls International plc)
• Original Assignee: Sensormatic Electronics Corporation (Johnson Controls International plc)
• Inventors: Nunally, Patrick O., MacCormack, David Ross
• Primary Examiner(s): Peeso, Thomas R.
• Application Number:
  US09/069,508
• Time in Patent Office: 678 Days
• Field of Search: 709/261, 709/266, 709/247
• US Class Current: 709/253
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: G08B 13/19669 Event triggers storage or change of storage policy
• CPC Class Codes: G08B 13/1968 Interfaces for setting up or customising the system
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: G08B 21/086 by monitoring a perimeter outside the body of the water
• CPC Class Codes: G10L 25/00 Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 muting semiconductor-based amplifiers when some special characteristics of a signal are sensed by a speech detector, e.g. sensing when no signal is present, H03G3/34
• CPC Class Codes: H04N 5/445 for displaying additional information H04N5/50 takes precedence
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 5/782 on tape
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 9/8042 involving data reduction
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal

Abstract

A flexible multimedia information analysis apparatus stores a database which includes both audio and video information. Also stored are a plurality of content analysis algorithms for analyzing the audio information in the database. A user can manipulate a mouse to select one of the analysis algorithms. The selected algorithm is used to analyze audio information in the database or on a "live" basis as the audio information is received. Audio analysis algorithms can be applied in tandem with moving image content analysis algorithms applied to stored video data or live input video streams.

Claims

1. A flexible multimedia information analysis apparatus, comprising:

(a) first storage means for storing a multimedia information data base, the data base including image data and sound data;

(b) second storage means for storing a plurality of sound data content analysis algorithms for analyzing sound data in the data base;

(c) user actuatable means for selecting one of said algorithms; and

(d) analysis means, responsive to said user actuatable means, for analyzing said sound data in the data base in accordance with said one of said algorithms selected by said user actuatable means.

2. An apparatus according to claim 1, further comprising:

a plurality of microphones for generating respective sequences of analog sound signals; and

means for digitizing the respective sequences of analog sound signals generated by said microphones and for supplying the digitized sequences of sound signals to said first storage means.

3. An apparatus according to claim 1, further comprising means for displaying a plurality of icons or alphanumeric labels each representing a respective one of said algorithms for analyzing sound data.

4. An apparatus according to claim 3, wherein said user actuatable means comprises means for selecting one of the displayed icons or alphanumeric labels.

5. An apparatus according to claim 4, wherein said means for selecting one of the displayed icons comprises a mouse.

6. An apparatus according to claim 1, wherein said user actuatable means includes means for setting a parameter for constraining execution of a selected one of said algorithms.

7. An apparatus according to claim 1, wherein said analysis means generates analysis data indicative of results of the analysis performed by said analysis means in accordance with said selected one of said algorithms and said first storage means includes means for storing said analysis data generated by said analysis means.

8. An apparatus according to claim 1, wherein at least one of said stored algorithms is for detecting an amplitude level of the sound data analyzed by the algorithm.

9. An apparatus according to claim 1, wherein at least one of said stored algorithms is for detecting a frequency characteristic of the sound data analyzed by the algorithm.

10. An apparatus according to claim 1, further comprising third storage means for storing a plurality of moving image content analysis algorithms for analyzing image data in the data base;

said user actuatable means being operable to select one of said moving image content analysis algorithms, said analysis means being responsive to selection of one of said moving image content analysis algorithms to analyze said image data in the data base in accordance with the selected moving image content analysis algorithm.

11. A method of storing and retrieving audio data, comprising the steps of:

storing data representing sound signals generated over a period of time;

selecting a search algorithm to be used in searching for a desired sound segment represented by the stored data;

setting a parameter for constraining execution of the selected search algorithm; and

searching said stored data in accordance with the elected search algorithm and the set parameter.

12. A method according to claim 11, wherein the selected search algorithm is for detecting an amplitude level of the sound signals represented by the stored data.

13. A method according to claim 12, wherein the parameter for constraining execution of the selected search algorithm represents a sound amplitude level.

14. A method according to claim 11, wherein the selected search algorithm is for detecting a frequency characteristic of the sound signals represented by the stored data.

15. A method according to claim 14, wherein the parameter for constraining execution of the selected search algorithm is for defining the frequency characteristic to be detected.

16. A flexible multimedia information analysis apparatus, comprising:

(a) an audio information source for providing audio information which represents at least one sequence of sound signals;

(b) storage means for storing a plurality of algorithms for analyzing the audio information provided by the audio information source;

(c) user actuatable means for selecting one of said algorithms; and

(d) analysis means responsive to said user actuatable means, for receiving the audio information provided by said audio information source and for analyzing the received audio information provided by the audio information source in accordance with said one of said algorithms selected by said user actuatable means.

17. An apparatus according to claim 16, wherein said audio information source includes a microphone for generating a sequence of sound signals, and conversion means for converting the sound signals to digital audio data.

18. An apparatus according to claim 17, wherein said audio information source includes a plurality of microphones which generate respective sequences of sound signals.

19. An apparatus according to claim 16, further comprising display means for displaying a plurality of icons or alphanumeric labels each representing a respective one of said algorithms for analyzing audio information.

20. An apparatus according to claim 19, wherein said user actuatable means comprises means for selecting one of the displayed icons or alphanumeric labels.

21. An apparatus according to claim 20, wherein said means for selecting one of the displayed icons comprises a mouse.

22. An apparatus according to claim 16, wherein said user actuatable means includes means for setting a parameter for constraining execution of a selected one of said algorithms.

23. An apparatus according to claim 16, wherein at least one of said stored algorithms is for detecting an amplitude level of the audio information analyzed by the algorithm.

24. An apparatus according to claim 16, wherein at least one of said stored algorithms is for detecting a frequency characteristic of the audio information analyzed by the algorithm.

25. A method of analyzing multimedia information, comprising the steps of:

(a) receiving an analog video signal which represents a dynamic sequence of images;

(b) digitizing the received analog video signal to form a sequence of video data frames which represent the dynamic sequence of images;

(c) first analyzing the sequence of video data frames using a predetermined video data analysis algorithm and generating first analysis data indicative of results of said first analyzing;

(d) storing the sequence of video data frames in a data base;

(e) storing the first analysis data in said data base in association with the stored sequence of video data frames;

simultaneously with said steps (a) to (e),

(f) receiving an analog audio signal which represents a sequence of audible sounds;

(g) digitizing the received analog audio signal to form a sequence of audio data which represents the audible sounds;

(h) second analyzing the sequence of audio data using a predetermined audio data analysis algorithm and generating second analysis data indicative of results of said second analyzing;

(i) storing the sequence of audio data in said data base; and

(j) storing the second analysis data in said data base in association with the stored sequence of audio data.

26. A method according to claim 25, wherein said second analyzing step includes detecting an amplitude level of said analog audio signal.

27. A method according to claim 25, wherein said second analyzing step includes detecting a frequency characteristic of said analog audio signal.

28. A method according to claim 25, wherein said first analyzing step includes detecting a moving object represented by said analog video signal.

29. A method according to claim 25, wherein said steps (d), (e), (i) and (j) include storing the video data frames, the audio data, the first analysis data and the second analysis data on at least one hard disk.

30. A method according to claim 29, wherein said steps (d) and (i) include storing the video data frames and the audio data together on the same hard disk.

31. A method according to claim 25, wherein said video data frames and said audio data are stored in a format which conforms to the .AVI standard.

32. An apparatus for analyzing video and audio information, comprising:

first receiving means for receiving an analog video signal which represents a dynamic sequence of images;

first A/D means, connected to said first receiving means, for digitizing the analog video signal to form a sequence of video data frames which represent the dynamic sequence of images;

first analysis means, connected to said first A/D means, for analyzing the sequence of video data frames using a predetermined video data analysis algorithm to generate first analysis data;

storage means, connected to said first A/D means and said first analysis means, for storing the sequence of video data frames formed by said first A/D means and said first analysis data generated by said first analysis means;

second receiving means for receiving an analog audio signal which represents a sequence of audible sounds;

second A/D means, connected between said second receiving means and said storage means, for digitizing the analog audio signal to form a sequence of audio data which represents the audible sounds;

said storage means storing said sequence of audio data formed by said second A/D means; and

second analysis means, connected between said second A/D means and said storage means, for analyzing the sequence of audio data using a predetermined audio data analysis algorithm to generate second analysis data, said second analysis data being stored in said storage means.

33. An apparatus according to claim 32, wherein said storage means includes:

compression means for receiving the sequence of video data frames from said A/D means and for applying a compression algorithm to the sequence of video data frames to form compressed video data; and

a storage medium on which said compressed video data is stored.

34. An apparatus according to claim 33, wherein said storage medium is a hard disk.

35. An apparatus according to claim 32, wherein said audio data analysis algorithm includes a process for detecting a predetermined frequency characteristic of said sequence of audible sounds.

36. An apparatus for analyzing multimedia information as said information is received, the apparatus comprising:

at least one video camera for generating a sequence of video signals;

at least one microphone for generating a sequence of audio signals;

conversion means, connected to said at least one video camera and said at least one microphone, for converting said video signals and said audio signals into video data and audio data, respectively; and

analysis means, connected to said conversion means, for applying at least one video information analysis algorithm to said video data, and for applying at least one audio information analysis algorithm to said audio data.

37. An apparatus according to claim 36, wherein said analysis means includes a program memory for storing a plurality of video analysis programs each corresponding to a respective video information analysis algorithm, said program memory also storing a plurality of audio analysis programs each corresponding to a respective audio information analysis algorithm;

the apparatus further comprising selection means for permitting a user to select a one of said video analysis programs for controlling said analysis means and a one of said audio analysis programs for controlling said analysis means.

38. An apparatus according to claim 37, wherein said plurality of video information analysis algorithms corresponding to said video analysis programs include a plurality of algorithms for detecting moving objects represented by said video data.

39. An apparatus according to claim 37, wherein said plurality of audio information analysis algorithms corresponding to said audio analysis programs include a plurality of algorithms for detecting frequency characteristics of audio signals represented by said audio data.

40. An apparatus according to claim 37, wherein said selection means permits the user to set respective parameters for constraining execution of the video information analysis algorithms and the audio information analysis algorithms.

41. An audio information analysis apparatus, comprising:

means for receiving audio information;

analysis means for analyzing the received audio information in accordance with a predetermined analysis algorithm; and

parameter means, actuatable by a user of the apparatus, for setting a parameter to constrain execution of said analysis algorithm, said analysis means being responsive to said parameter means so as to execute said analysis algorithm in accordance with the parameter set by said parameter means.

42. A method of searching a multimedia information data base, comprising the steps of:

storing a plurality of algorithms for identifying respective features represented by audio information in said data base;

selecting two of said stored algorithms for respectively identifying a first feature and a second feature;

selecting a logical operator to form a logical combination of said two selected algorithms; and

searching said data base to identify audio data sequences which satisfy said logical combination of said two selected algorithms.

43. A method of analyzing multimedia data, the method comprising the steps of:

(a) simultaneously receiving a stream of video data and a stream of audio data;

(b) simultaneously with step (a), applying a first analysis algorithm to the received stream of video data to determine whether a first feature is present in said stream of video data;

(c) simultaneously with steps (a) and (b), applying a second analysis algorithm to the received stream of audio data to determine whether a second feature is present in said stream of audio data; and

(d) if it is determined that said first feature is present in said stream of video data at a time when said second feature is determined to be present in said stream of audio data, generating an alarm signal.

44. A method of analyzing multimedia data according to claim 43, wherein:

said first analysis algorithm is for detecting motion indicative of breaking of a window; and

said second analysis algorithm is for detecting a sound indicative of breaking of the window.

45. A method of analyzing multimedia data according to claim 43, wherein:

said first analysis algorithm is for detecting motion indicative of a splash in a swimming pool; and

said second analysis algorithm is for detecting a sound indicative of a splash in the swimming pool.

46. A video surveillance system, comprising:

a video camera for generating a stream of video signals;

storage means, connected to said video camera, for selectively storing the stream of video signals;

a microphone for generating a stream of analog audio signals;

conversion means, connected to said microphone, for converting the stream of analog audio signals into a sequence of audio data signals; and

analysis means for receiving the sequence of audio data signals and applying a selected audio information analysis algorithm to the sequence of audio data signals to detect a predetermined feature of the stream of analog audio signals;

the system being operable in a mode such that said storage means is responsive to said analysis means so as to store the stream of video signals only at times when said analysis means detects the predetermined feature of the stream of analog audio signals.

47. A video surveillance system according to claim 46, further comprising a housing in which the storage means and the analysis means are contained.

48. A video surveillance system according to claim 47, wherein said storage means includes a hard disk for storing the stream of video signals in digital form.

49. A video surveillance system according to claim 47, wherein said conversion means is contained in said housing.

50. Apparatus for analyzing audio information, comprising:

a microphone for generating a stream of analog audio signals;

conversion means, connected to said microphone, for converting the stream of analog signals into a sequence of audio data signals;

first analysis means for receiving the sequence of audio data signals formed by the conversion means and for performing a first audio information content analysis algorithm with respect to the received sequence of audio data signals; and

second analysis means for receiving the sequence of audio data signals formed by the conversion means and for performing a second audio information content analysis algorithm with respect to the received sequence of audio data signals, said second audio information content analysis algorithm being different from said first audio information content analysis algorithm.

51. Apparatus according to claim 50, wherein said first analysis means detects an amplitude characteristic of the stream of analog audio signals and the second analysis means detects a frequency characteristic of the stream of analog audio signals.

52. Apparatus according to claim 50, wherein said first analysis means detects a first frequency characteristic of the stream of analog audio signals and the second analysis means detects a second frequency characteristic of the stream of analog audio signals, said second frequency characteristic being different from said first frequency characteristic.

53. Video information storage apparatus, comprising:

means for receiving a television broadcast signal, said signal including a video component and an audio component;

storage means, connected to said means for receiving, for selectively storing the received television broadcast signal in digital form; and

monitoring means for analyzing content of the audio component of the television broadcast signal;

wherein said storage means is responsive to the monitoring means such that said storage means stores the television broadcast signal only when the monitoring means detects a predetermined characteristic of the audio component of the television broadcast signal.

54. Video information storage apparatus according to claim 53, wherein said means for receiving includes a tuner connected to an antenna for receiving an off-air broadcast signal.

55. Video information storage apparatus according to claim 53, wherein said means for receiving includes a set-top box connected to a cable television system.

56. Video information storage apparatus according to claim 53, wherein:

the television broadcast signal represents a sporting event; and

said predetermined characteristic of the audio component of the television broadcast signal is an increase in an audience noise component of the audio component.

57. Video information storage apparatus, comprising:

means for receiving a television broadcast signal, said signal including a video component and an audio component;

storage means, connected to said means for receiving, for selectively storing the received television broadcast signal in digital form; and

monitoring means for analyzing content of the audio component of the television broadcast signal;

wherein said storage means is responsive to the monitoring means such that said storage means stores the television broadcast signal only when the monitoring means detects a predetermined characteristic of the video component of the television broadcast signal.

58. Video information storage apparatus, comprising:

means for receiving a television broadcast signal, said signal including a video component, an audio component, and a closed-captioning component;

storage means, connected to said means for receiving, for selectively storing in digital form portions of said television broadcast signal; and

monitoring means for detecting contents of the closed-captioning component of the television broadcast signal;

wherein said storage means is responsive to the monitoring means such that said storage means stores the television signal only when the monitoring means detects a predetermined character string in the closed-captioning component of the television broadcast signal.

59. Video information storage apparatus, comprising:

means for receiving a television broadcast signal;

storage means, connected to said means for receiving, for storing the received television broadcast signal;

monitoring means, connected to said means for receiving, for detecting contents of one of a video component of said received television broadcast signal, an audio component of said received broadcast television signal, and a closed-captioning component of said received broadcast television signal; and

index means, responsive to said monitoring means, for causing said storage means to store indexing information indicative of the detected contents of the television broadcast signal.