System and method for multi-modal audio mining of telephone conversations

• US 8,731,934 B2
• Filed: 02/15/2008
• Issued: 05/20/2014
• Est. Priority Date: 02/15/2007
• Status: Active Grant

• Current Assignee: DSI-ITI, Inc.
• Original Assignee: Digital Solutions Incorporated (DLH Holdings Corp.)
• Inventors: Olligschlaeger, Andreas M., Rokosky, James P.
• Primary Examiner(s): He, Jialong
• Application Number:
  US12/032,200
• Publication Number:
  US 20080201143A1
• Time in Patent Office: 2,286 Days
• Field of Search: 704/246, 704/270, 704/273, 379/67, 379/85, 379/191, 455/410, 455/411
• US Class Current: 704/270
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/686 using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 3/2281 Call monitoring, e.g. for law enforcement purposes; Call tracing; Detection or prevention of malicious calls
• CPC Class Codes: H04M 3/42221 Conversation recording systems at the subscriber's set H04M1/656

Abstract

A system and method for the automated monitoring of inmate telephone calls as well as multi-modal search, retrieval and playback capabilities for said calls. A general term for such capabilities is multi-modal audio mining. The invention is designed to provide an efficient means for organizations such as correctional facilities to identify and monitor the contents of telephone conversations and to provide evidence of possible inappropriate conduct and/or criminal activity of inmates by analyzing monitored telephone conversations for events, including, but not limited to, the addition of third parties, the discussion of particular topics, and the mention of certain entities.

Claims

1. A system for data-mining a monitored telephone conversation, the system comprising:

one or more microprocessors;

a memory;

a speech recognition processor configured to transcribe the monitored telephone conversation and associate a plurality of characteristics of the monitored telephone conversation with a transcript of the monitored telephone conversation;

an event detection processor configured to analyze the transcript of the monitored telephone conversation to detect one or more events within the monitored telephone conversation following the transcription of the monitored telephone conversation;

a topic database configured to include a slang term associated with both a major topic and a minor topic;

a topic detection processor configured to determine a topic of the monitored telephone conversation based upon querying the topic database with one or more extracted phrases from the transcript;

a tagging processor configured to associate metadata information indicative of the detected one or more events with the transcript of the monitored telephone conversation; and

a multimedia data warehouse configured to store at least the transcript of the monitored telephone conversation and the metadata information and characteristics associated therewith as a data record for the monitored telephone conversation.

2. The system according to claim 1, further comprising at least one domain specific language model, wherein the speech recognition processor is configured to utilize the at least one domain-specific language model to transcribe the monitored telephone conversation.

3. The system according to claim 2, wherein the at least one domain-specific language model is domain-specific to inmate speech.

4. The system according to claim 1, wherein the event detection processor comprises a three-way call detection processor configured to analyze at least the transcript of the monitored telephone conversation for an indication of a third party being added to the monitored telephone conversation.

5. The system according to claim 1, wherein the event detection processor comprises an entity identification processor configured to analyze at least the transcript of the monitored telephone conversation to identify one or more entities mentioned during the monitored telephone conversation, further comprising at least one entity database coupled to the event detection processor.

6. The system according to claim 5, wherein the at least one entity database includes at least one domain-specific Hidden Markov Model.

7. The system according to claim 6, wherein the at least one domain-specific Hidden Markov Model is domain-specific to inmate speech.

8. The system according to claim 5, wherein the entity identification processor is configured to search the at least one entity database using one or more Viterbi search algorithms to identify one or more entities mentioned during the monitored telephone conversation.

9. The system according to claim 5, wherein the entity identification processor is configured to apply one or more heuristic algorithms to at least the transcript of the monitored telephone conversation to heuristically identify one or more entities mentioned during the monitored telephone conversation.

10. The system according to claim 5, wherein the entity identification processor is configured to search the at least one entity database using one or more string matching algorithms to identify one or more entities mentioned during the monitored telephone conversation.

11. The system according to claim 1, wherein the topic detection processor is configured to search the topic database using one or more phrase-matching algorithms to identify the topics discussed during the monitored telephone conversation.

12. The system according to claim 1, wherein the topic database includes at least one domain-specific Hidden Markov Model and at least one domain-specific topic language model, and wherein the topic detection processor is configured to perform automatic topic annotation of the monitored telephone conversation.

13. The system according to claim 12, wherein the at least one domain-specific Hidden Markov Model and the at least one domain-specific topic language model are domain-specific to inmate speech.

14. The system according to claim 1, wherein the topic detection processor is configured to search the topic database using one or more text clustering algorithms to identify the topic discussed during the monitored telephone conversation.

15. The system according to claim 1, further comprising a synchronization processor configured to associate timing information for the detected one or more events with a sound recording of at least a portion of the monitored telephone conversation.

16. The system according to claim 1, further comprising a synchronization processor configured to associate position information for the detected one or more events with the transcript of the monitored telephone conversation.

17. The system according to claim 1, further comprising at least one of a spatial information database and a telephone subscriber information database coupled to the event detection processor, wherein the event detection processor is configured to associate information retrieved from at least one of the spatial information database and the telephone subscriber information database with at least the transcript of the monitored telephone conversation based upon the one or more events detected within the monitored telephone conversation.

18. The system according to claim 1, further comprising:

a multi-modal query interface configured to accept user-specified search criteria;

a search processor configured to search the multimedia data warehouse and retrieve one or more data records meeting the user-specified search criteria; and

a multi-modal output interface configured to display the retrieved one or more data records.

19. The system according to claim 1, further comprising:

a multi-modal query interface configured to accept user-specified search criteria;

a monitoring processor configured to monitor the multimedia data warehouse for addition of a data record meeting the user-specified search criteria; and

a notification processor configured to notify a user upon addition of the data record meeting the user-specified search criteria.

20. The system according to claim 1, wherein the minor topic is a subtopic of the major topic.

21. A method of data-mining a monitored telephone conversation, the method comprising:

transcribing a monitored telephone conversation;

extracting a plurality of characteristics of the monitored telephone conversation;

associating the extracted plurality of characteristics of the monitored telephone conversation with a transcript of the monitored telephone conversation;

analyzing the transcript of the monitored telephone conversation to detect one or more events within the monitored telephone conversation on a computing device following the transcription of the monitored telephone conversation;

determining a topic of the monitored telephone conversation based upon querying a topic database with one or more extracted phrases from the transcript, wherein the topic database includes a slang term associated with both a major topic and a minor topic;

associating metadata information indicative of the detected one or more events with the transcript of the monitored telephone conversation on a computing device; and

storing at least the transcript of the monitored telephone conversation and the characteristics and metadata information associated therewith in a multimedia data warehouse as a data record for the monitored telephone conversation.

22. The method according to claim 21, further comprising:

atomizing the data record for the monitored telephone conversation with respect to a plurality of data records stored within the multimedia data warehouse; and

creating one or more logical links between the data record for the monitored telephone conversation and one or more of the plurality of data records stored within the multimedia data warehouse.

23. The method according to claim 21, wherein the step of analyzing the transcript of the monitored telephone conversation to detect one or more events within the monitored telephone conversation comprises analyzing at least the transcript of the monitored telephone conversation for an indication a third party was added to the monitored telephone conversation.

24. The method according to claim 21, wherein the step of analyzing the transcript of the monitored telephone conversation to detect one or more events within the monitored telephone conversation comprises analyzing at least the transcript to identify one or more entities mentioned during the monitored telephone conversation.

25. The method according to claim 21, further comprising associating timing information for the detected one or more events with a sound recording of at least a portion of the monitored telephone conversation.

26. The method according to claim 21, further comprising associating position information for the detected one or more events with the transcript of the monitored telephone conversation.

27. The method according to claim 21, wherein the minor topic is a subtopic of the major topic.