System and method for multi-modal audio mining of telephone conversations

• US 9,552,417 B2
• Filed: 05/19/2014
• Issued: 01/24/2017
• Est. Priority Date: 02/15/2007
• Status: Active Grant

• Current Assignee: Global Tel-Link Corporation
• Original Assignee: Global Tel-Link Corporation
• Inventors: Olligschlaeger, Andreas M., Rokosky, Jim
• Primary Examiner(s): HE, JIALONG
• Application Number:
  US14/281,323
• Publication Number:
  US 20140254777A1
• Time in Patent Office: 981 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/686 using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 3/2281 Call monitoring, e.g. for law enforcement purposes; Call tracing; Detection or prevention of malicious calls
• CPC Class Codes: H04M 3/42221 Conversation recording systems at the subscriber's set H04M1/656

Abstract

A system and method for the automated monitoring of inmate telephone calls as well as multi-modal search, retrieval and playback capabilities for said calls. A general term for such capabilities is multi-modal audio mining. The invention is designed to provide an efficient means for organizations such as correctional facilities to identify and monitor the contents of telephone conversations and to provide evidence of possible inappropriate conduct and/or criminal activity of inmates by analyzing monitored telephone conversations for events, including, but not limited to, the addition of third parties, the discussion of particular topics, and the mention of certain entities.

Claims

1. A system for data-mining a monitored telephone conversation, the system comprising:

a multimedia data warehouse that stores a data record associated with a corresponding monitored telephone conversation;

a data transformation processor configured to receive data from at least one data source, the received data including at least audio of a monitored telephone conversation and metadata information regarding the monitored telephone conversation;

a data augmentation processor configured to augment the received data with data retrieved from one or more of a telephone subscriber database and a spatial information database based upon the metadata information regarding the monitored telephone conversation;

a speech recognition processor configured to transcribe the monitored telephone conversation, detect a plurality of characteristics of the monitored telephone conversation, and associate the detected plurality of characteristics with a transcript of the monitored telephone conversation; and

a data mapping processor configured to generate the data record for the monitored telephone conversation and store the data record to the multimedia data warehouse, the data record including at least the transcript of the monitored telephone conversation, the plurality of characteristics of the monitored telephone conversation, and the augmented received data that includes the audio, the metadata information, and the retrieved data.

2. The system according to claim 1, further comprising:

a data normalization processor configured to atomize the data record for the monitored telephone conversation with respect to the multimedia data warehouse; and

a link creation processor configured to create one or more logical links between the data record for the monitored telephone conversation and one or more data records within the multimedia data warehouse.

3. The system according to claim 2, wherein the data normalization processor is configured to utilize a fuzzy logic algorithm to atomize the data record for the monitored telephone conversation with respect to the multimedia data warehouse.

4. The system according to claim 2, wherein the data normalization processor is configured to utilize a matching algorithm to atomize the data record for the monitored telephone conversation with respect to the multimedia data warehouse.

5. The system according to claim 4, wherein the matching algorithm is a soundex algorithm.

6. The system according to claim 2, wherein the data normalization processor is configured to utilize a natural language processing algorithm to atomize the data record for the monitored telephone conversation with respect to the multimedia data warehouse.

7. The system according to claim 1, wherein the at least one data source comprises an inmate telephone system.

8. The system according to claim 7, wherein the at least one data source further comprises an inmate management system.

9. The system according to claim 7, wherein the at least one data source further comprises an investigative case management system.

10. The system according to claim 1, wherein the at least one data source originates from a single correctional facility.

11. The system according to claim 1, wherein the at least one data source originates from a plurality of correctional facilities.

12. A method of data-mining a monitored telephone conversation, the method comprising:

receiving data from at least one data source, the received data including at least audio of a monitored telephone conversation and metadata information regarding the monitored telephone conversation;

retrieving data from one or more of a telephone subscriber database and a spatial information database based upon the metadata information regarding the monitored telephone conversation;

associating the retrieved data with the received data;

transcribing the monitored telephone conversation;

extracting a plurality of characteristics of the monitored telephone conversation, the plurality of characteristics being aspects of the monitored telephone conversation other than spoken words detected by the transcribing of the monitored telephone conversation;

associating the extracted plurality of characteristics of the monitored telephone conversation with a transcript of the monitored telephone conversation;

generating a data record for the monitored telephone conversation, the data record including at least the transcript of the monitored telephone conversation, the plurality of characteristics of the monitored telephone conversation, the metadata information regarding the monitored telephone conversation, and the retrieved data; and

storing the data record for the monitored telephone conversation to a multimedia data warehouse.

13. A system for data-mining a monitored telephone conversation, the system comprising:

one or more microprocessors;

a memory;

a speech recognition processor configured to transcribe the monitored telephone conversation of a user, detect a plurality of characteristics of the monitored telephone conversation, and associate the detected plurality of characteristics with a transcript of the monitored telephone conversation;

an entity database configured to associate a plurality of entities with the user, the plurality of entities corresponding to particular people, places, and/or things that may be mentioned by the user during the monitored telephone conversation;

an entity identification processor configured to analyze the transcript of the monitored telephone conversation to identify one or more entities from among the plurality of entities associated with the user mentioned during the monitored telephone conversation, following the transcribing of the monitored telephone conversation;

an event detection processor configured to analyze the transcript of the monitored telephone conversation to detect one or more events within the monitored telephone conversation based on the identified one or more entities;

a tagging processor configured to associate metadata information indicative of the detected one or more events with the transcript of the monitored telephone conversation; and

a multimedia data warehouse configured to store at least the transcript of the monitored telephone conversation and the metadata information and the characteristics associated therewith as a data record for the monitored telephone conversation.

14. The system according to claim 13, wherein the entity database includes a domain-specific Hidden Markov Model.

15. The system according to claim 14, wherein the domain-specific Hidden Markov Model is domain-specific to inmate speech.

16. The system according to claim 13, wherein the entity identification processor is configured to search the entity database using one or more Viterbi search algorithms to identify one or more entities mentioned during the monitored telephone conversation.

17. The system according to claim 13, wherein the entity identification processor is configured to apply one or more heuristic algorithms to at least the transcript of the monitored telephone conversation to heuristically identify one or more entities mentioned during the monitored telephone conversation.

18. The system according to claim 13, wherein the entity identification processor is configured to search the entity database using one or more string matching algorithms to identify one or more entities mentioned during the monitored telephone conversation.

19. The system according to claim 13, further comprising:

a multi-modal query interface configured to accept user-specified search criteria;

a search processor configured to search the multimedia data warehouse and retrieve one or more data records meeting the user-specified search criteria; and

a multi-modal output interface configured to display the retrieved one or more data records.

20. The system according to claim 13, further comprising:

a multi-modal query interface configured to accept user-specified search criteria;

a monitoring processor configured to monitor the multimedia data warehouse for addition of a data record meeting the user-specified search criteria; and

a notification processor configured to notify a user upon addition of the data record meeting the user-specified search criteria.