System and method for multi-modal audio mining of telephone conversations

• US 10,120,919 B2
• Filed: 01/23/2017
• Issued: 11/06/2018
• Est. Priority Date: 02/15/2007
• Status: Active Grant

• Current Assignee: Global Tel-Link Corporation
• Original Assignee: Global Tel-Link Corporation
• Inventors: Olligschlaeger, Andreas M., Rokosky, Jim
• Primary Examiner(s): He, Jialong
• Application Number:
  US15/412,992
• Publication Number:
  US 20170147662A1
• Time in Patent Office: 652 Days
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00
• CPC Class Codes: G06F 16/686 using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 3/2281 Call monitoring, e.g. for law enforcement purposes; Call tracing; Detection or prevention of malicious calls
• CPC Class Codes: H04M 3/42221 Conversation recording systems at the subscriber's set H04M1/656

Abstract

A system and method for the automated monitoring of inmate telephone calls as well as multi-modal search, retrieval and playback capabilities for said calls. A general term for such capabilities is multi-modal audio mining. The invention is designed to provide an efficient means for organizations such as correctional facilities to identify and monitor the contents of telephone conversations and to provide evidence of possible inappropriate conduct and/or criminal activity of inmates by analyzing monitored telephone conversations for events, including, but not limited to, the addition of third parties, the discussion of particular topics, and the mention of certain entities.

Claims

1. A system for multi-modal audio mining of telephone data, the system comprising:

one or more circuits and/or processors configured to;

retrieve a data record from a call database;

augment the retrieved data record with subscriber information and spatial information;

analyze the augmented data record after it has been augmented to verify that each data element of the augmented data record is formatted consistent with a relevant format of a plurality of application specific data warehouses;

map each data element within the augmented data record to the plurality of application specific data warehouses; and

provide a multi-modal query and visualization of the augmented data record to identify events of interest,

wherein the augmented data record includes the spatial information, the subscriber information, a transcript of the monitored telephone conversation, a plurality of characteristics of the monitored telephone conversation and the retrieved data that includes audio of the monitored telephone conversation and metadata information of the monitored telephone conversation.

2. The system of claim 1, wherein the retrieved data record includes superfluous information that includes at least one of leading and trailing blanks and non-alphanumeric characters from strings.

3. The system of claim 1, wherein the spatial information includes geographic information relating to a called telephone number.

4. The system of claim 1, wherein each of the application specific data warehouses is a specialized database that stores information of a common type.

5. The system of claim 1, wherein the one or more circuits and/or processors are configured to map each data element by:

storing each data element in a respective one of the plurality of application specific data warehouses; and

generating and maintaining a master table for the data record that includes each data element of the augmented data record, and a corresponding specialized data warehouse in which each data element is stored.

6. The system of claim 1, wherein the one or more circuits and/or processors are further configured to:

remove superfluous information from the retrieved data record;

standardize data elements within the retrieved data record; and

transcribe the monitored telephone conversation, detect a plurality of characteristics of the monitored telephone conversation, and associate the detected plurality of characteristics with a transcript of the monitored telephone conversation.

7. A system for data-mining a monitored telephone conversation, the system comprising:

a data transformer configured to receive data from a data source, the received data including audio of a monitored telephone conversation and metadata information of the monitored telephone conversation;

a data augmenter configured to augment the received data with telephone subscriber data and spatial information data;

a speech recognizer configured to transcribe the monitored telephone conversation, detect a plurality of characteristics of the monitored telephone conversation, and associate the detected plurality of characteristics with a transcript of the monitored telephone conversation; and

a data mapper configured to generate a data record for the monitored telephone conversation and store the data record to a multimedia data warehouse, the data record including at least the transcript of the monitored telephone conversation, the plurality of characteristics of the monitored telephone conversation and the augmented received data that includes the audio, the metadata information, the telephone subscriber data and the spatial information data.

8. The system of claim 7, wherein the spatial information data includes geographic location information of a called number.

9. The system of claim 7, wherein the multimedia data warehouse includes a plurality of application specific databases, and

wherein the data mapper is further configured to map each data element within the data record to the plurality of application specific databases.

10. The system of claim 9, wherein the data mapper is further configured to map each data element by:

storing each data element in a respective one of the plurality of application specific databases; and

generating and maintaining a master table for the data record that includes each data element of the data record, and a corresponding specialized data warehouse in which each data element is stored.

11. The system of claim 7, further comprising:

a data normalization processor configured to atomize the data record for the monitored telephone conversation with respect to the multimedia data warehouse; and

a link creation processor configured to create one or more logical links between the data record for the monitored telephone conversation and one or more data records within the multimedia data warehouse.

12. The system of claim 11, wherein the data normalization processor is configured to utilize a matching algorithm to atomize the data record for the monitored telephone conversation with respect to the multimedia data warehouse.

13. A method for multi-modal audio mining of telephone data, the method comprising:

retrieving a data record from a call database;

augmenting the retrieved data record with subscriber information and spatial information;

analyzing the augmented data record to verify that each data element of the augmented data record is formatted consistent with a relevant format of a plurality of application specific data warehouses;

mapping each data element within the augmented data record to the plurality of application specific data warehouses; and

providing a multi-modal query and visualization of the augmented data record to identify events of interest,

wherein the augmented data record includes the spatial information, the subscriber information, a transcript of a monitored telephone conversation, a plurality of characteristics of the monitored telephone conversation and the retrieved data that includes audio of the monitored telephone conversation and metadata information of the monitored telephone conversation.

14. The method of claim 13, wherein the retrieved data record includes superfluous information including at least one of leading and trailing blanks and non-alphanumeric characters from strings.

15. The method of claim 13, wherein the spatial information includes geographic information relating to a called telephone number.

16. The method of claim 13, wherein each of the application specific data warehouses is a specialized database that stores information of a common type.

17. The method of claim 13, wherein the mapping each data element includes:

storing each data element in a respective one of the plurality of application specific data warehouses; and

generating and maintaining a master table for the data record that includes each data element of the data record, and a corresponding specialized data warehouse in which each data element is stored.