Method and apparatus for automatically identifying and annotating auditory signals from one or more parties

• US 10,070,283 B2
• Filed: 03/13/2014
• Issued: 09/04/2018
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Eolas Technologies, Inc.
• Original Assignee: Eolas Technologies, Inc.
• Inventors: Martin, David C., Doyle, Michael D.
• Primary Examiner(s): Wang-Hurst, Kathy
• Assistant Examiner(s): Perez, Julio
• Application Number:
  US14/208,278
• Publication Number:
  US 20140308933A1
• Time in Patent Office: 1,636 Days
• Field of Search: 455413, 4554121, 4554122, 704251
• CPC Class Codes: H04M 2250/68 with means for recording information, e.g. telephone number during a conversation recording of telephone conversations H04M1/656
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

In one embodiment, a mobile device application automatically identifies and annotates auditory data of a conversation between two or more parties. Digital auditory data is processed to identify mention of specific entities. An identified entity is annotated, based on context and relevance, to list one or more actions possible to perform with or on the identified entity, and the identified entity displayed while the conversation is ongoing. An action is selected by signals from an input interface.

Claims

1. A telecommunication system comprising:

one or more processors; and

a computer readable memory storing program code which, when executed by the one or more processors, causes the one or more processors to at least;

after a communication is established by a first electronic communication device and a second electronic communication device, receive, at the telecommunication system, from the first electronic communication device, digital auditory data corresponding to an ongoing conversation between a first person using the first electronic communication device and the second person using a second electronic communication device, the conversation including a plurality of words spoken by the first person and the second person during the ongoing conversation;

while the conversation is ongoing between the first person using the first electronic communication device and the second person using the second electronic communication device, process the digital auditory data to identify mention of specific entities including one or more of telephone numbers, email addresses, dates, times, locations and proper names, the processing of the digital auditory data including;

identifying high priority of the specific entities;

facilitating disambiguation of the identified high priority specific entities;

identifying an utterance of a first entity in the plurality of words, and determining one or more actions possible to perform with respect to the identified first entity, the one or more actions determined based on context and relevance of the identified first entity with respect to the plurality of words of the conversation;

annotate the identified first entity to create an annotated entity with text listing the one or more actions possible to perform with respect to the identified first entity; and

while the conversation is ongoing between the first person using the first electronic communication device and the second person using the second electronic communication device, cause a display device of the first electronic communication device to display the annotated entity such that the first electronic communication device,

while the conversation is ongoing between the first person using the first electronic communication device and the second person using the second electronic communication device, obtains, via an input device of the first electronic communication device, a user interaction from the first person, the user interaction is associated with a selection of an action included in the displayed annotated entity; and

responsive to the user interaction, perform the selected action with respect to the identified entity.

2. The telecommunication system of claim 1,

which includes a cloud-based phone recognition and entity identification, annotation and relevance processing system.

3. The telecommunication system of claim 1,

which includes a speech-to-text converter.

4. The telecommunication system of claim 3, where the program code, when executed, further causes the one or more processors to:

while the conversation is ongoing between the first person using the first electronic communication device and the second person using the second electronic communication device, cause the display device of the first electronic communication device to display blocks of converted text on the display device, such that the first electronic communication device;

while the conversation is ongoing between the first person using the first electronic communication device and the second person using the second electronic communication device, processes user interaction with the input device to select a block of converted text; and

while the conversation is ongoing between the first person using the first electronic communication device and the second person using the second electronic communication device, processes only the selected block of converted text.

5. The telecommunication system of claim 4, where the program code, when executed, further causes the one or more processors to cause the display device of the first electronic communication device to display the blocks of converted text on the display device, such that the first electronic communication device:

facilitates parsing the selected block of converted text to distinguish facts from message content;

facilitates searching online reference material for information related to a distinguished fact; and

facilitates displaying, via the display device, the information.

6. A telecommunication system comprising:

one or more processors; and

a computer readable memory, coupled to the one or more processors, storing program code which, when executed by the one or more processors, causes the one or more processors to at least;

after a communication is established by a first electronic communication device and a second electronic communication device, receive, at the telecommunication system, from the first electronic communication device, digital auditory data corresponding to an ongoing conversation between a first party using the first electronic communication device and a second party using the second electronic communication device during an interview, wherein the conversation includes a plurality of words spoken by the first party and the second party, and wherein the first party is an interviewer and the second party is an interviewee;

while the conversation is ongoing between the first party using the first electronic communication device and the second party using the second electronic communication device, cause a display device of the first electronic communication device to display initial prompts to the interviewer with initial lines of inquiry about high priority entities and relationships in an information model;

while the conversation is ongoing between the first party using the first electronic communication device and the second party using the second electronic communication device, process the digital auditory data to identify mention of high priority entities and relationships spoken by the first party and the second party during the interview, the processing of the digital auditory data including;

identifying an utterance of a first entity in the plurality of words, determining subsequent lines of inquiry with respect to the identified first entity, and causing the display device to display subsequent prompts to the interviewer to facilitate disambiguation of identified high priority entities,

the subsequent lines of inquiry based on context and relevance of the identified first entity with respect to the plurality of words of the conversation; and

while the conversation is ongoing between the first party using the first electronic communication device and the second party using the second electronic communication device,

cause the display device of the first electronic communication device to display subsequent prompts to the interviewer with the subsequent lines of inquiry with respect to the identified first entity.