Method of determining whether a phone call is answered by a human or by an automated device

• US 10,014,006 B1
• Filed: 05/08/2017
• Issued: 07/03/2018
• Est. Priority Date: 09/10/2013
• Status: Active Grant

• Current Assignee: Fonative, Inc. (Ontario Systems, LLC)
• Original Assignee: Ampersand Inc.
• Inventors: Smith, Stephen, Howitt, George, Campbell, Douglas
• Primary Examiner(s): Hashem, Lisa
• Application Number:
  US15/589,622
• Time in Patent Office: 421 Days
• CPC Class Codes: G10L 17/02 Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
• CPC Class Codes: G10L 17/08 Use of distortion metrics or a particular distance between probe pattern and reference templates
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04M 2203/2027 Live party detection
• CPC Class Codes: H04M 2203/306 Prerecordings to be used during a voice call
• CPC Class Codes: H04M 3/493 Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
• CPC Class Codes: H04M 3/53383 Message registering commands or announcements; Greetings

Abstract

A system receives an audio stream from an application, and performs signal analysis on the audio stream to compute (on sound and silence) a sequence of events. The sequence of events identifies a pattern comprised of at least two of sound, silence, frequency, magnitude, duration, tone, and Dual Tone Multi Frequency (DTMF). The pattern matches a similar pattern of at least two of sound, silence, frequency, magnitude, duration, tone, and DTMF, and the signal analysis is performed on sound and silence within the audio stream. The similar pattern is associated with at least one of an automated device or human answering the call. The system determines whether the sequence of events indicates the phone call is answered by the automated device or the human, based on rules in a heuristic engine and/or a known audio print from a library of known audio prints.

Claims

1. A method of determining whether a phone call is answered by an automated device or a human, the method comprising:

(a) receiving, at a processor, an audio stream from an application;

(b) performing, at the processor, signal analysis on the audio stream to compute a sequence of events in the audio stream, wherein the audio stream is comprised of sound and, silence, wherein the sequence of events is computed on the sound and silence, wherein the sequence of events identifies a pattern associated with the audio stream, wherein the pattern is comprised of at least two of sound, silence, frequency, magnitude, duration, tone, and Dual Tone Multi Frequency (DTMF);

(c) determining, at the processor, whether the phone call is answered by the automated device by using a library of known audio prints used by known automated devices and by using a heuristic engine comprising a plurality of rules,

wherein in using the library of known audio prints used by the known automated devices, the determining comprises;

(c1) determining, at the processor, that the pattern matches a similar pattern of at least two of sound, silence, frequency, magnitude, duration, tone, and DTMF in a known audio print in the library of known audio prints;

(c2) applying, at the processor, a silence constraint to match periods of silence in the sequence of events in the audio stream to periods of silence in the known audio print,

wherein the silence constraint is associated with the known audio print; and

(c3) based on the matching of the pattern to the similar pattern and on the application of the silence constraint, determining, at the processor, whether there is a first potential match between the sequence of events in the audio stream and the known audio print;

wherein in using the heuristic engine, the determining comprises;

(c4) providing, at the processor, the sequence of events as input to the heuristic engine;

(c5) applying, at the processor, the plurality of rules in the heuristic engine to the sequence of events; and

(c6) based on the application of the plurality of rules, determining, at the processor, whether there is a second potential match between the sequence of events and the plurality of rules in the heuristic engine;

(d) in response to there being either the first potential match or the second potential match, determining, at the processor, that the phone call is answered by the automated device; and

(e) in response to there being neither the first potential match nor the second potential match, determining, at the processor, that the phone call is answered by human.

2. The method of claim 1 wherein performing, at the processor, signal analysis on the audio stream to compute the sequence of events in the audio stream comprises:

receiving the audio stream real time as the phone call is answered;

partitioning the audio stream into a plurality of portions as the audio stream is received; and

performing the signal analysis on each of the plurality of portions.

3. The method of claim 2 wherein performing the signal analysis on each of the plurality of portions comprises:

identifying a most recent portion from the plurality of portions;

identifying at least one less recent portion from the plurality of portions; and

performing the signal analysis on the most recent portion and data associated with the at least one less recent portion to determine at least one of a sound event, a silence event, a tone event, a DTMF event, an event for the automated device answering the phone call, and an event for the human answering the phone call, wherein the data associated with the at least one less recent portion comprises signal analysis data and events associated with the at least one less recent portion.

4. The method of claim 3 wherein performing the signal analysis on each of the plurality of portions and data associated with the at least one less recent portion comprises:

calculating, from the most recent portion and data associated with the at least one less recent portion, values comprising at least one of duration of an entire stream, average magnitude of the portion, a maximum magnitude of the portion, a noise threshold, frequencies seen in the portion, a duration of persistent frequencies, a frequency of current dominant tone, a duration of current dominant tone, an overall duration of all silence portions, and a duration of most recent silence portion; and

generating a real time sequence indicating at each sequence point whether at least one of the sound event, the silence event, the tone event, the DTMF event, the event for the automated device answering the phone call, and the event for the human answering the phone call occurs.

5. The method of claim 4 wherein generating the real time sequence comprises:

examining the values with the heuristic engine to detect a pattern indicating at least one of the sound event, the silence event, the tone event, the DTMF event, the event for the automated device answering the phone call, and the event for the human answering the phone call.

6. The method of claim 5 wherein examining the values with the heuristic engine comprises:

utilizing at least one of a threshold, constraints, and parameters to examine the values calculated, wherein the threshold, constraints, and parameters are comprised of at least one of a tone magnitude threshold, a tone noise factor, a tone minimum frequency, a tone maximum frequency, a silence threshold, portion non-silence percent, a maximum initial silence, a maximum greeting, a tone portion percent, a low frequency start index, a frequency drift, a tone maximum range, and a gap portions permitted in a tone to calculate a weighted likelihood that at least one of the sound event, the silence event, the tone event, the DTMF event, the event for the automated device answering the phone call, and the event for the human answering the phone call has occurred; and

tuning the threshold, constraints and parameters to accurately determine at least one of the sound event, the silence event, the tone event, the DTMF event, the event for the automated device answering the phone call, and the event for the human answering the phone call.

7. The method of claim 6 further comprising:

upon detecting the human answering the phone call, performing at least one of;

i) transferring the phone call to a human agent to respond to the human answering the phone call;

ii) playing a pre-recorded message;

iii) playing an interactive voice response script; and

iv) notifying a calling application that the human has answered the phone call.

8. The method of claim 6 further comprising:

upon detecting the automated device answering the phone call, performing at least one of;

i) terminating the phone call;

ii) playing a pre-recorded message;

iii) playing a sequence of pre-recorded messages to the automated device; and

iv) notifying an application that the automated device has answered the phone call.

9. The method of claim 6 further comprising:

upon determining that the automated device answering the phone call based on the second potential match, identifying the sequence of events as a possible new known audio print to be added to the library of known audio prints.

10. The method of claim 6 wherein tuning the heuristic engine to accurately determine at least one of the sound event, the silence event, the tone event, the DTMF event, the event for the automated device answering the phone call, and the event for the human answering the phone call comprises:

performing the signal analysis on a plurality of call recordings with known data for each of the plurality of call recordings, wherein the known data comprises at least one of a timing and an occurrence of at least one of the sound event, the silence event, the tone event, the DTMF event, the event for the automated device answering the phone call, and the event for the human answering the phone call;

comparing an heuristic engine output to the known data;

adjusting heuristic engine tunable parameters to improve accuracy; and

repeating the performing of the signal analysis on the plurality of call recordings, the comparing of the heuristic engine output, and the adjusting of the heuristic engine tunable parameters to obtain a high accuracy rate on the plurality of call recordings.

11. The method of claim 6 wherein examining the values with the heuristic engine comprises at least one of:

tuning the heuristic engine to return a result within a limited period of time; and

tuning the heuristic engine to return the result regardless of a limitation on time, wherein the heuristic engine determines the automated device answering the phone call or the human answering the phone call.

12. The method of claim 4 further comprising:

upon receiving the silence event, playing at least one of a pre-recorded message and an interactive voice response script; and

upon receiving the sound event during the playing of the pre-recorded message or the interactive voice response script, ceasing at least one of the pre-recorded message and the interactive voice response script wherein the sound event is at least one of the automated device answering the phone call and the human answering the phone call.

13. The method of claim 12 further comprising:

until determining the automated device answering the phone call or the human answering the phone call, performing steps of i) playing the at least one of the pre-recorded message and the interactive voice response script and ii) ceasing the at least one of the pre-recorded message and the interactive voice response script.

14. The method of claim 4 wherein performing the signal analysis on each of the plurality of portions comprises:

determining a frequency associated with the tone detected in a portion; and

weighting a tone, based on the frequency, to determine if the tone comprises at least one of the tone event and the event for the automated device answering the phone call.

15. The method of claim 14 wherein determining the frequency associated with the tone detected in the portion comprises:

detecting the frequency is within a threshold associated with automated devices; and

determining the automated device is answering the phone call based on the frequency.

16. The method of claim 14 wherein weighting the tone comprises:

establishing a probability associated with the frequency, wherein the probability that the frequency is associated with the automated device answering the phone call diminishes as the frequency is identified as a lower frequency than a higher frequency, and wherein the probability that the frequency is associated with the automated device answering the phone call increases as the frequency is identified as the higher frequency.

17. The method of claim 14 wherein determining the frequency associated with the tone detected in the portion comprises:

identifying a strongest frequency in the portion; and

identifying a tone associated with the strongest frequency in the portion as a dominant tone in the portion.

18. The method of claim 17 wherein identifying the strongest frequency in the portion comprises:

tracking a plurality of tones in the portion;

eliminating any tones that drop out in the portion; and

identifying the strongest frequency from remaining tones in the portion.

19. The method of claim 17 wherein identifying the tone associated with the strongest frequency in the portion as the dominant tone in the portion comprising:

tracking a plurality of frequencies in the portion;

identifying the strongest frequency as being a predetermined percentage stronger than a next strongest frequency; and

identifying the tone associated with the strongest frequency in the portion as an only dominant tone in the portion.

20. The method of claim 2 wherein performing the signal analysis on the most recent portion and data associated with the at least one less recent portion comprises:

detecting a tone in the most recent portion; and

determining that an automated device has answered the phone call.

21. The method of claim 2 wherein performing the signal analysis on the most recent portion and data associated with the at least one less recent portion comprises:

detecting a DTMF in the most recent portion; and

determining that the human has answered the phone call.

22. The method of claim 1 wherein performing, at the processor, signal analysis on the audio stream to compute a sequence of events in the audio stream comprises:

receiving the audio stream real time as the phone call is answered; and

when a duration of the audio stream exceeds a duration of the known audio print by a multiple of a tunable interval, computing a stream audio print on the audio stream and determining whether the stream audio print matches the known audio print.

23. A computer program product for determining whether a phone call is answered by an automated device or a human, the computer program product comprising:

a non-transitory computer readable medium having computer readable program code embodied therewith, the program code executable by a computing processor to;

receive an audio stream from an application;

perform signal analysis on the audio stream to compute a sequence of events in the audio stream, wherein the audio stream is comprised of sound and silence, wherein the sequence of events is computed on the sound and silence, wherein the sequence of events identifies a pattern associated with the audio stream, wherein the pattern is comprised of at least two of sound, silence, frequency, magnitude, duration, tone, and Dual Tone Multi Frequency (DTMF);

determine whether the phone call is answered by the automated device by using a library of known audio prints used by known automated devices and by using a heuristic engine comprising a plurality of rules,

wherein in using the library of known audio prints used by the known automated devices, the determine comprises;

determine that the pattern matches a similar pattern of at least two of sound, silence, frequency, magnitude, duration, tone, and DTMF in a known audio print in the library of known audio prints;

apply a silence constraint to match periods of silence in the sequence of events in the audio stream to periods of silence in the known audio print, wherein the silence constraint is associated with the known audio print; and

based on the matching of the pattern to the similar pattern and on the application of the silence constraint, determine whether there is a first potential match between the sequence of events in the audio stream and the known audio print;

wherein in using the heuristic engine, the determine comprises;

provide the sequence of events as input to the heuristic engine;

apply the plurality of rules in the heuristic engine to the sequence of events; and

based on the application of the plurality of rules, determine whether there is a second potential match between the sequence of events and the plurality of rules in the heuristic engine;

in response to there being either the first potential match or the second potential match, determine that the phone call is answered by the automated device; and

in response to there being neither the first potential match nor the second potential match, determine that the phone call is answered by the human.

24. A system comprising:

a computing processor; and

a non-transitory computer readable medium operationally coupled to the processor, the computer readable medium having computer readable program code embodied therewith to be executed by the computing processor, the computer readable program code configured to;

receive an audio stream from an application;

perform signal analysis on the audio stream to compute a sequence of events in the audio stream, wherein the audio stream is comprised of sound and silence, wherein the sequence of events is computed on the sound and silence, wherein the sequence of events identifies a pattern associated with the audio stream, wherein the pattern is comprised of at least two of sound, silence, frequency, magnitude, duration, tone, and Dual Tone Multi Frequency (DTMF);

determine whether the phone call is answered by the automated device by using a library of known audio prints used by known automated devices and by using a heuristic engine comprising a plurality of rules,

wherein in using the library, of known audio prints used by the known automated devices, the determine comprises;

determine that the pattern matches a similar pattern of at least two of sound, silence, frequency, magnitude, duration, tone, and DTMF in a known audio print in the library of known audio prints;

apply a silence constraint to match periods of silence in the sequence of events in the audio stream to periods of silence in the known, audio print, wherein the silence constraint is associated with the known audio print; and

based on the matching of the pattern to the similar pattern and on the application of the silence constraint, determine whether there is a first potential match between the sequence of events in the audio stream and the known audio print;

wherein in using the heuristic engine, the determine comprises;

provide the sequence of events as input to the heuristic engine;

apply the plurality of rules in the heuristic engine to the sequence of events; and

based on the application of the plurality of rules, determine whether there is a second potential match between the sequence of events and the plurality of rules in the heuristic engine;

in response to there being either the first potential match or the second potential match, determine that the phone call is answered by the automated device; and

in response to there being neither the first potential match nor the second potential match, determine that the phone call is answered by a human.