Dynamic control of voice codec data rate

• US 9,208,798 B2
• Filed: 04/08/2013
• Issued: 12/08/2015
• Est. Priority Date: 04/09/2012
• Status: Active Grant

• Current Assignee: Board of Regents of the University of Texas System (University of Texas System)
• Original Assignee: Board of Regents of the University of Texas System (University of Texas System)
• Inventors: Ward, Nigel, Baumann, Timo, Karkhedkar, Shreyas A., Novick, David G.
• Primary Examiner(s): Colucci, Michael
• Application Number:
  US13/858,605
• Publication Number:
  US 20140303968A1
• Time in Patent Office: 974 Days
• Field of Search: 704/207, 704/500, 704/270, 704/230, 704/229, 704/223, 704/219, 704/208, 704/205, 846/62, 706/46, 386/353, 381/98, 381/86, 375/240.09, 375/240.02, 348/497, 348/14.12
• US Class Current: 1/1
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 19/22 Mode decision, i.e. based on audio signal content versus external parameters
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals

Abstract

A method, system, and computer-usable non-transitory storage device for dynamic voice codec adaptation are disclosed. The voice codec adapts in real time to devote more bits to audio quality when it is most needed, and fewer bits to less important parts of utterances are disclosed. Dialog knowledge is utilized for compression opportunities to adjust the bitrate moment-by-moment, based on the inferred value of each frame. Frame importance and appropriate transmission fidelity is predicted based on prosodic features and models of dialog dynamics. This technique provides the same communications quality with less spectrum needs, fewer antennas, and less battery drain.

Claims

1. A method for dynamic codec adaptation, comprising:

estimating an importance value of a data frame utilizing information, wherein estimating said importance value of said data frame utilizing said information further comprises measuring sub-metrics, said sub-metrics comprising at least one of objective informational importance, dialog-control importance, interpersonal-relationship-management importance, and meta-dialog importance;

dynamically varying a data rate based on said estimated said importance value of said data frame;

devoting a greater bitrate to data frame quality in real time for a data frame with an assigned high importance value;

devoting less bitrate to data frame quality in real time for a data frame with an assigned low importance value; and

transmitting important voice frames, communications, and utterances with greater accuracy and a certain quality level for a data rate, wherein important voice frames comprise audio with a high pitch range, pitch height, volume, and overlapped speech by multiple speakers.

2. The method of claim 1 wherein said data frame comprises at least one of an audio data frame and a video data frame and wherein said information comprises at least one of a prosodic feature and a model of dialog dynamics, wherein said prosodic feature comprises at least one of pitch height of a speaker or a listener, pitch range of said speaker or said listener, speaking rate of said speaker or said listener, volume of said speaker or said listener, voicing properties of said speaker or said listener, phonetic propertes of said speaker or said listener.

3. The method of claim 1 further comprising:

gathering inferences on a transient mental state of a speaker comprising at least one of said speaker trying to think of a word, said speaker expressing confusion, said speaker expressing sympathy, said speaker expressing frustration, and said speaker expressing determination;

gathering inferences of a current transient mental state of a listener;

and gathering inferences of a current dialog state or an upcoming dialog state, wherein said current dialog state and said upcoming dialog state are modeled based on taxonomy of dialog states comprising at least one of floor-holding, giving short feedback, floor-yielding, asking a question, waiting for an answer;

or

a dimensional value comprising at least one of degree of engagement, degree of certainty, degree of positive feeling, and degree of local dominance.

4. The method of claim 1 wherein estimating said importance value of said data frame utilizing information further comprises:

analyzing an inferred current utterance type of a speaker, an inferred past utterance type of said speaker, or future-expected utterance type of said speaker, said utterance types comprising a question, a statement, and an acknowledgement;

analyzing a timing relative to said current frame of said utterance types, wherein said timing comprises a start time and an end time;

gathering utterance types of an interlocutor;

analyzing behavior of a speaker or a listener including eye gaze, facial gestures, and hand gestures; and

analyzing an output of a speech recognizer.

5. The method of claim 1 further comprising normalizing a local feature relative to an average behavior of an interlocutor with average behavior computed over at least one of a longer time frame, an entire dialog, or a previous dialog or set of dialogs.

6. The method of claim 1 further comprising indicating which aspects need to be most accurately transmitted, for a speech frame said aspects comprising at least one of a high-frequency spectral region and pitch, and for a video frame said aspects comprising at least one of a region, an object, a movement, a chroma value, a luma value, a spatia-temporal feature, and a spatial frequency range.

7. A system for dynamic codec adaptation, comprising:

a processor;

a data bus coupled to said processor; and

a computer-usable non-transitory storage medium embodying computer code, said computer-usable non-transitory storage medium being coupled to said data bus, said computer program code comprising program instructions executable by said processor, said program instructions comprising;

program instructions to estimate an importance value of a data frame utilizing information, wherein said program instructions to estimate said importance value of said data frame further comprises program instructions to measure sub-metrics, said sub-metrics comprising at least one of objective informational importance, dialog-control importance, interpersonal-relationship-management importance, and meta-dialog importance;

program instructions to dynamically vary a data rate based on said estimated said importance value of said data frame;

program instructions to devote a greater bitrate to data frame quality in real time for a data frame with an assigned high importance value;

program instructions to devote less bitrate to data frame quality in real time for a data frame with an assigned low importance value; and

program instruction to transmit important voice frames, communications, and utterances with greater accuracy and a certain quality level for a data rate, wherein important voice frames comprise audio with a high pitch range, pitch height, volume, and overlapped speech by multiple speakers.

8. The system of claim 7 wherein said data frame comprises at least one of an audio data frame and a video data frame and wherein said information comprises at least one of a prosodic feature and a model of dialog dynamics, wherein said prosodic feature comprises at least one of pitch height of a speaker or a listener, pitch range of said speaker or said listener, speaking rate of said speaker or said listener, volume of said speaker or said listener, voicing properties of said speaker or said listener, phonetic properties of said speaker or said listener.

9. The system of claim 7 further comprising:

program instructions to gather inferences on a transient mental state of a speaker comprising at least one of said speaker trying to think of a word, said speaker expressing confusion, said speaker expressing sympathy, said speaker expressing frustration, and said speaker expressing determination;

program instructions to gather inferences of a current transient mental state of a listener; and

program instructions to gather inferences of a current dialog state or an upcoming dialog state wherein said current dialog state and said upcoming dialog state are modeled based on taxonomy of dialog states comprising at least one of floor-holding, giving short feedback, floor-yielding, asking a question, waiting for an answer;

or

a dimensional value comprising at least one of degree of engagement, degree of certainty, degree of positive feeling, and degree of local dominance.

10. The system of claim 7 wherein estimating said importance value of said data frame utilizing information further comprises:

program instructions to analyze an inferred current utterance type of a speaker, an inferred past utterance type of said speaker, or future-expected utterance type of said speaker, said utterance types comprising a question, a statement, and an acknowledgement;

program instructions to analyze a timing relative to said current frame of said utterance types, wherein said timing comprises a start time and an end time;

program instructions to gather utterance types of an interlocutor;

program instructions to analyze behavior of a speaker or a listener including eye gaze, facial gestures, and hand gestures; and

program instructions to analyze an output of a speech recognizer.

11. The system of claim 10 further comprising program instructions to normalize a local feature relative to an average behavior of an interlocutor with average behavior computed over at least one of a longer time frame, an entire dialog, or a previous dialog or set of dialogs.

12. The system of claim 10 further comprising program instructions to indicate which aspects need to be most accurately transmitted, for a speech frame said aspects comprising at least one of a high-frequency spectral region and pitch, and for a video frame said aspects comprising at least one of a region, an object, a movement, a chroma value, a luma value, a spatio-temporal feature, and a spatial frequency range.

13. A computer-usable non-transitory storage device storing computer program code for dynamic codec adaptation, said computer program code comprising program instructions executable by a processor, said program instructions comprising:

program instructions to estimate an importance value of a data frame utilizing information, wherein said program instructions to estimate said importance value of said data frame utilizing said information comprises program instructions to measure sub-metrics, said sub-metrics comprising at least one of objective informational importance, dialog-control importance, interpersonal-relationship-management importance, and meta-dialog importance;

program instructions to dynamically vary a data rate based on said estimated said importance value of said data frame;

program instructions to devote a greater bitrate to data frame quality in real time for a data frame with an assigned high importance value; and

program instructions to devote less bitrate to data frame quality in real time for a data frame with an assigned low importance value; and

program instructions to transmit important voice frames, communications, and utterances with greater accuracy and a certain quality level for a data rate, wherein important voice frames comprise audio with a high pitch range, pitch height, volume, and overlapped speech by multiple speakers.

14. The computer-usable non-transitory storage device of claim 13 wherein said data frame comprises at least one of an audio data frame and a video data frame and wherein said information comprises at least one of a prosodic feature and a model of dialog dynamics, wherein said prosodic feature comprises at least one of pitch height of a speaker or a listener, pitch range of said speaker or said listener, speaking rate of said speaker or said listener, volume of said speaker or said listener, voicing properties of said speaker or said listener, phonetic properties of said speaker or said listener.

15. The computer-usable non-transitory storage device of claim 13 further comprising:

program instructions to gather inferences on a transient mental state of a speaker comprising at least one of said speaker trying to think of a word, said speaker expressing confusion, said speaker expressing sympathy, said speaker expressing frustration, and said speaker expressing determination;

program instructions to gather inferences of a current transient mental state of a listener; and

program instructions to gather inferences of a current dialog state or an

upcoming dialog state wherein said current dialog state and said upcoming dialog state are modeled based on taxonomy of dialog states comprising at least one of floor-holding, giving short feedback, floor-yielding, asking a question, waiting for an answer;

or

a dimensional value comprising at least one of degree of engagement, degree of certainty, degree of positive feeling, and degree of local dominance.

16. The computer-usable non-transitory storage device of claim 13 wherein estimating said importance value of said data frame utilizing information further comprises:

program instructions to normalize a local feature relative to an average behavior of an interlocutor with average behavior computed over at least one of a longer time frame, an entire dialog, or a previous dialog or set of dialogs;

program instructions to analyze an inferred current utterance type of a speaker, an inferred past utterance type of said speaker, or future-expected utterance type of said speaker, said utterance types comprising a question, a statement, and an acknowledgement;

program instructions to analyze a timing relative to said current frame of said utterance types, wherein said timing comprises a start time and an end time;

program instructions to gather utterance types of an interlocutor;

program instructions to analyze behavior of a speaker or a listener including eye gaze, facial gestures, and hand gestures; and

program instructions to analyze an output of a speech recognizes.

17. The computer-usable non-transitory storage device of claim 16 further comprising program instructions to indicate which aspects need to be most accurately transmitted, for a speech frame said aspects comprising at least one of a high-frequency spectral region and pitch, and for a video frame said aspects comprising at least one of a region, an object, a movement, a chroma value, a luma value, a spatia-temporal feature, and a spatial frequency range.