Speech processing method and apparatus for deciding emphasized portions of speech, and program therefor

• US 8,793,124 B2
• Filed: 04/05/2006
• Issued: 07/29/2014
• Est. Priority Date: 08/08/2001
• Status: Active Grant

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Kojima, Haruhiko, Hidaka, Kota, Nakajima, Shinya, Mizuno, Osamu, Kuwano, Hidetaka
• Primary Examiner(s): YEN, ERIC L
• Application Number:
  US11/397,803
• Publication Number:
  US 20060184366A1
• Time in Patent Office: 3,037 Days
• Field of Search: 704208-210, 704/231, 704/256.1
• US Class Current: 704/208
• CPC Class Codes: G10L 25/00 Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 muting semiconductor-based amplifiers when some special characteristics of a signal are sensed by a speech detector, e.g. sensing when no signal is present, H03G3/34

Abstract

A scheme to judge emphasized speech portions, wherein the judgment is executed by a statistical processing in terms of a set of speech parameters including a fundamental frequency, power and a temporal variation of a dynamic measure and/or their derivatives. The emphasized speech portions are used for clues to summarize an audio content or a video content with a speech.

Claims

1. A speech processing method performed using a processor for deciding whether a portion of input speech is emphasized or not based on a set of speech parameters for each frame, comprising the steps of:

(a) obtaining from a codebook a plurality of speech parameter vectors each corresponding to a respective set of speech parameters obtained from respective ones of a plurality of frames in the portion of the input speech, said codebook storing, for each of a plural number of predetermined speech parameter vectors, a corresponding pair of a normal-state appearance probability and an emphasized-state appearance probability both predetermined using a training speech signal, each of said plural number of predetermined speech parameter vectors being composed of a set of speech parameters including at least one of a fundamental frequency, power and a temporal variation of dynamic-measure and/or an inter-frame difference in at least one of those speech parameters, and obtaining from said codebook a pair of an emphasized-state appearance probability and a normal-state appearance probability both corresponding to each speech parameter vector obtained for the respective ones of the plurality of frames in the portion of the input speech;

(b) using the processor, calculating an emphasized-state likelihood of the portion of the input speech by multiplying together emphasized-state appearance probabilities corresponding to the respective speech parameter vectors for the plurality of frames in the portion of the input speech, and calculating a normal-state likelihood of the portion of the input speech by multiplying together normal-state appearance probabilities corresponding to the respective speech parameter vectors for the plurality of frames in the portion of the input speech; and

(c) deciding whether the portion of the input speech is emphasized or not based on said calculated emphasized-state likelihood and said calculated normal-state likelihood, and outputting a decision result of said deciding, the decision result indicating whether the portion of the input speech is emphasized or not,

wherein the codebook stores, for each of the plural predetermined speech parameter vectors, a respective independent emphasized-state appearance probability and a respective set of conditional emphasized-state appearance probabilities, both used as respective said emphasized-state appearance probability, and stores, for each of the plural predetermined speech parameter vectors, a respective independent normal-state appearance probability and a set of conditional normal-state appearance probabilities, both used as respective said normal-state appearance probability, such that there is at least stored a separate conditional emphasized-state appearance probability and a separate conditional normal-state appearance probability for a possible speech parameter vector that immediately follows the respective speech parameter vector in the codebook, and

wherein the step of calculating the emphasized-state likelihood in said step (b) is implemented by multiplying together the independent emphasized-state appearance probability and the conditional emphasized-state appearance probabilities corresponding to the speech parameter vectors of respective first frame and subsequent frames in said portion of the input speech, and the step of calculating the normal-state likelihood in said step (b) is implemented by multiplying together the independent normal-state appearance probability and the conditional normal-state appearance probabilities corresponding to the speech parameter vectors of respective said first frame and said subsequent frames in said portion of the input speech.

2. The method of claim 1, wherein said codebook stores, for the plural number of predetermined speech parameter vectors, respective codes representing the respective predetermined speech parameter vectors, and said step (a) further includes a step of quantizing each set of speech parameters obtained from respective one of the plurality of the frames in the portion of the input speech by using said codebook to obtain the code.

3. The method of claim 2, wherein a set of speech parameters of each of said plural number of predetermined speech parameter vectors includes at least temporal variation of dynamic measure.

4. The method of claim 2, wherein a set of speech parameters of each of said plural number of predetermined speech parameter vectors includes at least a fundamental frequency, power and temporal variation of dynamic measure.

5. The method of claim 2, wherein a set of speech parameters of each of said plural number of predetermined speech parameter vectors includes at least a fundamental frequency, power and temporal variation of dynamic-measure or an inter-frame difference in each of the parameters.

6. The method of claim 2, wherein said deciding step (c) is based on said calculated emphasized-state likelihood being larger than said calculated normal likelihood.

7. The method of claim 2, wherein said step (c) is performed based on a ratio of said calculated emphasized-state likelihood to said calculated normal-state likelihood.

8. The method of any one of claims 3 to 5 and 2, wherein said step (a) is based on normalizing each of said speech parameters in each set obtained from respective ones of the plurality of frames in said portion of the input speech by an average of corresponding speech parameters over said plurality of frames in said portion of the input speech to produce normalized speech parameters, a set of said normalized speech parameters obtained for each frame being used as said set of speech parameters for each said frame.

9. The method of claim 2, wherein said step (b) includes a step of calculating a conditional probability of emphasized-state by linear interpolation of said independent emphasized-state appearance probability and said conditional emphasized-state appearance probabilities.

10. The method of claim 2, wherein said step (b) includes a step of calculating a conditional probability of normal state by linear interpolation of said independent normal-state appearance probability and said conditional normal-state appearance probabilities.

11. The method of claim 1,

wherein said step (a) includes a step of deciding, as a speech block, a series of speech sub-blocks in which an average power of a voiced portion in the last sub-block in said series is smaller than a product of an average power of said last sub-block and a constant, and

wherein said step (c) includes a step of comparing said calculated emphasized-state likelihood with said normal-state likelihood to decide, as a portion of summarized speech, a speech block including a speech sub-block which is decided to be an emphasized sub-block, and outputting the portion of summarized speech.

12. The method of claim 1,

wherein said step (a) includes a step of deciding, as a speech block, a series of speech sub-blocks in which an average power of a voiced portion in the last sub-block is smaller than a product of an average power of said last sub-block and a constant, and

wherein said step (c) includes:

(c-1) a step of calculating a likelihood ratio of said calculated emphasized state likelihood to said normal state likelihood;

(c-2) a step of deciding a speech sub-block of the series of sub-blocks to be in an emphasized state if said likelihood ratio is greater than a threshold value; and

(c-3) a step of deciding a speech block including the emphasized speech sub-block as a portion of summarized speech, and outputting the portion of summarized speech.

13. The method of claim 12, wherein said step (c) further includes a step of varying the threshold value, and repeating the steps (c-2) and (c-3) to obtain portions of summarized speech with a desired summarization ratio.

14. The method of claim 1, wherein said step (a) includes the steps of:

(a-1) judging each frame as voiced or unvoiced;

(a-2) judging, as a speech sub-block, every portion which includes a voiced portion of at least one frame and which is laid between unvoiced portions longer than a predetermined number of frames; and

(a-3) judging, as a speech block, a series of at least one speech sub-block including a final sub-block, in which an average power of a voiced portion in said final sub-block is smaller than an average power of said final sub-block multiplied by a constant,

wherein said step (c) includes a step of judging every speech sub-block as said portion of the input speech, judging a speech block including an emphasized speech sub-block as a portion of summarized speech, and outputting the portion of summarized speech.

15. The method of claim 14, wherein;

said step (b) includes a step of calculating each normal-state likelihood for respective speech sub-block based on said normal-state appearance probabilities; and

said step (c) includes the steps of;

(c-1) judging, as a provisional portion, each speech block including a speech sub-block, for which a likelihood ratio of said emphasized-state likelihood to said normal-state likelihood is larger than a threshold;

(c-2) calculating a total duration of provisional portions or a ratio of a total duration of whole portions to said total duration of provisional portions as a summarization ratio; and

(c-3) adjusting a threshold to adjust a number of provisional portions so that a total duration of the provisional portions is equal or approximate to a predetermined summarization time, or said summarization ratio is equal or approximate to a predetermined summarization ratio.

16. The method of claim 15 wherein said step (c-3) includes:

(c-3-1) increasing said threshold to decrease the number of provisional portions, when said total duration of the provisional portions is longer than said predetermined summarization time, or said summarization ratio is smaller than said predetermined summarization ratio, and repeating said steps (c-1) and (c-2);

(c-3-2) decreasing said threshold to increase the number of provisional portions, when said total duration of the provisional portions is shorter than said predetermined summarization time or said summarization ratio is larger than said predetermined summarization ratio and repeating said steps (c-1) and (c-2).

17. The method of claim 14,

wherein said step (b) includes a step of calculating each normal-state likelihood for respective speech sub-blocks based on said normal-state appearance probabilities;

and

wherein said step (c) includes the steps of;

(c-1) calculating a likelihood ratio of said emphasized-state likelihood to said normal-state likelihood for each said speech sub-block;

(c-2) calculating a total duration by accumulating durations of each said speech block including a speech sub-block in a decreasing order of said likelihood ratio; and

(c-3) deciding said speech blocks as portions to be summarized, at which a total duration of provisional portions is equal or approximate to a predetermined summarization time, or a summarization ratio is equal or approximate to a predetermined summarization ratio.

18. A non-transitory computer-readable storage medium having program code recorded thereon that, when executed by the processor, execute the method of any one of claim 3-5, 6-7, 10 or 2.

19. A speech processing method performed using a processor for deciding whether a portion of input speech is emphasized or not based on a set of speech parameters for each frame using an acoustical model including a codebook,

wherein said codebook stores, as a normal initial-state appearance probability and an emphasized initial-state appearance probability, both for each of a plural number of predetermined speech parameter vectors, a corresponding pair of normal-state appearance probability and an emphasized-state appearance probability, both predetermined using a training speech signal, a predetermined number of states including an initial state and a final state, state transitions each defining a transition from each state to itself or another state, an output probability table storing emphasized-state output probabilities and normal-state output probabilities both for each of the plural number of speech parameter vectors at the respective states and a transition probability table storing an emphasized-state transition probability and a normal-state transition probability both for each of the state transitions, and

wherein each of said speech parameter vectors is composed of a set of speech parameters including at least one of a fundamental frequency, power and a temporal variation of dynamic-measure and/or an inter-frame difference in at least one of those parameters,

the method comprising the steps of:

judging each frame as voiced or unvoiced;

judging, as a speech sub-block, a portion which includes a voiced portion of at least one frame and which is laid between unvoiced portions longer than a predetermined number of frames;

obtaining from the codebook an emphasized initial-state probability and a normal initial-state probability both corresponding to a speech parameter vector which is a quantized set of speech parameters for an initial frame in said speech sub-block;

obtaining from the output probability table emphasized-state output probabilities and normal-state output probabilities both for respective state transitions corresponding to respective speech parameter vectors each of which is a quantized set of speech parameters obtained for respective one of frames after said initial frame in said speech sub-block, and obtaining from the transition probability table emphasized-state transition probabilities and normal-state transition probabilities both corresponding to state transitions for respective frames after said initial frame in said speech sub-block;

calculating, using the processor, a probability of emphasized-state by multiplying together said emphasized initial-state probability, said emphasized-state output probabilities and said emphasized-state transition probabilities both along every path of state transitions via the predetermined number of states and calculating, using the processor, a probability of normal-state by multiplying together said normal initial-state probability, said output probability and said normal-state transition probability both alone every state transition path;

deciding a largest one or total sum of the probabilities of emphasized-state for all the state transition paths as an emphasized-state likelihood and a largest one or total sum of the probabilities of normal-state for all the state transition paths as a normal-state likelihood; and

comparing said emphasized-state likelihood with said normal-state likelihood to decide whether the speech sub-block is emphasized state or normal state.

20. A speech processing apparatus for deciding whether a portion of input speech is emphasized or not based on a set of speech parameters for each frame of said input speech, said apparatus comprising:

a codebook which stores, for each of a plural number of predetermined speech parameter vectors, a corresponding pair of a normal state appearance probability and an emphasized-state appearance probability, both predetermined using a training speech signal, each of said predetermined speech parameter vectors being composed of a set of speech parameters including at least two of a fundamental frequency, power and temporal variation of dynamic measure and/or an inter-frame difference in at least one of those speech parameters;

means for obtaining from said codebook a plurality of speech parameter vectors each corresponding to a respective set of speech parameters for obtained from each of a plurality of frames in the portion of the input speech;

a normal state likelihood calculating part that calculates a normal-state likelihood of the portion of the input speech by multiplying together normal-state appearance probabilities corresponding to the respective speech parameter vectors for the plurality of frames in the portion of the input speech;

an emphasized-state likelihood calculating part that calculates an emphasized-state likelihood of the portion of the input speech by multiplying together emphasized-state appearance probabilities corresponding to the respective speech parameter vectors for the plurality of frames in the portion of the input speech;

an emphasized state deciding part that decides whether the portion of the input speech is emphasized or not based on a comparison of said calculated emphasized-state likelihood to said calculated normal-state likelihood; and

outputting unit that outputs the decision result representing whether the portion of the input speech is emphasized or not,

wherein the codebook further stores, for each of the plural predetermined speech parameter vectors, a respective independent emphasized-state appearance probability and a respective independent normal-state appearance probability, both predetermined using the training speech signal, and stores for each of the plural predetermined speech parameter vectors, a respective set of conditional emphasized-state appearance probabilities and a respective set of conditional normal-state appearance probabilities, both predetermined using the training speech signal, such that there is at least stored a separate conditional emphasized-state appearance probability and a separate conditional normal-state appearance probability for a possible instance speech parameter vector that immediately follows the respective speech parameter vector in the codebook,

wherein said emphasized-state likelihood calculating part is configured to calculate the emphasized-state likelihood by multiplying together an independent emphasized-state appearance probability and conditional emphasized-state appearance probabilities corresponding to the speech parameter vectors of respective first frame and subsequent frames in the portion of the input speech, and

wherein said normal-state likelihood calculating part is configured to calculate the normal-state likelihood by multiplying together an independent normal-state appearance probability and conditional normal-state appearance probabilities corresponding to the speech parameter vectors of respective first frame and subsequent frames in the portion of the input speech.

21. The apparatus of claim 20, wherein said codebook stores, for the plural predetermined speech parameter vectors, respective codes representing the respective speech parameter vectors, and said means for obtaining a speech parameter vector is configured to quantize each set of speech parameters obtained from respective one of the plurality of the frames in the portion of the input speech by using said codebook to obtain the code.

22. The apparatus of claim 21, wherein a set of speech parameters of each of said plural predetermined speech parameter vectors includes at least a temporal variation of dynamic measure.

23. The apparatus of claim 21, wherein a set of speech parameters of each of said plural predetermined speech parameter vectors includes at least a fundamental frequency, a power and a temporal variation of dynamic measure.

24. The apparatus of claim 21, wherein a set of speech parameters of each of said plural predetermined speech parameter vectors includes at least a fundamental frequency, power and a temporal variation of a dynamic-measure or an inter-frame difference in each of the parameters.

25. The apparatus of any one of claims 22 to 24 and 21, wherein said emphasized-state deciding part includes emphasized state deciding means for deciding, said for the portion of the input speech, whether a ratio of said emphasized-state likelihood to said normal state likelihood is higher than a predetermined value, and if so, deciding that the portion of the input speech is emphasized.

26. The apparatus of claim 21, further comprising:

an unvoiced portion deciding part that decides whether each frame of said input speech is an unvoiced portion;

a voiced portion deciding part that decides whether each frame of said input speech is a voiced portion;

a speech sub-block deciding part that decides that every portion preceded and succeeded by more than a predetermined number of unvoiced portions and including a voiced portion is a speech sub-block;

a speech block deciding part that decides that when an average power of said voiced portion included in the last speech sub-block in said sequence of speech sub-blocks is smaller than a product of the average power of said speech sub-block and a constant, the sequence of the speech sub-blocks is a speech block; and

a summarized portion output part that decides that a speech block including a speech sub-block which is decided as emphasized by said emphasized state deciding part is a portion of summarized speech, and that outputs said speech block as the portion of summarized speech.

27. The apparatus of claim 26, wherein

said normal-state likelihood calculating part is configured to calculate the normal-state likelihood of each said speech sub-block;

and

said emphasized state deciding part includes;

a provisionally summarized portion deciding part that decides that a speech block including a speech sub-block is a provisionally summarized portion if a likelihood ratio between the emphasized-state likelihood of said portion decided by said speech sub-block deciding part as said speech sub-block to its normal-state likelihood is higher than a reference value; and

a summarized portion deciding part that calculates the total amount of time of said provisionally summarized portions, or as the summarization rate, a ratio of the overall time of the entire portion of said input speech to said total amount of time of said provisionally summarized portions, that calculates said reference value on the basis of which the total amount of time of said provisionally summarized portions becomes substantially equal to a predetermined value or said summarization rate becomes substantially equal to a predetermined value, and that determines said provisionally summarized portions as portions of summarized speech.

28. The apparatus of claim 26, wherein

said normal-state likelihood calculating part is configured to calculate a normal-state likelihood of said each said speech sub-block;

and

said emphasized state deciding part includes;

a provisionally summarized portion deciding part that calculates a likelihood ratio of said emphasized-state likelihood of each speech sub-block to its normal-state likelihood, and that provisionally decides that each speech block including speech sub-blocks having likelihood ratios down to a predetermined likelihood ratio in descending order is a provisionally summarized portion; and

a summarized portion deciding part that calculates the total amount of time of provisionally summarized portions, or as the summarization rate, a ratio of said total amount of time of said provisionally summarized portions to the overall time of the entire portion of said input speech, that calculates said predetermined likelihood ratio on the basis of which the total amount of time of said provisionally summarized portions becomes substantially equal to a predetermined value or said summarization rate becomes substantially equal to a predetermined value, and that determines said provisionally summarized portions as portions of summarized speech.