Method and system of speech evaluation

• US 8,620,665 B2
• Filed: 10/31/2011
• Issued: 12/31/2013
• Est. Priority Date: 11/09/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Hasdell, Charlie Mustafa-Ali, Jopling, Steven Gregory, Morris, Andrew Cameron
• Primary Examiner(s): Godbold, Douglas
• Application Number:
  US13/285,412
• Publication Number:
  US 20120116767A1
• Time in Patent Office: 792 Days
• Field of Search: 704/1, 704/231, 704/235, 704/236, 704/251, 704270-275, 708/162, 434/276, 434/269, 434/285
• US Class Current: 704/270
• CPC Class Codes: G10H 1/361 Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems
• CPC Class Codes: G10H 2210/091 for performance evaluation, i.e. judging, grading or scoring the musical qualities or faithfulness of a performance, e.g. with respect to pitch, tempo or other timings of a reference performance
• CPC Class Codes: G10L 25/00 Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 muting semiconductor-based amplifiers when some special characteristics of a signal are sensed by a speech detector, e.g. sensing when no signal is present, H03G3/34

Abstract

A method is provided for user speech performance evaluation with respect to a reference performance for which a phoneme mark-up is available. The method includes capturing input speech from the user and formatting it as frames. For a respective frame of the input speech, the method generates probability values for a plurality of phonemes, generates a probability value for a phoneme class based upon the generated probability values for a plurality of phonemes belonging to that phoneme class. For a plurality of frames of the input speech, the method further includes averaging the phoneme class probability values corresponding to the plurality of frames of the input speech. The method also includes calculating a user speech performance score based upon the average.

Claims

1. A method of user speech performance evaluation with respect to a reference performance for which a phoneme mark-up is available, the method comprising the steps of:

capturing input speech from a user and formatting it as frames; and

for a respective frame of the input speech,

generating probability values for a plurality of phonemes;

generating a probability value for a phoneme class based upon the generated probability values for a plurality of phonemes belonging to that phoneme class; and

for a plurality of frames of the input speech,

averaging the phoneme class probability values corresponding to the plurality of frames of the input speech; and

calculating a user speech performance score based upon the average.

2. A method according to claim 1, comprising the step of:

time shifting an alignment of the input speech frames responsive to the phoneme mark-up of the reference performance.

3. A method according to claim 2, in which the phoneme class for which probability values are generated is the phoneme class comprising the phoneme mark-up to which the respective frame of input speech has been aligned.

4. A method according to claim 2 in which the step of time shifting the alignment of the input speech frames responsive to the phoneme mark-up of the reference performance uses a dynamic programming method.

5. A method according to claim 2, in which a maximum permissible time shift of input speech frames is preset.

6. A method according to claim 1, in which the probability values for phonemes are phoneme posterior probabilities.

7. A method according to claim 1, in which the step of averaging the phoneme class probability values is conducted over the longer of a pre-selected value in the range of 200 ms to 600 ms and the duration of the most recently spoken word by the user.

8. A method according to claim 1, in which the step of calculating a user speech performance score comprises calculating a cumulative distribution function for a distribution having a known standard deviation and a mean modified by a difficulty level.

9. A method according to claim 1, comprising the step of normalising a user speech performance score for a word according to a normalisation factor specific to a current reference performance and adding the normalised score to a performance score.

10. A method according to claim 1, comprising the steps of

generating for display a line of one or more graphic elements each corresponding to a word in a current portion of the reference performance, with each graphic element comprising one or more empty spaces;

and

upon calculation of a user speech performance score for one of the words,

modifying the corresponding graphic element to wholly or partially fill one or more of the empty spaces as a function of both the user speech performance score and the number of empty spaces in the graphic element.

11. A method according to claim 10, in which the number of empty spaces ascribed to each graphic element is determined by rhythm data other than phoneme or syllable data associated with the reference performance.

12. A tangible, non-transitory computer program product comprising a storage medium on which is stored computer readable program code, the program code, when executed by a processor, causes the processor to implement a method of user speech performance evaluation with respect to a reference performance for which a phoneme mark-up is available, the method comprising the steps of:

capturing input speech from a user and formatting it as frames; and

for a respective frame of the input speech,

generating probability values for a plurality of phonemes;

generating a probability value for a phoneme class based upon the generated probability values for a plurality of phonemes belonging to that phoneme class; and

for a plurality of frames of the input speech,

averaging the phoneme class probability values corresponding to the plurality of frames of the input speech; and

calculating a user speech performance score based upon the average.

13. An entertainment device for evaluating a user speech performance with respect to a reference performance for which a phoneme mark-up is available, the entertainment device comprising;

an audio input operable to capture input speech from a user and format it as frames;

a phoneme probability generator operable to generate probability values for a plurality of phonemes;

a phoneme class probability generator operable to generate a probability value for a phoneme class based upon the generated probability values for a plurality of phonemes belonging to that phoneme class;

an averaging logic operable to average phoneme class probability values corresponding to a plurality of frames of the input speech; and

a calculating logic operable to calculate a user speech performance score based upon the average.

14. The apparatus of claim 13, comprising:

an input speech time shifter operable to time shift the alignment of the input speech frames responsive to the phoneme mark-up of the reference performance.

15. The apparatus of claim 14, in which in which the phoneme class for which probability values are generated is the phoneme class comprising the phoneme mark-up to which the respective frame of input speech has been aligned.