Speech recognition from concurrent visual and audible inputs

• US 20050096902A1
• Filed: 12/09/2004
• Published: 05/05/2005
• Est. Priority Date: 10/09/1998
• Status: Abandoned Application

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kondo, Tetsujiro, Yoshiwara, Norifumi
• Application Number:
  US11/009,337
• Publication Number:
  US 20050096902A1
• Time in Patent Office: Days
• US Class Current: 704/222
• CPC Class Codes: G06F 18/10 Pre-processing; Data cleansing
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06V 10/72 Data preparation, e.g. statistical preprocessing of image or video features
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/065 Adaptation
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

With respect to each of codes corresponding to code vectors in a code book stored in a code book storage section, an expectation degree storage section stores an expectation degree at which observation is expected when an integrated parameter with respect to a word as a recognition target is inputted. A vector quantization section vector-quantizes the integrated parameter and outputs a series of codes of a code vector which has a shortest distance to the integrated parameter.

Claims

1. A learning apparatus for carrying out learning an expectation degree at which a vector quantization result is observed and which is used for vector-quantizing an input series and recognizing whether or not the input series corresponds to a recognition target, based on the vector quantization result, comprising:

vector quantization means for vector-quantizing a time series of learning data pieces comprising image data and noise data and for outputting a series of identifiers each indicating a code vector; and

calculator means for calculating an expectation degree of each of the identifiers, from the series of identifiers obtained from the time series of learning data pieces.

2. The apparatus according to claim 1, wherein with respect to each of the time series of learning data pieces, the vector quantization means outputs an identifier of a code vector which has a shortest distance to the learning data piece.

3. The apparatus according to claim 1, wherein the calculator means obtains an average value of expectation degrees of an identifier with respect to each of a plurality of learning data pieces used for learning for recognition of one same recognition target, as a final expectation degree of the identifier.

4. A learning method for carrying out learning an expectation degree at which a vector quantization result is observed and which is used for vector-quantizing an input series and recognizing whether or not the input series corresponds to a recognition target, based on the vector quantization result, comprising the steps of:

vector-quantizing a time series of learning data pieces comprising image data and noise data and of outputting a series of identifiers each indicating a code vector; and

calculating an expectation degree of each of the identifiers, from the series of identifiers obtained from the time series of learning data pieces.

5. A recording medium which records a program for making a computer execute learning an expectation degree at which a vector quantization result is observed and which is used for vector-quantizing an input series and recognizing whether or not the input series corresponds to a recognition target, based on the vector quantization result, the program comprises:

a vector quantization step of vector-quantizing a time series of learning data pieces comprising image data and noise data and of outputting a series of identifiers each indicating a code vector; and

a calculation step of calculating an expectation degree of each of the identifiers, from the series of identifiers obtained from the time series of learning data pieces.

6. A learning apparatus for obtaining a distance transition model expressing transition of a distance between a standard series and a code vector used for vector quantization, comprising:

normalization means for performing normalization of a time axis with respect to a time series of learning data pieces comprising image data and noise data and for outputting the standard series; and

distance calculation means for calculating a distance between the standard series and the code vector and for outputting transition of the distance.

7. The apparatus according to claim 6, further comprising approximation means for approximating a polynomial to the transition of the distance.

8. The apparatus according to claim 7, wherein the approximation means determines a degree of the polynomial such that an accumulation value of a distance to one of code vectors in a code book that minimizes the distance to the standard series is minimized.

9. A learning method for obtaining a distance transition model expressing transition of a distance between a standard series and a code vector used for vector quantization, comprising the steps of

performing normalization of a time axis with respect to a time series of learning data pieces comprising image data and noise data and of outputting the standard series;

and

calculating a distance between the standard series and the code vector and of outputting transition of the distance.

10. A recording medium which records a program for making a computer execute learning for obtaining a distance transition model expressing transition of a distance between a standard series and a code vector used for vector quantization, characterized by comprising:

a normalization step of performing normalization of a time axis with respect to a time series of learning data pieces comprising image data and noise data and of outputting the standard series; and

a distance calculation step of calculating a distance between the standard series and the code vector and of outputting transition of the distance.

11. A recognition apparatus for recognizing whether or not a time series of input data pieces comprising image data and noise data corresponds to at least one recognition target, comprising:

integration means for integrating a time series of first input data pieces and a time series of second input data pieces, thereby to output a time series of integrated data pieces; and

recognition means for recognizing whether or not the time series of first or second input data pieces corresponds to at least one recognition target, based on transition of a distance obtained from a vector based on the time series of integrated data pieces.

12. The apparatus according to claim 11, wherein the recognition means performs first matching processing based on the transition of the distance obtained from the vector based on the time series of integrated data pieces, and second matching processing based on a space distribution obtained from the vector based on the time series of integrated data pieces, and recognizes whether or not the time series of first or second input data pieces corresponds to at least one recognition target, based on results of the first and second matching processing.

13. A recognition method for recognizing whether or not a time series of input data pieces comprising image data and noise data corresponds to at least one recognition target, comprising the steps of:

integrating a time series of first input data pieces and a time series of second input data pieces, thereby to output a time series of integrated data pieces; and

recognizing whether or not the time series of first or second input data pieces corresponds to at least one recognition target, based on transition of a distance obtained from a vector based on the time series of integrated data pieces.

14. A recording medium which records a program for making a computer execute recognition processing for recognizing whether or not a time series of input data pieces comprising image data and noise data corresponds to at least one recognition target, wherein the program comprises:

an integration step of integrating a time series of first input data pieces and a time series of second input data pieces, thereby to output a time series of integrated data pieces; and

a recognition step of recognizing whether or not the time series of first or second input data pieces corresponds to at least one recognition target, based on transition of a distance obtained from a vector based on the time series of integrated data pieces.

15. A recognition apparatus for recognizing whether or not a time series of input data pieces corresponds to a recognition target, comprising:

storage means for storing an expectation degree at which observation is expected when the time series of input data pieces is inputted, with respect to each of identifiers corresponding to code vectors used for vector quantization;

vector quantization means for vector-quantizing the time series of input data pieces and for outputting a series of identifiers indicating code vectors;

properness detection means for obtaining properness as to whether or not the time series of input data pieces corresponds to the recognition target, with use of the series of identifiers obtained from the input data and the expectation degrees of the identifiers; and

recognition means for recognizing whether or not the time series of input data pieces corresponds to the recognition target, based on the properness, wherein the vector quantization means outputs an identifier of a code vector which has a shortest distance to the input data piece with respect to each of the time series of input data pieces.

16. A recognition apparatus for recognizing whether or not a time series of input data pieces corresponds to a recognition target, comprising:

storage means for storing an expectation degree at which observation is expected when the time series of input data pieces is inputted, with respect to each of identifiers corresponding to code vectors used for vector quantization;

vector quantization means for vector quantizing the time series of input data pieces and for outputting a series of identifiers indicating code vectors;

properness detection means for obtaining properness as to whether or not the time series of input data pieces corresponds to the recognition target, with use of the series of identifiers obtained from the input data and the expectation degrees of the identifiers; and

recognition means for recognizing whether or not the time series of input data pieces corresponds to the recognition target, based on the properness, wherein the time series of input data pieces is an integrated parameter which integrates a characteristic parameter of a speech and a characteristic parameter of an image of lips when the speech is spoken.

17. A recognition apparatus comprising:

detection means for detecting a characteristic parameter with respect to each of a plurality of input data pieces;

normalization means for normalizing the characteristic parameter of each of the plurality of input data pieces;

integration means for integrating a plurality of normalized characteristic parameters into an integrated parameter; and

recognition means for recognizing whether or not one or more of the plurality of input data pieces correspond to a recognition target, based on the integrated parameter, wherein the plurality of data pieces include at least data pieces of an image and a speech.

18. The apparatus according to claim 17, wherein the recognition means recognizes the speech.

19. A recognition apparatus comprising:

detection means for detecting a characteristic parameter with respect to each of a plurality of input data pieces;

normalization means for normalizing the characteristic parameter of each of the plurality of input data pieces;

integration means for integrating a plurality of normalized characteristic parameters into an integrated parameter;

a recognition means for recognizing whether or not one or more of the plurality of input data pieces correspond to a recognition target, based on the integrated parameter; and

time axis normalization means for normalizing the integrated parameter in a time axis direction.

20. A recognition apparatus for recognizing whether or not a time series of input data pieces corresponds to at least one recognition target, comprising:

integration means for integrating a time series of first input data pieces and a time series of second input data pieces, thereby to output a time series of integrated data pieces;

recognition means for recognizing whether or not the time series of first or second input data pieces corresponds to at least one recognition target, based on transition of a distance obtained from a vector based on the time series of integrated data pieces, wherein a code vector in a code book used for vector quantization and a distance transition model expressing transition of a distance to a standard series, the recognition means accumulates the distance when a vector quantization result obtained by vector-quantizing a vector based on the integrated data pieces, with use of the code book is observed, thereby to recognize whether or not the time series of the first or second input data pieces correspond to at least one recognition target, based on an accumulation result.