Continuous sign-language recognition apparatus and input apparatus

• US 5,699,441 A
• Filed: 09/09/1996
• Issued: 12/16/1997
• Est. Priority Date: 03/10/1992
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Abe, Masahiro, Sagawa, Hirohiko
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Tran, Phuoc
• Application Number:
  US08/716,911
• Time in Patent Office: 463 Days
• Field of Search: 382/100, 382/181, 382/215, 382/190, 382/207, 382/209, 395/2.5, 395/2.8, 341/20, 341/21, 434/112
• US Class Current: 382/100
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06F 40/55 Rule-based translation
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00

Abstract

A continuous sign-language recognition apparatus and input apparatus which employ expressions of template patterns of sign-language word as well as the template patterns of sign-language word to realize high speed and highly accurate sign-language recognition. Each of component patterns constituting a template pattern of a sign-language word is expressed by a feature vector representing the pattern when the pattern is a static pattern, or by a time-series pattern of a feature vector when the pattern is a dynamic pattern. Also, using the template patterns of sign-language word, different matching methods are applied to the static and dynamic patterns for recognizing each component pattern, and the respective results are integrated on the basis of a temporal overlap of the respective component patterns to continuously recognize sign-language words.

Claims

1. A continuous sign-language recognition apparatus comprising:

sign-language input means for converting a series of sign-language words into sign-language patterns which are time-series patterns of feature vectors and inputting said sign-language patterns;

a dictionary of template pattern of sign-language word for storing template patterns of sign-language word; and

sign-language word recognition means coupled to said sign-language input means and said dictionary for matching said sign-language patterns with said template patterns of sign-language word to recognize sign-language words expressed in said sign-language patterns,

each of said template patterns of sign-language word being expressed as a feature vector for the sign-language word having a static pattern, and as a time-series pattern of a feature vector for the sign-language word having a dynamic pattern;

each of said sign-language patterns and each of said template patterns of sign-language word being expressed as a combination of a plurality of component patterns; and

each of said template patterns of sign-language word being expressed as a feature vector for the component pattern having a static pattern, and as a time-series pattern of a feature vector for the component pattern having a dynamic pattern;

said sign-language word recognition means includes matching portions of component pattern for matching the respective component patterns constituting one of said sign-language patterns with the component patterns constituting said template patterns of sign-language word, and a detection portion of sign-language word for integrating the matching results of said matching portions of component pattern to detect a sign-language word;

wherein each of said matching portions of component pattern includes;

selection means for determining whether each of the component patterns is static or dynamic with reference to said dictionary of template pattern of sign-language word;

static pattern detection means for matching one of said sign-language patterns with said template patterns of sign-language word, in response to a component pattern which is a static pattern, and outputting pattern detected positions; and

dynamic pattern detection means for matching one of said sign-language patterns with said template patterns of sign-language word, in response to a component pattern which is a dynamic pattern, and outputting pattern detected positions.

2. A continuous sign-language recognition apparatus according to claim 1, wherein

said each component pattern is weighted with a weighting coefficient indicative of an importance degree thereof with respect to the whole pattern, and pattern matching is performed using said weighting coefficient.

3. A continuous sign-language recognition apparatus according to claim 1, wherein

said detection portion of sign-language word includes:

calculation means for calculating an overlap of the pattern detected positions of the respective component patterns outputted from said matching portions of component pattern; and

decision means of overlap for determining the existence of one of said template patterns of sign-language word that is matched with one of said sign-language patterns, if the calculated overlap of the respective component patterns is equal to or more than a predetermined value.

4. A continuous sign-language recognition apparatus according to claim 3, wherein

said decision means of overlap determines, when said component patterns are each constituted of a static pattern, that one of said template patterns of sign-language word is matched with one of said sign-language patterns, if the respective component patterns are detected within a predetermined time period.

5. A continuous sign-language recognition apparatus according to claim 3, wherein

said detection portion of sign-language word includes a detection part of simultaneous gesture for detecting a simultaneous gesture in sign-language words from detection results of a plurality of sign-language word patterns, based on an overlap of said sign-language word patterns.

6. A continuous sign-language recognition system according to claim 1, wherein said template patterns of sign language word are expressed by a variation in the gesture element pattern and appended information indicative of a semantic variation of the gesture element pattern, and

said sign-language word recognition means detects the variation of an expression or format of the gesture element pattern, and calculates and extracts appended information described in the template pattern.

7. A continuous sign-language recognition system comprising:

means for converting a series of sign-language words into sign-language patterns which are time-series patterns of feature vectors and inputting said converted sign-language patterns;

means for storing template patterns for sign-language words;

pattern matching means for matching said input sign-language patterns with said template patterns of sign-language word, to recognize a sign-language word expressed in said input sign-language patterns;

recognizing means responsive to said matching means for recognizing a sign-language word from the results of matching between the input sign-language pattern and the template patterns; and

means for outputting the sign-language word recognized by said recognizing means;

each of said template patterns for sign-language words includes a description for identifying whether each of gesture elements each constituting a sign-language word is a static pattern or a dynamic pattern, and a pattern of the gesture element expressed as a format that is different depending upon whether the gesture element is a static pattern or a dynamic pattern;

said pattern matching means, referring to said template patterns in said storing means, selecting a relevant one of different matching schemes in accordance with whether each gesture element in the template patterns is static or dynamic and performing a matching of said input sign language patterns with the pattern of each gesture element using the selected matching scheme, independently from the other gesture elements; and

said sign-language word recognizing means integrating recognition results of the gesture elements each constituting a sign-language word, based on a time relation with the recognition results.

8. A continuous sign-language recognition system according to claim 7, wherein the gesture element patterns of sign-language word in the template patterns is expressed as a feature vector for the sign-language word having a static pattern, and as a time-series pattern of a feature vector for the sign-language word having a dynamic pattern.

9. A continuous sign-language recognition system according to claim 7, wherein each of said gesture element patterns is weighted with a weighting coefficient indicative of an importance degree thereof with respect to the whole pattern, and pattern matching is performed using said weighting coefficient.

10. A continuous sign-language recognition system according to claim 7, wherein said sign-language word recognition means calculates an overlap of detection positions between gesture elements detected by the matching of the gesture element patterns, and determines that a sign-languages word corresponding to a template pattern matched with the input sign-language pattern is present, when the calculated overlap for inter-gesture elements exceeds a predetermined value.

11. A continuous sign-language recognition system according to claim 10, wherein when all of gesture element patterns constituting a sign-language word are static patterns, said sign-language word recognition means determines that a sign-language word corresponding to a template pattern matched with the input sign-language pattern is present in response to a detection of the detected gesture element within a predetermined time range.

12. A continuous sign-language recognition system according to claim 10, wherein said sign-language word recognition means detects concurrent operation in a sign language from detection results of a plurality of sign-language word patterns, based on an overlap thereof.

13. A continuous sign-language recognition system comprising:

means for converting a series of sign-language words into sign-language patterns which are time-series patterns of feature vectors and inputting said sign-language patterns;

pattern storage means for storing template patterns of sign-language morphemes;

word store means for storing sign-language words each expressed as a combination of sign-language morphemes constituting a sign-language word;

pattern matching means for matching said sign-language patterns with said template patterns of sign-language morphemes, in order to recognize sign-language morphemes expressed in said sign-language patterns;

morpheme recognition means responsive to said pattern matching means for recognizing sign-language morphemes from the results of matching between one of said sign-language patterns and the template patterns of sign-language morphemes;

word recognition means connected to said morpheme recognition means for recognizing a sign-language word by matching the recognized sign-language morphemes with a combination of sign-language morphemes constituting a sign-language word; and

means for outputting the sign-language word recognized by said word recognizing means;

said template patterns stored in said sign-language morpheme template storing means including a description of identifying whether each of gesture elements constituting a sign-language morpheme is a static pattern or a dynamic pattern, and a pattern of the gesture element expressed as a format different depending upon whether the gesture element is a static pattern or a dynamic pattern;

said pattern matching means for matching the input sign-language patterns with the sign-language morpheme template patterns, referring to said template patterns in said pattern storage means, selecting a relevant one of different matching schemes in accordance with whether each gesture element in the template patterns is static or dynamic, and performing a matching of said sign language patterns with the pattern of each gesture element using the selected matching scheme, independently from the other gesture elements; and

said morpheme recognition means integrating recognition results of the gesture elements each constituting a sign-language morpheme, based on a time relation with the recognition results to recognize the sign-language morpheme.

14. An input system comprising:

sign-language input means for converting a series of gestures into gesture patterns which are time-series patterns of feature vectors and inputting the converted gesture patterns;

storing means for storing template patterns of gestures;

matching means for matching the input gesture patterns with the template patterns of the gesture stored in said storing means, in order to recognize a gesture expressed in the gesture pattern;

gesture recognizing means responsive to said matching means for recognizing the gesture expressed in the gesture pattern, from the results of matching between the input gesture patterns and the template patterns; and

means for performing a processing corresponding to the recognized gesture;

each of said gesture template patterns stored in said template pattern storing means including a description of identifying whether each of gesture elements constituting a gesture is a static pattern or a dynamic pattern, and a pattern of the gesture element expressed as a format different depending upon whether the each gesture element is a static pattern or a dynamic pattern;

said matching means for matching the input gesture patterns with the gesture template patterns, referring to said template patterns in said storing means, selecting a relevant one of different matching schemes in accordance with whether each of gesture elements in the template patterns is static or dynamic and performing a matching of said gesture patterns with the pattern of each gesture element using the selected matching scheme, independently from the other gesture elements; and

said gesture recognizing means integrating recognition results of the gesture elements constituting a gesture, based on a time relation with the recognition results.

15. An input system according to claim 14, wherein the template of the gesture is expressed as a feature vector for the gesture having a static pattern, and as a time-series pattern of a feature vector for the gesture having a dynamic pattern.

16. An input system according to claim 14, wherein said gesture recognition means calculates an overlap of recognition (detection) positions between gesture elements recognized (detected) by the matching of the gesture element patterns, and determines that a gesture corresponding to a template pattern matched with the input gesture pattern is present, when the calculated overlap for inter-gesture elements exceeds a predetermined value.

17. An input system according to claim 16, wherein when all of gesture elements constituting a gesture are static patterns, said gesture recognizing means determines that a gesture corresponding to a template pattern matched with the input gesture pattern is present in response to a detection of the detected gesture element within a predetermined time range.

18. An input system according to claim 14, wherein the gesture template pattern is expressed by a pattern of gesture elements and an appended information indicative of data to be extracted from the gesture pattern;

and

said gesture recognizing means calculates and extract necessary data from the gesture pattern based on the recognition result of the gesture and the appended information in the template pattern.