Sign recognition apparatus and method and sign translation system using same

• US 5,887,069 A
• Filed: 12/06/1995
• Issued: 03/23/1999
• Est. Priority Date: 03/10/1992
• Status: Expired due to Term

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Abe, Masahiro, Arai, Kiyoshi, Ichikawa, Akira, Toda, Yuji, Sakou, Hiroshi, Shimura, Takanori, Inoue, Kiyoshi, Sagawa, Hirohiko
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Tran, Phuoc
• Application Number:
  US08/567,875
• Time in Patent Office: 1,203 Days
• Field of Search: 382/100, 382/107, 382/159, 382/160, 382/209, 382/215, 395/2.5, 395/2.52, 395/2.53, 395/2.8, 395/2.86, 434/112, 434/116
• US Class Current: 382/100
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

In continuous sign language recognition, reference sign language patterns of good quality are generated. Continuous sign language patterns and reference sign language patterns are efficiently compressed. The compressed continuous sign language patterns are sequentially and directly matched with the compressed reference sign language patterns to recognize the sign language at high speed and with high precision. A reference sign pattern generating unit generates a reference sign language pattern by normalizing sample patterns while taking into consideration of their nonlinear compression/expansion and by calculating an average of the sample patterns. A continuous sign language recognition unit recognizes a continuous sign language at high speed by sequentially matching the continuous sign language of time sequential patterns with reference sign language patterns while allowing nonlinear expansion and compression in the time domain. A sign language translation system is provided in which a sign language as well as the facial expression or emotion of a sign language user is recognized and converted into a spoken language with emotion, and transferred in the form of texts, voices, sign languages of another type to a particular partner among a plurality of unknown system users.

Claims

1. A sign language recognition apparatus for recognizing a continuously expressed sign language, comprising:

input means for inputting the continuously expressed sign language as a time sequential pattern;

means for generating a reference sign language pattern by averaging sample time sequential patterns;

means for storing said reference sign language pattern;

means for deriving characteristic points of said sign language time sequential pattern and said reference sign language pattern;

means for compressing said sign language time sequential pattern and said reference sign language pattern in accordance with said derived characteristic points; and

means for matching said compressed time sequential pattern with said compressed reference sign language pattern to recognize said continuously expressed sign language.

2. A sign language recognition apparatus according to claim 1, wherein said means for generating said reference sign language pattern includes means for obtaining correspondence points between said sample time sequential patterns having a nonlinear expansion/compression in a time axis direction, while nonlinearly moving said sample time sequential patterns in the time axis direction, and obtaining an average of said sample time sequential patterns at each said correspondence point.

3. A sign language recognition apparatus according to claim 1, wherein said matching means includes means for recognizing said continuously expressed sign language by sequentially matching said sign language time sequential pattern with said reference sign language pattern stored in advance while allowing a nonlinear expansion/compression in a time axis direction.

4. A sign language recognition apparatus according to claim 1, further comprising means for compressing a sign language pattern by using, as said characteristic points specific to said sign language time sequential pattern, at least one of a time when a velocity vector becomes minimum, a time when a direction change of the velocity vector exceeds a threshold value, a time when an acceleration of the velocity vector becomes minimum.

5. A sign language recognition apparatus according to claim 4, further comprising means for compressing said sign language time sequential pattern by using at least ones of pattern vectors at said respective characteristic points, and time lengths between said characteristic points.

6. A recognition apparatus for recognizing a continuous pattern, comprising:

input means for inputting a continuous pattern as a time sequential pattern;

means for generating a reference continuous pattern by averaging sample time sequential patterns;

means for storing said reference continuous pattern;

means for deriving characteristic points of said input time sequential pattern and said reference continuous pattern;

means for compressing said input time sequential pattern and said reference continuous pattern in accordance with said derived characteristic points; and

means for matching said compressed input time sequential pattern with said compressed reference continuous pattern to recognize said continuous pattern.

7. A sign language recognition method of recognizing a continuously expressed sign language, comprising the steps of:

inputting sample sign language patterns for generating a reference sign language pattern as a first time sequential pattern, and generating said reference sign language pattern by averaging said first time sequential patterns;

inputting a sign language to be recognized as a second time sequential pattern;

deriving characteristic points of said reference sign language pattern and said second time sequential pattern and;

compressing said reference sign language pattern and said second time sequential pattern in accordance with said derived characteristic points; and

matching said compressed reference sign language pattern with said compressed second time sequential pattern to recognize said sign language.

8. A sign language recognition method according to claim 8, further comprising the steps of:

deriving first characteristic points from said reference sign language pattern, and in accordance with said first characteristic points,

compressing said reference sign language pattern;

deriving second characteristic points from said second time sequential pattern, and in accordance with said second characteristic points, compressing said second time sequential pattern; and

matching said compressed reference sign language pattern with said compressed second time sequential pattern to recognize said sign language.

9. A sign language translation system comprising:

(a) a first input means for inputting a continuously expressed sign language as time sequential data,

(b) means for recognizing a sign language inputted as said time sequential data, and including

(b-1) a sign language word dictionary for storing sign language time sequential data of each sign language word;

(b-2) means for calibrating second sign language time sequential data inputted from said first input means; and

(b-3) means for matching said time sequential data calibrated by said calibration means with said sign language time sequential data stored in said sign language word dictionary, and recognizing the sign language word corresponding to said time sequential data;

(c) means for translating said recognized sign language into a spoken language;

(d) a second input means for inputting a portrait of a sign user; and

(e) look recognition means for recognizing the look of the portrait to obtain the emotion type and the emotion degree;

wherein said translating means translates said recognized sign language into a spoken language according to the emotion type and the emotion degree in said look recognition means.

10. A sign language translation system according to claim 9, further comprising:

matching means for receiving said second sign language time sequential data of a sign language word and said first sign language time sequential data corresponding to said second sign language time sequential data, obtaining and outputting correspondence therebetween at each timing; and

selection means for selecting one of said second sign language time sequential data and an output of said matching means, and outputting said selected one to said calibration means,

wherein said calibration means learns recognition parameters for the calibration in accordance with an output of said matching means.

11. A sign language translation system according to claim 10, wherein a neural network is further provided for learning said recognition parameters for the calibration.

12. A sign language translation system according to claim 9, wherein said translating means adds a dependent word to the sign language word outputted from said sign language recognition means, in accordance with a rule, and outputting a spoken language.

13. A sign language translation system according to claim 9, further comprising means for inputting a portrait of a sign language user, look recognition means for recognizing the look of the portrait to obtain the emotion type and the emotion degree (intensity), and processor means for inputting a spoken language outputted from said spoken language translation means and the emotion type and degree outputted from said look recognition means, and outputting a spoken language added with emotional adjectives.

14. A sign language translation system according to claim 13, further comprising sound output means for synthesizing and outputting sounds, wherein said processor means outputs sounds corresponding to the emotion type and degree.

15. A sign language translation system according to claim 13, further comprising test output means for outputting a text, wherein said processor means outputs said text corresponding to the emotion type and degree.

16. A sign language translation system according to claim 13, further comprising sign language graphics means for outputting a sign language as CG, wherein said processor means outputs said CG corresponding to the emotion type and degree.

17. A sign language translation system according to claim 13, further comprising voice input means having a microphone and voice recognition means.

18. A sign language translation system according to claim 13, further comprising character input means for inputting a character from a keyboard.

19. A sign language translation system according to claim 9, wherein said sign language translation system is installed at each of a plurality of stations connected to a local area network so that information can be transferred between a plurality of said sign language translation systems.

20. A sign language translation system, comprising:

(a) a first input means for inputting a continuously expressed sign language as time sequential data,

(b) means for recognizing a sign language inputted as said time sequential data, and including

(b-1) a sign language word dictionary for storing sign language time sequential data of each sign language word;

(b-2) means for calibrating second sign language time sequential data inputted from said first input means; and

(b-3) means for matching said time sequential data calibrated by said calibration means with said sign language time sequential data stored in said sign language word dictionary, and recognizing the sign language word corresponding to said time sequential data;

(c) means for translating said recognized sign language into a spoken language; and

(d) means for generating sign language words and displaying the generated sign language words on a display;

wherein said translating means translates said recognized sign language into a spoken language and said sign language generating means generates sign language for display on said display unit from said spoken language.

21. A plurality of sign language translation systems interconnected by a local area network (LAN) through respective LAN interfaces, each of said systems comprising:

(a) a first input means for inputting a continuously expressed sign language as time sequential data,

(b) means for recognizing a sign language inputted as said time sequential data, and including

(b-1) a sign language word dictionary for storing sign language time sequential data of each sign language word;

(b-2) means for calibrating second sign language time sequential data inputted from said first input means; and

(b-3) means for matching said sign language time sequential data calibrated by said calibration means with said time sequential data stored in said sign language word dictionary, and recognizing the sign language word corresponding to said time sequential data;

(c) means for translating said recognized sign language into a spoken language;

(d) a second input means for inputting a portrait of a sign user;

(e) look recognition means for recognizing the look of the portrait to obtain the emotion type and the emotion degree;

wherein said translating means translates said recognized sign language into a spoken language according to the emotion type and the emotion degree in said look recognition means.

22. A sign language recognition method of recognizing a continuously expressed sign language, comprising the steps of:

inputting a sign language to be recognized as a time sequential pattern;

deriving characteristic points of said input time sequential pattern;

compressing said input time sequential pattern in accordance with said derived characteristic points; and

matching said compressed input time sequential pattern with a compressed reference sign language pattern to recognize said sign language;

wherein said compressed reference sign language is compressed in accordance with characteristic points of a reference time sequential pattern.

23. A sign language recognition method according to claim 22:

wherein each of said characteristic points represents a pattern at a time, at least, when a velocity vector becomes minimum and when a direction of the velocity vector is changed.

24. A sign language recognition method according to claim 23:

wherein said input time sequential pattern and said reference time sequential pattern are compressed using said characteristic points and a time length between said characteristic points; and

wherein said compressed input time sequential pattern is matched with a compressed reference sign language pattern using said characteristic points and said time length between said characteristic points, while allowing a nonlinear expansion/compression in a time axis direction.

25. A sign language recognition method according to claim 22, wherein said reference time sequential pattern is generated by averaging a plurality of sample time sequential patterns.

26. A sign language recognition method according to claim 25, further comprising the steps of:

obtaining correspondence points between said sample time sequential patterns having a nonlinear expansion/compression in a time axis direction; and

calculating an average of said plurality of sample time sequential patterns at each of said correspondence points.

27. A method of recognizing a continuous pattern, comprising the steps of:

inputting a continuous pattern to be recognized as a time sequential pattern;

deriving characteristic points of said input time sequential pattern;

compressing said input time sequential pattern in accordance with said derived characteristic points; and

matching said compressed input time sequential pattern with a compressed reference continuous pattern to recognize said continuous pattern;

wherein said compressed reference continuous pattern is compressed in accordance with characteristic points of a reference time sequential pattern.

28. A sign language translation method, comprising the steps of:

inputting a continuously expressed sign language as time sequential data;

calibrating said input time sequential data for absorption of personal difference between sign language users;

recognizing said input sign language by matching said calibrated time sequential data with a sign language time sequential data stored in a sign language word dictionary;

translating said recognized sign language into a spoken language; and

outputting said translated spoken language.

29. A sign language translation method according to claim 28, wherein said translated spoken language is outputted as voices, texts on a display.

30. A sign language translation method according to claim 28, further comprising the steps of:

inputting a portrait of a sign user; and

recognizing the look of the portrait to obtain an emotion type and a emotion degree;

wherein said translated spoken language is reflected the emotion type and the emotion degree.