Chronological telephone system

• US 6,181,778 B1
• Filed: 02/26/1998
• Issued: 01/30/2001
• Est. Priority Date: 08/30/1995
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Hataoka, Nobuo, Ikeda, Hisashi, Kaneko, Youichi, Fujisawa, Hiromichi, Sagawa, Hirohiko, Ohki, Masaru, Sakiyama, Tomoko
• Primary Examiner(s): Le, Huyen
• Assistant Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US09/029,557
• Time in Patent Office: 1,069 Days
• Field of Search: 379/52, 379/93.15, 340/825.19, 382/100, 704/2, 704/3, 704/271, 704/277, 704/235, 704/260, 348/14, 348/15
• US Class Current: 379/52
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00
• CPC Class Codes: H04M 11/066 Telephone sets adapted for data transmision
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 5/45 Picture in picture , e.g. displaying simultaneously another television channel in a region of the screen
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A sign language telephone device is offered which enables an aurally handicapped person who uses the sign language to converse with a normal person at a distant place who does not know the sign language. The sign language telephone device is placed on the side of the aurally handicapped person, and hand gestures of the sign language inputted from a sign language input means are recognized as the sign language, and the recognized sign language is translated to Japanese. The translated Japanese word train is converted to synthesized voices and it is transmitted to a videophone on the side of a normal person. The voices from the videophone are recognized, and the recognized Japanese is translated to the sign language to generate sign language animations and they are displayed on the screen of a TV set on the side of the aurally handicapped person. According to the present invention, it is made possible for an aurally handicapped person to have conversation easily with a normal person at a distant place who does not know the sign language through an existing network.

Claims

1. A sign language telephone system comprising:

a videophone device; and

a sign language telephone control device operatively connected to said videophone device via a network, which translates an input sign language, generates synthesized voices for transmission to said videophone device via said network, recognizes voices from said videophone device, and generates a sign language animation from recognized results for a visual display, said sign language telephone control device comprising;

a sign-language-to-Japanese-translation subsystem comprising;

a sign language recognition unit which recognizes and translates the input sign language to a sign language words train;

a sign-language-to-Japanese-translation unit which translates the sign language words train into Japanese;

a voice synthesizing unit which synthesizes the translated Japanese as voices; and

a gesture-to-sign language animation conversion unit which generates animations by using positional data of hand gestures from the input sign language; and

a Japanese-to-sign-language-translation subsystem comprising which translates voices from said videophone device into a sign language words train.

2. A sign language telephone system as claimed in claim 1, further comprising a television set, a camera, a microphone, a speaker and sign language input means for providing the input sign language.

3. A sign language telephone system as claimed in claim 2, wherein said sign language telephone control device further comprises means to enable a visual display of a Japanese character train, which is a result of translation of the input sign language as part of said images to be transmitted to said videophone device via said network.

4. A sign language telephone system as claimed in claim 2, wherein said sign language telephone control device further comprises means to enable a visual display of a Japanese character train, which is a result of translation of the input sign language and at least one animation among animations generated from said animation generation unit, in a part of the screen of said television set.

5. A sign language telephone system as claimed in claim 2, wherein said Japanese-to-sign-language-translation subsystem comprises:

a voice recognition unit which recognizes Japanese voices from said videophone device via said network;

a Japanese-to-sign-language-translation unit which translates Japanese recognized from said voice recognition unit into the sign language; and

a sign language animation generation unit which displays the translated sign language as sign language animations.

6. A sign language telephone system as claimed in claim 2, wherein said sign-language-to-Japanese-translation subsystem further comprises means to select voice quality or adjustment of voice synthesis.

7. A sign language telephone system as claimed in claim 1, wherein said sign language telephone control device further comprises language translation means which allows communications between the sign language as a first language and the voices as a second language.

8. A sign language telephone system comprising:

a videophone device;

a television set;

a camera;

a microphone;

speakers;

sign language input means; and

a sign language telephone control device connected to said television set, said camera, said microphone, said speakers, said sign language input means, and operatively connected to said videophone device via a network, which translates an input sign language from said sign language input means, generates synthesized voices for transmission to said videophone device via said network, recognizes voices from said videophone device, and generates a sign language animation based on recognized voices for a visual display, said sign language telephone control device comprising;

a sign-to-language-to-Japanese-translation subsystem comprising;

a sign language recognition unit which recognizes and translates the input sign language to a sign language words train;

a sign-language-to-Japanese-translation unit which translates the sign language words train into Japanese;

a voice synthesizing unit which synthesizes the translated Japanese as voices; and

a sign-language-to-Japanese-translation control unit which controls operation in one of a translation mode and a non-translation mode, including enabling translation of the sign language words train from the sign language recognition unit to Japanese by the sign-language-to-Japanese-translation unit during said translation mode, and terminating translation of the sign language words train to Japanese during said non-translation mode; and

a Japanese-to-sign-language-translation subsystem which translates voices from said videophone device into a sign language words train.

9. A sign language telephone system as claimed in claim 8, wherein said sign language telephone control device comprises means to decide between a translation mode a non-translation mode with a button, means to decide said translation mode when an operator is facing said camera, and said non-translation mode when a hand of said operator is at a home position, or when movement of the hand of said operator is stopped, and means in which hand gestures are set beforehand and one of said translation mode and said non-translation mode is decided by the hand gesture.

10. A sign language telephone system as claimed in claim 8, wherein said sign-language-to-Japanese-translation subsystem further comprises a gesture-to-sign-language animation conversion unit which generates animations by using positional data of hand gestures from the input sign language as data to move animations.

11. A sign language telephone system comprising:

a videophone device;

a television set;

a camera;

a microphone;

speakers;

sign language input means; and

a sign language telephone control device connected to said television set, said camera, said microphone, said speakers, said sign language input means, and operatively connected to said videophone device via a network, which translates an input sign language from said sign language input means, generates synthesized voices for transmission to said videophone device via said network, recognizes voices from said videophone device, and generates a sign language animation based on recognized voices for a visual display, said sign language telephone control device comprising;

a sign-to-language-to-Japanese-translation subsystem operable in an actual image mode, an animation mode, a suppression mode, and comprising;

a sign language recognition unit which recognizes and translates the input sign language to a sign language words train;

a sign-language-to-Japanese-translation unit which translates the sign language words train into Japanese;

a voice synthesizing unit which synthesizes the translated Japanese as voices;

a gesture-to-sign-language animation conversion unit which generates sign language animations by using positional data of hand gestures from the input sign language as data to move animations; and

an image generation unit which generates sign language animations, and which enables a visual display of an actual image from said camera during said actual image mode, sign language animations during said animation mode, and either a designated still image or no image during said suppression mode; and

a Japanese-to-sign-language-translation subsystem which translates voices from said videophone device into a sign language words train.

12. A sign language telephone system as claimed in claim 11, wherein said sign-language-to-Japanese-translation subsystem further comprises an answer phone control unit which generates a response message to a telephone call made using actual images input from said camera, sign language animations generated from said gesture to sign language animation conversion unit, an image selected from still pictures registered in a memory, Japanese character train obtained from translating the sign language to Japanese and voices generated from said voice synthesizing unit.

13. A sign language telephone system comprising:

a videophone device;

a television set;

a camera;

a microphone;

speakers;

sign language input means; and

a sign language telephone control device connected to said television set, said camera, said microphone, said speakers, said sign language input means, and operatively connected to said videophone via a network, which translates an input sign language from said sign language input means, generates synthesized voices for transmission to said videophone device via said network, recognizes voices from said videophone device, and generates a sign language animation based on recognized voices for a visual display, said sign language telephone control device comprising;

a sign-to-language-to-Japanese-translation subsystem operable in an actual image mode, an animation mode, a suppression mode, and comprising;

a sign language recognition unit which recognizes and translates the input sign language into a sign language words train;

a sign-language-to-Japanese-translation unit which translates the sign language words train into Japanese;

a voice synthesizing unit which synthesizes the translated Japanese as voices;

a gesture-to-sign-language animation conversion unit which generates sign language animations by using positional data of hand gestures from the input sign language as data to move animations; and

an image, voice and character synchronizing unit which generates images and voices to be transmitted to said videophone device via said network by composing images, a character train and voices, while synchronizing the translated Japanese and the voices generated in said voice synthesizing unit; and

a Japanese-to-sign-language-translation subsystem which translates voices from said videophone device into a sign language words train.

14. A sign language telephone system as claimed in claim 13, wherein said sign-language-to-Japanese-translation subsystem further comprises a videophone image generation unit which generates images to be sent to said videophone device via said network by adding Japanese characters recognized by said Japanese-to-sign-language-translation subsystem to images generated from said image, voice and character synchronizing unit.

15. A sign language telephone system comprising:

a videophone device;

a television set;

a camera;

a microphone;

speakers;

sign language input means; and

a sign language telephone control device connected to said television set, said camera, said microphone, said speakers, said sign language input means, and operatively connected to said videophone device via a network, which translates an input sign language from said sign language input means, generates synthesized voices for transmission to said videophone device via said network, recognizes voices from said videophone device, and generates sign language animations based on recognized voices for a visual display, said sign language telephone control device comprising;

a sign-language-to-Japanese-translation subsystem comprising;

a sign language recognition unit which recognizes and translates the input sign language into a sign language words train;

a sign-language-to-Japanese-translation unit which translates the sign language words train into Japanese; and

a voice synthesizing unit which synthesizes the translated Japanese as voices; and

a Japanese-to-sign-language-translation subsystem comprising;

a voice recognition unit which recognizes Japanese voices spoken from said videophone device via said network;

a Japanese-to-sign-language-translation unit which translates the Japanese recognized into the sign language;

a sign language animation generation unit which generates sign language animations based on the translated sign language; and

an image, voice and character synchronizing unit which composes a character train, sign language animations and images while synchronizing with the translated Japanese, the sign language animations generated from said sign language animation generation unit and actual images transmitted from said videophone device via said network.

16. A sign language telephone system comprising:

a videophone device;

a television set;

sign language input means; and

a sign language telephone control device connected to said television set, said sign language input means, and operatively connected to said videophone device via a network, which translates an input sign language from said sign language input means, generates synthesized voices for transmission to said videophone device via said network, recognizes voices from said videophone device, and generates sign language animations based on recognized voices for a visual display, said sign language telephone control device comprising;

a sign-language-to-Japanese-translation subsystem comprising;

a sign language recognition unit which recognizes and translates the input sign language to a sign language words train;

a sign-language-to-Japanese-translation unit which translates the sign language words train to Japanese;

a voice synthesizing unit which synthesizes the translated Japanese as voices; and

means which enables a visual display of the actual images received from said videophone device with the translated sign language and the sign language animations generated from said sign language animation generation unit on said television set; and

a Japanese-to-sign-language-translation subsystem comprising;

a voice recognition unit which recognizes Japanese voices spoken from said videophone device via said network;

a Japanese-to-sign-language-translation unit which translates the Japanese recognized into the sign language; and

a sign language animation generation unit which generates sign language animations based on the translated sign language.

17. A sign language telephone system as claimed in claim 16, wherein said sign language telephone control device further comprises at least 1 means out of 2 means shown below:

one means is a display means to display 2 display areas as shown below on said television screen by turns;

one is an image display area to display actual images transmitted from said videophone device and another is a display area to display said sign language animations generated from said sign language animation generation unit in recognizing the voices transmitted from said videophone device and translating to animations; and

an enlargement means to enlarge the display area for the sign language animations generated from said sign language animation generation unit in recognizing the voices transmitted from said videophone device and translating to the sign language.

18. A sign language telephone system comprising:

a display monitor;

sign language input means; and

a sign language telephone control device connected to said display monitor, and operatively connected to a remote telephone device via a network, which operates in one of a translation mode and a non-translation mode to translate an input sign language from said sign language input means into a character train of a designated language for transmission to said remote telephone device via said network, and which recognizes voice spoken from said remote telephone device via said network in said designated language for translating the recognized voice into a sign language for a visual display on said display monitor in one of a sign language animation and a character train.

19. A sign language telephone system as claimed in claim 18, wherein said sign language telephone control device comprising first and second translation subsystems for translations between the sign language input from said sign language input means by an aurally handicapped user and voice information spoken by a normal user from said remote telephone device.

20. A sign language telephone system as claimed in claim 19, wherein said first translation subsystem translates the sign language from said sign language input means by said aurally handicapped user into a character train of said designated language, synthesizes said character train into voice information in said designated language for transmission to said remote telephone device, provides a visual display of said character train on said display monitor to confirm whether the sign language input by said aurally handicapped user has been correctly translated, and converts hand gestures of the sign language from said sign language input means into sign language animations for a visual display on said display monitor.

21. A sign language telephone system as claimed in claim 19, wherein said second translation subsystem recognizes and translates voice information spoken by a normal user in said designated language into a character train of said designated language, converts the recognized voice information into a sign language for a visual display on said display monitor, and composes the recognized voice information with an image from a camera for transmission to said remote telephone device to confirm whether voice information spoken by said normal user has been correctly recognized.

22. A sign language telephone system as claimed in claim 19, wherein said first translation subsystem comprises:

a sign language recognition unit which recognizes and translates the input sign language to a sign language words train;

a translation unit which translates the recognized sign language words train into a character train of said designated language; and

a voice synthesizing unit which synthesizes the translated language as voice information for transmission to said remote telephone device via said network.

23. A sign language telephone system as claimed in claim 22, wherein said first translation subsystem further comprises:

a countenance recognition unit which recognizes a countenance of said aurally handicapped user captured by said camera; and

a conversion unit which generates sign language animations based on positional data of hand gestures from said input sign language and the recognized countenance from said countenance recognition unit.

24. A sign language telephone system as claimed in claim 22, wherein said first translation subsystem further comprises a translation control unit which controls operation in one of a translation mode and a non-translation mode, including enabling translation of the sign language words train from the sign language recognition unit into said designated language during said translation mode;

and terminating translation of the sign language words train into said designated language during said non-translation mode.

25. A sign language telephone system as claimed in claim 22, wherein said first translation subsystem further comprises means for enabling a visual display of an actual image captured from said camera during an actual image mode, said sign language animations during an animation mode;

and one of a still image or no image during a suppression mode.

26. A sign language telephone system as claimed in claim 22, wherein said second translation subsystem comprises:

a voice recognition unit which recognizes the voice information in said designated language from said remote telephone device via said network; and

a translation unit which translates the voice information from said videophone device into a sign language words train for a visual display on said display monitor.

27. A sign language telephone system comprising:

a videophone device;

a camera;

sign language input means; and

a sign language telephone control device connected to said camera and said sign language input means, and operatively connected to said videophone device via a network, said sign language telephone control device including first and second translation subsystems for translations between the sign language and voice information,

wherein said first translation subsystem comprises;

a sign language recognition unit which recognizes and translates the input sign language to a sign language words train;

a translation unit which translates the recognized sign language words train into a designated language;

a voice synthesizing unit which synthesizes the translated language as voice information for transmission to said videophone device via said network;

a countenance recognition unit which recognizes a countenance of user captured by said camera; and

a conversion unit which generates sign language animations based on positional data of hand gestures from said input sign language and the recognized countenance from said countenance recognition unit; and

wherein said second translation subsystem comprises;

a voice recognition unit which recognizes the voice information in said designated language from said videophone device via said network; and

a translation unit which translates the voice information from said videophone device into a sign language words train.

28. A sign language telephone network comprising:

a network;

a first videophone connected to said network, and equipped with sign language input means to input a sign language from an aurally handicapped user;

a second videophone connected to said network, and equipped with at least a camera to capture an image of a normal user; and

a sign language interpretation server connected to said network, which translates the sign language input form said first videophone, into a character train in a designated language for transmission to said second videophone via said network, and which translateds voice spoken from said second videophone into a sign language for a visual display on said first videophone,

wherein said sign language interpretation server comprises first and second translation subsystems for translations between the sign language input from said first videophone by an aurally handicapped user and voice information spoken from said second videophone by a nomal user, via said network,

wherein said first translation subsystem translates the sign language from said aurally handicapped user into a character train of said designated language, synthesizes said character train into voice information in said designated language for transmission to said second videophone, provides a visual display of said character train on said first videophone to confirm whether the sign language from said aurally handicapped user had been correctly translated, and converts hand gestures of the sign language into sign language animations for a visual display on said first videophone; and

wherein said second translation subsystem recognizes and translates voice information spoken from said normal user in said designated language into a character train of said designated language, converts the recognized voice informtioninto a sign language for a visual display on said first videophone, and composes the recognized voice information with an image from said camera for transmission to said videophone to confirm whether voice information spoken from said normal user has been correctly recognized.

29. A sign language telephone network as claimed in claim 28, wherein said first translation subsystem comprises:

a sign language recognition unit which recognizes and translates the input sign language to a sign language words train;

a translation unit which translates the recognized sign language words train into a character train of said designated language; and

a voice synthesizing unit which synthesizes the translated language as voice information for transmission to said remote telephone device via said network.

30. A sign language telephone network as claimed in claim 29, wherein said first translation subsystem further comprises:

a countenance recognition unit which recognizes a countenance of said aurally handicapped user captured by said camera; and

a conversion unit which generates sign language animations based on positional data of hand gestures from said input sign language and the recognized countenance from said countenance recognition unit.

31. A sign language telephone network as claimed in claim 30, wherein said first translation subsystem further comprises a translation control unit which controls operation in one of a translation mode and a non-translation mode, including enabling translation of the sign language words train from the sign language recognition unit into said designated language during said translation mode;

and terminating translation of the sign language words train into said designated language during said non-translation mode.

32. A sign language telephone network as claimed in claim 29, wherein said second translation subsystem comprises:

a voice recognition unit which recognizes the voice information in said designated language from said remote telephone device via said network; and

a translation unit which translates the voice information from said videophone device into a sign language words train for a visual display on said display monitor.

33. A sign language telephone network as claimed in claim 31, wherein said first translation subsystem further comprises means for enabling a visual display of an actual image captured from said camera during an actual image mode, said sign language animations during an animation mode;

and one of a still image or no image during a suppression mode.