Method of providing sign language animation to a monitor and process therefor

• US 20020140718A1
• Filed: 03/29/2001
• Published: 10/03/2002
• Est. Priority Date: 03/29/2001
• Status: Abandoned Application

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Philips Electronics North America Corporation (Koninklijke Philips N.V.)
• Inventors: Lin, Yun-Ting, Yan, Yong
• Application Number:
  US09/821,524
• Publication Number:
  US 20020140718A1
• Time in Patent Office: Days
• US Class Current: 345/706
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: G10L 21/10 Transforming into visible information

Abstract

A system and method for generating an animation video signal of sign language gestures corresponding to words in an audio/video signal for display on a monitor screen. A speech component is isolated from the audio/video signal and the spoken words in the speech signal are recognized. The spoken words are then used to identify sign language gestures which are mapped onto an animation model for generating an animation signal. The animation signal is used to animate a character icon stored in the monitor to display sign language gestures corresponding to the words of the speech signal.

Claims

1. A method of displaying, on a monitor having a display screen, a sign language animation of a speech component of an audio/video signal while simultaneously displaying, on the monitor display screen, a visual image corresponding to a video component of the audio/video signal, comprising the steps of:

mapping the speech component to a sign language animation model to generate animation model parameters corresponding to sign language images;

generating an animation signal from said animation model parameters by using a processor connected to the monitor; and

rendering, from said animation signal, an animation image on a portion of the monitor, said animation image containing sign language gestures corresponding to the speech component of the audio/video signal.

3. The method of claim 1, wherein the audio/video signal is provided to the monitor by a transmitter remotely located from the monitor, and wherein said mapping step is performed remotely from the monitor.

4. The method of claim 3, wherein the mapping step is performed proximate the transmitter.

5. The method of claim 4, further comprising the step of transmitting the animation model parameters to the monitor.

6. The method of claim 1, wherein the processor comprises a memory containing data for multiple character icons and wherein said animation image is rendered by animating a select one of the multiple character icons.

7. The method of claim 1, wherein the processor is activated by selecting a function on a monitor control device.

8. The method of claim 6, wherein the select one character icon includes a face having a mouth and wherein said animation image further comprises the step of animating the mouth to simulate speech corresponding to the speech component of the audio/video signal.

9. The method of claim 6, wherein said memory includes commands corresponding to a dictionary of sign language symbols and wherein said mapping step comprises correlating spoken words from the speech signal to the sign language symbols.

10. The method of claim 1, wherein Synthetic Natural Hybrid Coding (SNHC) is used to generate the animation model parameters.

2. The method of claim 2, further comprising the step of receiving, before performing said mapping step, the audio/video signal at the monitor, and isolating the speech component from the audio/video signal, wherein said isolating step is performed by the processor.

11. A method of displaying, on a monitor having a display screen, a sign language animation of a speech component of an audio/video signal while simultaneously displaying, on the monitor display screen, a visual image corresponding to a video component of the audio/video signal, comprising the steps of:

isolating the speech component from an audio component of the audio/video signal;

identifying words represented by the isolated speech component;

mapping the identified words to a sign language animation model to generate animation model parameters corresponding to sign language images;

transmitting the audio/video signal and the animation model parameters to the monitor;

receiving the transmitted audio/video signal at the monitor;

generating an animation signal from said animation model parameters by using a processor connected to the monitor;

displaying a video component of the audio/video signal on the monitor display screen; and

rendering, from said animation signal, an animation image on a portion of the monitor display screen, said animation image containing sign language gestures corresponding to the speech component of the audio/video signal.

12. The method of claim 11, wherein the processor comprises a memory containing data for multiple character icons and wherein said animation image is rendered by animating a select one of the multiple character icons.

13. The method of claim 11, wherein the processor is activated by selecting a function on a monitor control device.

14. The method of claim 12, wherein the select one character icon includes a face having a mouth and wherein said animation image further comprises the step of animating the mouth to simulate speech corresponding to the speech component of the audio/video signal.

15. The method of claim 11, wherein said memory includes commands corresponding to a dictionary of sign language symbols and wherein said mapping step comprises correlating spoken words from the speech signal to the sign language symbols.

16. The method of claim 11 wherein Synthetic Natural Hybrid Coding (SNHC) is used to generate the animation model parameters.

18. The system of claim 17, wherein said mapping means comprises Synthetic Natural Hybrid Coding (SNHC).

19. The system of claim 17, wherein the processor comprises a memory containing data for multiple character icons and wherein said animation image is rendered by animating a select one of the multiple character icons.

20. The system of claim 19, wherein the select one character icon includes a face having a mouth and wherein said animation image further comprises the step of animating the mouth to simulate speech corresponding to the speech component of the audio/video signal.

21. The system of claim 19, wherein said memory includes commands corresponding to a dictionary of sign language symbols and wherein said mapping means comprises means for correlating spoken words from the speech signal to the sign language symbols.

17. A system for producing an animation image on a monitor display screen to display, to a viewer of the monitor, sign language gestures corresponding to a speech signal derived from an audio signal component of an audio/video signal, the system comprising:

a transmitter for transmitting the audio/video signal to the monitor;

a receiver connected to the monitor for receiving the transmitted signal;

a memory connected to the monitor for storing sign language animation model parameters corresponding to at least one animation character icon;

a processor connected to the receiver and to the memory for isolating the speech signal from the audio signal component of the transmitted audio/video signal, the processor comprising means for identifying words represented by the isolated speech signal and means for mapping the identified words to the sign language animation model parameters for generating an animation signal; and

means for rendering the animation image on the monitor using the animation signal to animate the at least one animation character icon.

22. A system for producing an animation image on a monitor display screen to display, to a viewer of the monitor, sign language gestures corresponding to a speech signal derived from an audio signal component of an audio/video signal, the system comprising:

a transmitter processor for isolating the speech signal from the audio signal component of the audio/video signal, the processor comprising means for identifying words represented by the isolated speech signal and means for mapping the identified words to a sign language animation model for generating animation model parameters corresponding to sign language images;

a transmitter for transmitting the audio/video signal and the animation model parameters to the monitor;

a receiver connected to the monitor for receiving the transmitted signal and animation model parameters;

a memory connected to the monitor for storing an animation model of at least one animation character icon;

a receiver processor for generating an animation signal from the animation model parameters for animating the at least one character icon; and

means for rendering the animation image on the monitor using the animation signal to animate the at least one animation character icon.

23. The system of claim 22, wherein said transmitter processor is capable of accessing commands corresponding to a dictionary of sign language symbols and wherein said mapping means comprises means for correlating spoken words from the speech signal to the sign language symbols.

24. The system of claim 22, wherein said mapping means comprises Synthetic Natural Hybrid Coding (SNHC).