Matching, synchronization, and superposition on orginal speaking subject images of modified signs from sign language database corresponding to recognized speech segments

• US 5,890,120 A
• Filed: 05/20/1997
• Issued: 03/30/1999
• Est. Priority Date: 05/20/1997
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Haskell, Barin Geoffry, Swain, Cassandra Turner
• Primary Examiner(s): Hudspeth, David R.
• Assistant Examiner(s): Smits, Talivaldis Ivars
• Application Number:
  US08/858,966
• Time in Patent Office: 679 Days
• Field of Search: 704/235, 704/271
• US Class Current: 704/271
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00

Abstract

A synthesized signing image is generated which includes an original image of a subject and a sign image superimposed on the original subject image. The sign image is selected from a sign language database based on the speech (i.e., a letter or word) spoken by the subject. Based on the size and tone (intensity) of the subject'"'"'s face in the original image, the database sign image is modified to match the intensity and have a proportional size to the subject'"'"'s face. The modified database sign image is then superimposed on the original image of the subject to create a synthesized image in which the subject appears to be signing.

Claims

1. A method of generating a synthesized signing image comprising the steps of:

storing a sign language database including a sign image corresponding to each of a plurality of speech segments;

obtaining an original image of a subject;

identifying a characteristic of the subject in the original image;

receiving a speech segment spoken by the subject;

identifying one of said database sign images corresponding to the speech segment spoken by the subject;

modifying the identified sign image to match the characteristic of the subject in the original image; and

superimposing the modified sign image on the original image of the subject to generate a synthesized signing image.

2. The method of claim 1 wherein said step of storing comprises the steps of:

manually signing each of a plurality of sign images corresponding to speech segments;

capturing each signed image using a camera;

digitizing each captured sign image; and

storing each digitized sign image.

3. The method of claim 1 wherein said step of identifying a characteristic comprises the steps of:

segmenting the subject'"'"'s face in the original image; and

calculating a relative size of the subject'"'"'s face in the original image compared to the rest of the original image.

4. The method of claim 1 wherein said step of identifying a characteristic comprises the steps of:

segmenting the subject'"'"'s face in the original image; and

calculating an average intensity of the subject'"'"'s face in the original image.

5. The method of claim 1 wherein said step of identifying a characteristic comprises the steps of:

segmenting the subject'"'"'s face in the original image;

calculating a relative size of the subject'"'"'s face in the original image compared to the rest of the original image;

calculating a relative size of one or more hands in the identified sign image compared to the rest of the sign image; and

calculating a size modification factor indicating the amount which the hands in the identified sign image should be zoomed or dezoomed to be a predetermined proportion to the subject'"'"'s face in the original image, said size modification factor being calculated based on said relative sizes of the hands and the subject'"'"'s face.

6. The method of claim 1 wherein said step of identifying a characteristic comprises the steps of:

segmenting the subject'"'"'s face in the original image;

calculating an average intensity of the subject'"'"'s face in the original image;

calculating an average intensity of one or more hands in the identified sign image, said sign image comprising a plurality of pixels, each pixel having a corresponding intensity value; and

calculating, based on the identified sign image and the average intensities of the hands and the subject'"'"'s face, a pixel intensity adjustment for each pixel in the identified sign image.

7. The method of claim 1 wherein said step of identifying a characteristic comprises the step of identifying one or more characteristics selected from the group consisting of:

the average intensity of the subject'"'"'s face in the original image; and

the relative size of the subject'"'"'s face in the original image.

8. The method of claim 7 wherein said step of modifying the identified sign image comprises the step of modifying the average intensity of one or more hands in the identified sign image to approximately match the average intensity of the subject'"'"'s hands.

9. The method of claim 7 wherein said step of modifying the identified sign image comprises the step of modifying the size of one or more hands in the identified sign image to be proportional in size to the subject'"'"'s face.

10. The method of claim 1 wherein said step of identifying one of said database sign images comprises the steps of:

converting the speech segment spoken by the subject into text; and

identifying one of said database sign images corresponding to the text.

11. The method of claim 10 wherein:

said step of converting the speech segment comprises the step of identifying a letter spoken by the subject; and

said step of identifying one of said database sign images comprises the step of identifying a database sign image corresponding to the letter spoken by the subject.

12. The method of claim 10 wherein:

said step of converting the speech segment comprises the step of identifying one or more words spoken by the subject; and

said step of identifying one of said database sign images comprises the step of identifying a database sign image corresponding to the one or more words spoken by the subject.

13. A method of generating a synthesized signing image comprising the steps of:

storing a sign language database including a sign image corresponding to each of a plurality of speech segments, each said sign image comprising one or more signing hands;

obtaining an original image of a subject, said subject including a face;

calculating the relative size of the subject'"'"'s face in the original image;

calculating the relative size of the one or more hands in the sign image;

receiving a speech segment spoken by the subject;

identifying one of said database sign images corresponding to the speech segment spoken by the subject;

modifying the size of the hands in the identified sign image to be a predetermined proportion to the size of the subject'"'"'s face in the original image; and

superimposing the modified sign image on the original image of the subject to generate a synthesized signing image.

14. A method of generating a synthesized signing image comprising the steps of:

storing a sign language database including a sign image corresponding to each of a plurality of speech segments, each said sign image comprising one or more signing hands having a natural variation in intensity due to the shape of the hands;

obtaining an original image of a subject, said subject including a face;

calculating the average intensity of the subject'"'"'s face;

calculating the average intensity of the hands in the identified sign image;

receiving a speech segment spoken by the subject;

identifying one of said database sign images corresponding to the speech segment spoken by the subject;

modifying the intensity of the hands in the identified sign image to generally match the intensity of the subject'"'"'s face in the original image, while maintaining a variation in the intensity of the hands in the modified sign image which is similar to the variation in the intensity of the hands in the database sign image; and

superimposing the modified sign image on the original image of the subject to generate a synthesized signing image.

15. A method of generating a synthesized signing image comprising the steps of:

storing a sign language database at both a transmitting system and a receiving system, said sign language database including a sign image corresponding to each of a plurality of speech segments and a sign image code corresponding to each sign image;

performing the following steps at the transmitter system;

a) obtaining an original image of a subject;

receiving a speech segment spoken by the subject;

b) identifying one of said database sign images corresponding to the speech segment spoken by the subject;

c) transmitting the original image and the sign image code corresponding to the identified sign image;

performing the following steps at the receiving system;

d) receiving the transmitted original image and the sign image code corresponding to the identified sign image;

e) identifying a characteristic of the subject in the received original image;

f) modifying the sign image corresponding to the received sign image code to match the characteristic of the subject in the received original image; and

g) superimposing the modified sign image on the original image of the subject to generate a synthesized signing image.

16. A method of processing an original image to facilitate the generation of a synthesized signing image:

a) storing a sign language database including a sign image corresponding to each of a plurality of speech segments and a sign image code corresponding to each sign image;

b) obtaining an original image of a subject;

c) receiving a speech segment spoken by the subject, said speech segment corresponding to the original image;

d) identifying one of said database sign images corresponding to the speech segment spoken by the subject;

e) measuring one or more characteristics of the original image;

f) measuring one or more characteristics of the identified database sign image;

g) calculating, based on said measured characteristics of the original image and the sign image, one or more modification parameters indicating the amount which the sign image should be modified to approximately match the one or more measured characteristics of the original image; and

h) transmitting to a remote location a signal comprising the sign image code corresponding to the identified sign image, the one or more calculated modification parameters, and the original image, thereby to facilitate the generation of a synthesized signing image at the remote location.

17. The method of claim 16 wherein said steps b)-h) are performed for a plurality of original images, said method further comprising the step of determining for each original image whether one or more of the calculated modification parameters have changed as compared to the characteristics of other original images, and said transmitted signal including the calculated modification parameters only if the parameters have changed.

18. The method of claim 16 and further comprising the steps of:

repeating steps b)-e) for each of a plurality of subsequent original images;

comparing the one or more measured characteristics of one of the subsequent original images to the measured charactersitics of an earlier original image;

determining if the one or more characteristics of the subsequent original image have changed based on said step of comparing;

repeating steps f) and g) for the subsequent image if the measured characteristics have changed; and

transmitting a signal corresponding to one of the subsequent original images, said signal comprising a sign image code of one of the subsequent original images, and the subsequent original image, said signal corresponding to one of the subsequent original images further including the modification parameters only if the one or more characteristics of the subsequent original image have changed.

19. A method of generating a synthesized signing image comprising the steps of:

storing a sign language database including a sign image corresponding to each of a plurality of speech segments and a sign image code corresponding to each sign image;

receiving a signal including an original image of a subject and a sign image code corresponding to a database sign image, and one or more modification parameters;

modifying the database sign image corresponding to the received sign image code based on the received one or more modification parameters;

superimposing the modified sign image on the original image of the subject to generate a synthesized signing image.

20. An encoding system for encoding a signal to facilitate the generation of a synthesized signing image at a remote location, comprising:

a memory storing a sign language database including a sign image corresponding to each of a plurality of speech segments;

a video input device for capturing an original image of a subject;

an audio input device for receiving speech from the subject;

a speech analyzer coupled to said audio input device, said speech analyzer identifying the database sign image corresponding to a received speech segment;

a processor coupled to said memory, said speech analyzer and said video input device, said processor calculating a modification parameter, said modification parameter indicating how a database sign image should be modified to match a characteristic of an original image;

a data encoder coupled to said processor, said encoder encoding a signal including an identification of a database sign image corresponding to a received speech segment, a modification parameter for the sign image, and a corresponding original image for transmission over a transmission medium to a remote location.

21. A decoding system for generating a synthesized signing image based on a received signal, said decoding system comprising:

a memory storing a sign language database including a sign image corresponding to each of a plurality of speech segments;

a data decoder receiving a signal via a transmission medium, said signal comprising an identification of a database sign image, a modification parameter for the sign image, and a corresponding original image of a subject;

an image processor coupled to the memory and the data decoder, said image processor modifying the database sign image identified in the received signal, said sign image being modified based on said received modification parameter, said image processor superimposing the modified sign image on the received original image;

a display coupled to the image processor, said display displaying a synthesized signing image of the modified sign image superimposed on the original image.

22. An apparatus for generating a synthesized signing image comprising the steps of:

means for storing a sign language database including a sign image corresponding to each of a plurality of speech segments;

means for obtaining an original image of a subject;

identifying a characteristic of the subject in the original image;

means for receiving a speech segment spoken by the subject;

means for identifying one of said database sign images corresponding to the speech segment spoken by the subject;

means for modifying the identified sign image to match the characteristic of the subject in the original image; and

means for superimposing the modified sign image on the original image of the subject to generate a synthesized signing image.