Creating visual images of lip movements

• US 4,260,229 A
• Filed: 01/23/1978
• Issued: 04/07/1981
• Est. Priority Date: 01/23/1978
• Status: Expired due to Term

• Current Assignee: Richard W. Bloomstein
• Original Assignee: Richard W. Bloomstein
• Inventors: Bloomstein, Richard W.
• Primary Examiner(s): Hix, L. T.
• Assistant Examiner(s): Brady, W. J.
• Application Number:
  US05/871,538
• Time in Patent Office: 1,170 Days
• Field of Search: 352/87, 352/50, 340/324 A, 340/324 AD, 340/705, 340/725, 340/707, 358/82, 364/602, 364/855, 364/858
• US Class Current: 352/50
• CPC Class Codes: G03B 31/02 in which sound track is on a moving-picture film
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: H04N 19/20 using video object coding

Abstract

A system and method that creates visual images of lip movements on film, video tape, or other recorded media. Speech sounds are analyzed, digitally encoded and transmitted to a data memory device. Stored within the data memory device is a program for producing output data that creates visual images of lip movements corresponding to the speech sounds. Under control of the data for the speech sounds, the graphical output from the data memory device is sent to a graphic output device and related display equipment to produce the graphical display. This display may be combined with the speech sounds so that the resultant audio-visual composite (e.g. a film strip) contain lip movements corresponding to the speech sounds.

Claims

1. A method of graphically creating lip images which comprises providing a phoneme encoded representation of speech sounds, providing an encoded representation of selected lip configurations, transmitting both of said encoded representations to a data memory device for storage therein, and transmitting from said data memory device to a graphical output device and under control of a coded speech input and a coded selected lip configuration, both of which are transmitted to said data memory device, coded signals that cause said graphical display device to display lip images approximately corresponding to the uncoded form of said coded speech input.

2. A method according to claim 1 in which the coded speech input selects a phoneme output from the data memory device, which phoneme output most nearly represents the speech input.

3. A method of creating visual images of lip movements corresponding to speech sounds comprising providing a coded representation of speech sounds that are associated with lip movements from visual information, which lip movements do not correspond to the speech sounds, transmitting said coded representation to a data memory device for storage therein, storing in said data memory device coded data for creating a graphical representation of new lip movements corresponding to the speech sounds, and transmitting from said data memory device to a graphical output device and under control of the data representing the coded speech data for the new lip movements corresponding to said speech sounds.

4. A method according to claim 3 including extracting the coded speech sounds from a series of frames of audio and said visual information, storing said coded speech sounds in said data memory device on a frame-by-frame basis, optically scanning said series of frames in sequence to provide an encoded graphical image of the visual information and transmitting the encoded graphical image data to said data memory device, and transmitting from said data memory device to said graphical output device the visual information data and the data for the new lip movements that replace the first-mentioned lip movements.

5. A method according to claim 4 in which the frames that are scanned are images from frames on a strip of photographic film, and the speech sounds are on a sound track for the film.

6. A method of creating audio-visual media having visual images and associated speech sounds comprising providing a coded representation of speech sounds, transmitting said coded representation to a data memory device for storage therein, storing in said data memory device coded data for creating graphical representations corresponding to the speech sounds, transmitting from said data memory device to a graphical output device and under control of the data for the coded speech representation the data for the graphical representations corresponding to the speech sounds, and combining the graphical representations with visual images in a predetermined manner on an audio-visual recorded medium.

7. Apparatus for creating visual images of lip movements corresponding to speech sounds comprising means for providing a coded representation of speech sounds that are associated with lip movements from visual information, a data memory device, means for transmitting said coded representation to said data memory device for storage therein, said data memory device having stored therein coded data for creating a graphical representation of predetermined lip movements corresponding to the speech sounds, a graphical output device, and means for transmitting to said graphical output device and under control of the coded speech representation data, the data for the predetermined lip movements corresponding to the speech sounds.

8. Apparatus according to claim 7 including means for extracting the coded speech sounds from a series of frames of audio and said visual information, means for transmitting said coded speech sounds to said data memory device on a frame-by-frame basis for storge therein on said basis, and means for optically scanning said series of frames in sequence to provide an encoded graphical image of the visual information.