Speech synchronized animation

• US 4,884,972 A
• Filed: 11/26/1986
• Issued: 12/05/1989
• Est. Priority Date: 11/26/1986
• Status: Expired due to Term

• Current Assignee: Sierra Entertainment, Inc. (Vivendi SE), Sierra On-Line, Incorporated (Vivendi SE)
• Original Assignee: Brightmail, Inc. (Gen Digital Inc.)
• Inventors: Gasper, Elon
• Primary Examiner(s): Picard, Leo P.
• Application Number:
  US06/935,298
• Time in Patent Office: 1,105 Days
• Field of Search: 434/167, 434/169, 434/172, 434/185
• US Class Current: 434/185
• CPC Class Codes: G09B 19/06 Foreign languages with audible presentation of material to be studied G09B5/04
• CPC Class Codes: G09B 5/065 Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

A system for providing randomly-accessible animated objects having synchronous audio and visual features suitable for use as an education system or video entertainment system is provided. The speech sounds and the associated visual and behavioral characteristics of language are broken into their constituent parts to produce encoded communication patterns and characteristics associated with speech. In response to coded input signal representing speech sounds or combinations of a speechsounds forming words, a coded model is constructed including a voice and animated visual image having synchronized audio and associated visual characteristics to speak the sound or words which were input. A user is presented with a viewing screen having several rows of what are perceived to be domino-like tiles each inscribed with one or more letters. An animated character including a talking head is also displayed on the screen. The user selects tiles with a mouse or other input device and moves the tiles about the screen to form words or other combinations of letters. As a tile is selected, the animated character pronounces the name and sound of the letter inscribed on the tile. When words are formed, the animated character pronounces the words or the sound of the character combinations formed.

Claims

1. Apparatus for producing and displaying randomly-accessible animated objects having synchronized audio and visual features, said apparatus comprising:

a program-controlled microprocessor;

integrator means, coupled to the microprocessor, for producing signals representing encoded elements of sound and encoded elements of constituent object parts in response to command signals from the microprocessor;

sound emitting means, coupled to the microprocessor and to the integrator means for producing sound in response to the signals representing encoded elements of sound; and

display means coupled to the microprocessor, to the integrator means and sound emitting means for producing visual images of the animated objects in response to the signals representing encoded elements of constituent object parts, the visual images of the animated objects being synchronized with the sound.

2. Apparatus as in claim 1 further comprising input means coupled to the microprocessor for inputting program data to the microprocessor, said program data including audio and visual characteristics associated with a selected animated object.

3. Apparatus as in claim 2 wherein said animated objects include synthesized actors representative of real or imaginary persons, each synthesized actor being associated with a predetermined set of text-to-phonetic translation rules, said display means responsive to said signals representing encoded elements of constituent object parts for producing visual images of said synthesized actors lip-synchronized with the sound produced by said sound emitting means.

4. Apparatus as in claim 3 wherein said sound emitting means includes at least one narrator device, each narrator device responsive to a predetermined set of phonetic codes designating sound elements in a predetermined manner, said narrator device responsive to said signals representing encoded elements of sound and to said associated predetermined set of text-to-phonetic translation rules to provide the voice for an associated synthesized actor, each narrator device associated with at least one synthesized actor.

5. Apparatus as in claim 4 wherein said narrator device comprises audio processor means, a speech synthesizer and an audio output device.

6. Apparatus as in claim 1 wherein said integrator means includes translator means for translating a symbol set associated with said elements of sound into a phonetic representation of said symbol set and into a coded bit string representing orthophonetic characteristics corresponding to said elements of sound.

7. Apparatus as in claim 1 further comprising random access memory means for storing data representing said elements of sound, for storing data representing behavior traits associated with said elements of sound and for storing data representing visual images of said constituent object parts, the microprocessor and the display means responsive to said signals to access said data for generating anthropomorphic objects having behavior traits associated with said elements of sound.

8. Apparatus as in claim 7 wherein said data representing visual images of said constituent object parts includes data representating behavior traits of said animated objects not associated with speech.

9. Apparatus as in claim 7 wherein said data representing behavior traits associated with said elements of sound include data representative of articulations of lip and face parts associated with speech, said display means and said sound emitting means responsive to said data for generating said anthropomorphic objects having lip-synchronized speech.

10. Apparatus as in claim 9 wherein said animated objects include synthesized actors representative of real or imaginary persons, said display means and said sound emitting means responsive to said data for generating said synthesized actors having lip-synchronized speech.

11. Apparatus as in claim 1 further comprising:

animation means coupled to the integrator means, to the sound emitting means and to the display means, said animation means responsive to said signals for generating an encoded model of the animated object, the display means responsive to said encoded model to display the visual images of the animated objects.

12. Apparatus as in claim 11 wherein said animation means comprises:

first animation means responsive to said signals for generating a first digitized encoded model defining the characteristics of anthropomorphic objects; and

second animation means responsive to said signals for generating a second digitized encoded model defining the characteristics of physical objects.

13. Apparatus as in claim 12 wherein said anthropomorphic objects comprise animated cartoon characters having a person-like appearance.

14. Apparatus as in claim 12 wherein said physical objects comprise a plurality of domino-like tiles, each of said tiles having at least one symbol inscribed thereon, each of said symbols associated with at least one different speech sound.

15. Apparatus as in claim 12 wherein said first animation means is responsive to said signals for generating a second digitized encoded model defining the characteristics of a synthesized actor representative of a real or of an imaginary person.

16. Apparatus as in claim 15 wherein said display means includes time coordinating means coupled to said sound emitting means and to said first and second animation means for providing near synchronization between the display of visual images of said animated objects and said sound.

17. Apparatus as in claim 16 wherein said sound emitting means includes audio processor means for providing real time feedback to said time coordinating means responsive to said real time feedback for providing real time synchronization between said sound and the display of visual images of said animated objects.

19. The method of claim 8 including the step of generating a third set of signals defining the time coordination protocol between said synthesized voice and said images of animated objects for producing said synchronized display of said images of animated objects synchronized with said elements of sound.

20. The method of claim 19 wherein said third set of signals defines asynchronous time coordination between said synthesized voice and said images of animated objects for providing said synchronized display of said images of animated objects.

21. The method of claim 19 wherein the step of generating said third set of signals includes the step of generating a real time feedback signal for defining a synchronous time coordination between said synthesized voice and said images of animated objects synchronizing in real time the display of said images of animated objects with said synthesized voice.

18. A method for producing an displaying randomly-accessible animated objects having synchronized audio and visual features, said method comprising the steps of:

generating a set of first signals representing encoded elements of sound and encoded elements of constituent object parts;

generating a set of second signals in response to said set of first signals, said set of second signals representing images of animated objects composed of said constituent object parts;

electronically synthesizing a voice in response to said set of first signals pronouncing said elements of sound represented by said set of first signals; and

generating a video output and displaying the images of said animated objects represented by the said set of second signals, the display of said images of animated objects synchronized with said synthesized elements of sound.

22. A method of instruction for teaching language skills or the like comprising the steps of:

displaying a plurality of first animated images on a screen, each of said first animated images having at least one of a plurality of graphic symbols inscribed thereon, each of said plurality of symbols associated with speech sound;

receiving an input signal and selecting at least one of said first animated images in response to said input signal;

generating electronically a voice sound pronouncing the associated speech sound of the symbols inscribed on said selected one of said first animated images; and

displaying a second animated image on said screen, said second animated image including a talking head having facial features, said second animated image responsive to said input signal for displaying an animated sequence of body movements including movements of head and facial features simulating the speaking of said associated speech sound in synchrony with said voice sound.

23. The method of claim 22 wherein the step of receiving an input signal and selecting at least one of said first animated images includes the step of highlighting said selected first animated image to distinguish and emphasize said selected first animated image.

24. The method of claim 23 wherein the step of highlighting said selected first animated images includes accenting said symbol inscribed thereon simultaneously with generating said voice sound.

25. The method of claim 22 including the steps of moving said selected first animated image from a first location to a second location on said screen;

and

selecting at least one additional first animated image in response to at least one additional input signal and moving said additional selected first animated image to a third location adjacent said second location to form a combination of said symbols.

26. The method of claim 25 further including the steps of:

highlighting each of said selected first animated images simultaneously with the selection of each first animated images;

forming one first animated image at said second location by combining each selected additional first animated image with said first animated image at said second location when it is moved to said third location, said one first animated image having the symbols of each selected first animated image inscribed thereon;

pronouncing the associated speech sound in sequence of each of said symbols inscribed on said one first animated image and

accenting each symbol simultaneously with the pronunciation of its associated speech sounds.

27. The method of claim 22 wherein the step of displaying a second animated image on said screen includes displaying said second animated image including animated hands for forming sign language hand symbols representing the associated speech sound of the graphic symbols inscribed on said selected one of said first animated images.

28. A system for producing animated orthographic objects in combination with at least one animated character having synchronized audio and visual characteristics associated with the orthographic objects, said system comprising:

a program-controlled microprocessor;

integrator means coupled to the microprocessor, responsive to command signals from the microprocessor for producing first signals representing encoded elements of sound and second signals representing encoded rules mapping said elements of sound to associated constituent object parts and behavior traits;

first animation means coupled to the integrator means and to the microprocessor, responsive to said first signals for generating a first encoded model representative of said animated orthographic object;

second animation means coupled to the integrator means and to the microprocessor responsive to said second signals for generating a second encoded model representative of said animated character;

sound emitting means, coupled to the microprocessor and to the integrator means for producing sound in response to said first signals representing encoded elements of sound; and

display means coupled to the microprocessor and to the first and second animation means, said display means responsive to said first encoded model for producing and displaying visual images of said animated orthographic object, said display means responsive to said second encoded model for producing and displaying visual images of said animated character in synchrony with the sound.

29. A system as in claim 28 further comprising positioning means coupled to said display means responsive to input signals for repositioning said animated orthographic objects from one position to another position in said display.

30. A system as in claim 29 wherein said animated orthographic objects include a plurality of anagram images, each of said anagram images having at least one of a plurality of alphanumeric symbols displaced thereon, said positioning means responsive to said input signals for arranging at least one of said anagram images to selectively form desired combinations of at least one of said alphanumeric symbols, said sound emitting means responsive to said first signals representing encoded elements of sound producing sound associated with said desired combination of alphanumeric symbols, said display means responsive to said second encoded model for producing and displaying visual images of said animated characters in synchrony with said associated sound.

31. A system as in claim 30 further including phonic means coupled to said integrator means and to said sound emitting means responsive to said first signals for dividing said desired combination of alphanumeric symbols into its constituent phonic parts and providing a third set of signals respresentative of said constituent phonic parts, said sound emitting means responsive to said third set of signals for producing individual sounds associated with said constituent phonic parts, and said display means responsive to said third set of signals for highlighting the anagram image displaying the alphanumeric symbol or combination of alphanumeric symbols associated with an individual constituent phonic part synchronous with said associated sound.

32. A system as in claim 31 further including means responsive to said third set of signals for integrating said sounds associated with said individual constituent phonic parts to provide a smooth and continuous pronunciation of said desired combination of alphanumeric symbols, said display means responsive to said set of signals for successively highlighting each displayed alphanumeric symbol during that portion of said associated sound pronunciation in which its individual associated sound is most prominent.

33. A system as in claim 31 further including means responsive to said third set of signals for generating a fourth set of signals, said display means responsive to said fourth set of signals for selectively highlight adjacent displayed alphanumeric symbols which have an influence on the associated individual sound of an individual alphanumeric symbol or group of alphanumeric symbols when it is pronounced in combination with the sounds associated with said desired combination of alphanumeric symbols.