Script control for lip animation in a scene generated by a computer rendering engine

• US 8,674,996 B2
• Filed: 10/27/2008
• Issued: 03/18/2014
• Est. Priority Date: 05/21/1999
• Status: Expired due to Fees

• Current Assignee: DigiMedia Tech, LLC (IP Investments Group LLC)
• Original Assignee: Quonsil PL 3 LLC
• Inventors: Kulas, Charles J.
• Primary Examiner(s): LIN, SHEW FEN
• Application Number:
  US12/259,292
• Publication Number:
  US 20090184967A1
• Time in Patent Office: 1,968 Days
• Field of Search: 345/467, 345/473, 704/275, 704/276, 704/231, 704/235, 715/706, 715/716
• US Class Current: 345/473
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: G06T 2210/08 Bandwidth reduction
• CPC Class Codes: G06T 2213/04 Animation description language
• CPC Class Codes: G06T 2213/08 Animation software package
• CPC Class Codes: G06T 2213/12 Rule based animation

Abstract

A system for controlling a rendering engine by using specialized commands. The commands are used to generate a production, such as a television show, at an end-user'"'"'s computer that executes the rendering engine. In one embodiment, the commands are sent over a network, such as the Internet, to achieve broadcasts of video programs at very high compression and efficiency. Commands for setting and moving camera viewpoints, animating characters, and defining or controlling scenes and sounds are described. At a fine level of control math models and coordinate systems can be used make specifications. At a coarse level of control the command language approaches the text format traditionally used in television or movie scripts. Simple names for objects within a scene are used to identify items, directions and paths. Commands are further simplified by having the rendering engine use defaults when specifications are left out. For example, when a camera direction is not specified, the system assumes that the viewpoint is to be the current action area. The system provides a hierarchy of detail levels. Movement commands can be defaulted or specified. Synchronized speech can be specified as digital audio or as text which is used to synthesize the speech.

Claims

1. A method for controlling a rendering engine to produce audio, the method comprising:

receiving commands at a user computing device specifying one or more character actions for a computer-modeled character displayed on a display device of the user computing device,

wherein the commands are from a content provider at a first location and the user computing device is at a second location remote from the first location,

wherein the commands comprise predetermined instructions in a control script for the one or more character actions and an electronic representation of human speech, and

wherein one or more of the predetermined instructions include a specified duration; and

animating lip movement of the computer-modeled character displayed on the display device of the user computing device at the second location,

wherein the animating is performed according to the predetermined instructions in the control script received from the content provider and uses one or more pre-computed lip movements associated with selected phonetic sounds,

wherein the animating comprises synchronizing an audio representation of a human speech with the animated lip movement of the computer modeled character, and

wherein the synchronizing is performed by maintaining a constant playback rate by indicating a start time and duration time corresponding to a set of one or more of the predetermined instructions in the control script such that each resulting animation that comprises a pre-computed lip movement fits into the corresponding specified duration.

2. The method of claim 1 wherein the electronic representation of human speech in the control script comprises an audio waveform.

3. The method of claim 1 wherein receiving commands at the user computing device further comprises receiving animation information in the control script for animating lip movement of the computer-modeled character in accordance with the electronic representation of human speech.

4. The method of claim 3 wherein receiving animation information for animating lip movement of the computer-modeled character comprises receiving digital video.

5. The method of claim 1 wherein the electronic representation of human speech in the control script comprises text representation of the speech, and wherein the method further comprises:

performing speech synthesis at the user computing device in response to the predetermined instructions in the control script.

6. The method of claim 1 wherein the electronic representation of human speech in the control script comprises speech described by text, and wherein:

animating lip movement of the computer-modeled character displayed on the display device at the user computing device comprises automatically animating lips associated with the computer-modeled character in response to the text-described speech.

7. The method of claim 6 wherein automatically animating lips associated with the computer-modeled character comprises:

using a sound analysis routine at the user computing device at the second location to identify a phonetic sound represented by the text; and

determining a lip animation based on the identified phonetic sound.

8. The method of claim 1 wherein default character actions are automatically specified at the user computing device if omitted by the control script, and wherein animating lip movement of the computer-modeled character at the second location comprises using frames of a lip animation from the user computing device at the second location.

9. The method of claim 1 wherein the rendering engine is located on the user computing device at the second location, and wherein animating lip movement of the computer-modeled character displayed on the display device at the user computing device comprises animating lip movement of the computer-modeled character based, at least in part, on default actions stored in a memory of the user computing device at the second location.

10. The method of claim 1, further comprising animating the lip movement of the computer-modeled character at one or more third locations remote from the first and second locations contemporaneously with animating the lip movement of the computer-modeled character at the second location.

11. The method of claim 1 wherein receiving commands at the user computing device from the content provider at the first location comprises receiving predetermined instructions in the control script created by a human author.

12. The method of claim 1 wherein receiving commands at the user computing device from the content provider at the first location comprises receiving predetermined instructions in the control script created automatically without input from a human author.

13. The method of claim 1 wherein, when the lips of the computer-modeled character are visible in a particular frame, the method further comprises using a speech analysis process at the user computing device to interpret a speech track currently being played at the user computing device and trigger lip movements of the computer-modeled character in accordance with the speech track.

14. The method of claim 1 wherein the electronic representation of human speech in the control script comprises text-based commands.

15. A computer-readable storage medium having stored thereon computer-executable instructions that, if executed by a computing device, cause the computing device to perform operations for controlling a rendering engine, the operations comprising:

receiving commands at a computer specifying one or more character actions for a computer-modeled character displayed at the computer,

wherein the commands are from a content provider at a first location and the computer is at a second location remote from the first location,

wherein the commands comprise predetermined instructions in a control script for the one or more character actions and an electronic representation of human speech; and

wherein one or more of the predetermined instructions include a specified duration; and

animating lip movement of the computer-modeled character displayed at the computer at the second location,

wherein the animating is performed according to the predetermined instructions in the control script,

wherein the animating comprises synchronizing an audio representation of a human speech with the animated lip movement of the computer modeled character, and

wherein the synchronizing is performed by maintaining a constant playback rate by indicating a start time and duration time corresponding to a set of one or more of the predetermined instructions in the control script such that each resulting animation that comprises a pre-computed lip movement fits into the corresponding specified duration.

16. The computer-readable storage medium of claim 15 wherein default character actions are automatically specified by the rendering engine at the computer at the second location if omitted by the control script, and wherein animating lip movement of the computer-modeled character comprises animating lip movement of the computer-modeled character based, at least in part, on default actions from the rendering engine of the computer at the second location.

17. The computer-readable storage medium of claim 15 wherein the electronic representation of human speech in the control script comprises speech specified by text, and wherein:

animating lip movement of the computer-modeled character comprises automatically animating lips associated with the computer-modeled character via the rendering engine to achieve the text-specified speech.

18. The computer-readable storage medium of claim 15 wherein the electronic representation of human speech in the control script comprises speech specified by text, and wherein the operations further comprise:

using a sound analysis routine at the computer device at the second location to identify a phonetic sound represented by the text; and

determining a lip animation for the computer-modeled character based on the identified phonetic sound.

19. The computer-readable storage medium of claim 15 wherein the electronic representation of human speech in the control script comprises speech specified by text, and wherein the operations further comprise:

performing speech synthesis at the second location in response to the control script, and wherein the speech synthesis comprises specifying tone, pitch, accent, and/or emotional intensity for animation of the lip movement of the computer-modeled character to achieve the text-specified speech.

20. A method, comprising:

transmitting commands from a content provider computer at a first location to a digital computing device at a second location remote from the first location,

wherein the commands are predetermined instructions in a control script specifying one or more character actions and speech specified by text for a computer-modeled character displayed on the digital computing device, and wherein one or more of the predetermined instructions include a specified duration,

wherein the digital computing device is configured to perform speech synthesis at the second location in response to the speech specified by text in the control script,

wherein the digital computing device is further configured to animate lip movement of the computer-modeled character based, at least in part, on the predetermined instructions in the control script and the speech synthesis, the animating lip movement of the computer-modeled character comprising—

for the predetermined instructions including specified durations, synchronizing audio representation of a human speech with the animated lip movement of the computer-modeled character such that a constant playback rate is maintained by indicating a start time and duration time such that each resulting animation fits into the corresponding specified duration; and

wherein the speech synthesis comprises specifying tone, pitch, accent, and/or emotional intensity for animation of the lip movement of the computer-modeled character to achieve the text-specified speech.