Audio-driven visual representations

• US 6,369,822 B1
• Filed: 08/12/1999
• Issued: 04/09/2002
• Est. Priority Date: 08/12/1999
• Status: Expired due to Term

• Current Assignee: Creative Technology Ltd.
• Original Assignee: Creative Technology Ltd.
• Inventors: Seefeldt, Alan, Peevers, Alan
• Primary Examiner(s): Vo, Cliff N.
• Application Number:
  US09/373,405
• Time in Patent Office: 971 Days
• Field of Search: 345/418, 345/420, 345/421, 345/422, 345/427, 345/473, 345/474, 345/475
• US Class Current: 345/473
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

A method and system to drive transformations of a visual representation, in real-time, that synchronizes the audio and visual outputs and controls the magnitude of object deformation to be visual pleasing. In one embodiment, sudden increases in the spectral energy are detected to time the initialization of deformations and a smoothed signal is derived from the time varying spectral energy curve to control the magnitude of the deformations.

Claims

1. A method of driving a computer generated animation with an audio signal, with the animation comprising a number of frames rendered sequentially at a frame rate, with each frame for displaying at least a first object, the method comprising the steps of:

for a current frame, capturing a frame'"'"'s worth of said audio signal to obtain a captured segment of the audio signal and extracting an energy characteristic of the captured segment of the audio signal;

utilizing the energy characteristic of the captured segment to generate a conditioned control signal having an amplitude which responds rapidly to a rapid increase in magnitude of energy characteristic;

when the conditioned control signal amplitude increases rapidly in magnitude, initiating a deformation of the first object to be rendered in the current frame; and

delaying the conversion of the captured segment of the audio signal, corresponding to the current frame, into a sound signal until the current frame is rendered and displayed to synchronize the occurrence of increase in magnitude of the energy characteristic to the display of the initiation of the deformation.

2. The method as recited in claim 1 wherein said steps of extracting an energy characteristic of the captured segment of the audio signal further and generating a conditioned control signal further comprises the steps of:

performing a series of half-overlapping discrete Fourier transforms (DFTs) on the captured segment;

determining a maximum energy signal for the captured signal based on the frequency bins of one of the DFTs; and

normalizing the maximum energy signal from different captured segments to form a normalized energy signal as a control signal.

3. The method as recited in claim 1 further including the steps of:

breaking said audio signal into a plurality of sub-bands;

calculating a conditioned control signal for each sub-band; and

utilizing the conditioned control signals to control different deformations based on energy characteristics of the different sub-bands.

4. The method of claims 1 further comprising the step of:

compressing the amplitude of the conditioned control signal into a fixed dynamic range so that the amplitude of peaks of the conditioned control signal is independent of average energy characteristic amplitude.

5. A method for audio-driving an animation sequence to deform an object displayed by the sequence, with the animation sequence comprising a series of frames displayed sequentially and separated in time by a frame period, with the method comprising the following steps performed during a frame period:

storing a segment of an audio driving signal corresponding to the frame period;

while delaying the playback of the segment of the audio driving signal;

processing said segment of said audio driving signal to extract a maximum energy characteristic of the segment;

filtering and normalizing said maximum energy characteristic to generate a smoothed conditioned control signal with an amplitude that increases rapidly when the maximum energy characteristic of the audio signal increases rapidly and decreases slowly from its maximum regardless of whether the energy then decreases rapidly;

based on the amplitude of the conditioned control signal, calculating the magnitude of a shape deformation having a deformation magnitude modulated by the amplitude of the smoothed conditioned control signal; and

rendering the frame with a deformation magnitude of a deformation of the first object modulated by the smoothed conditioned control signal amplitude;

subsequent to rendering, displaying a rendered frame and converting the segment of the audio signal to sound so that a displayed frame is synchronized with the audio driving signal.

6. A system comprising:

a processor;

a buffer, in data communication with the processor, the buffer holding digital data representing an audio signal; and

a memory, in data communication with both the buffer and the processor, the memory storing a program to be executed by the processor, the program including information corresponding to an object, a first process for capturing a time segment of the audio signal, extracting an energy characteristic, indicating a magnitude of signal energy of the time segment, therefrom, and utilizing said energy characteristic to form a smoothed conditioned control signal having an amplitude which increases rapidly when the signal energy magnitude increases rapidly and the smoothed conditioned control signal having an amplitude which decreases slowly when the magnitude of the frequency energy decreases rapidly; and

a second process to vary object deformation function amplitudes associated with the object in response to the amplitude of the smoothed conditioned control signal while capturing an additional time segment of the audio signal, thereby defining a modified representation.

7. The system as recited in claim 6 wherein the object is stored in the memory as a plurality of bins, a first subset of which defines an array of vertices and a second subset of which defines a plurality of normals to the object.

8. The system as recited in claim 6 further including a visual display, in data communication with both the memory and the processor to produce a visual image of the modified representation, an audio generating device connected to a playback buffer wherein the buffer includes an output coupled to transmit audio information to both the memory and the playback buffer, with the audio generating device being adapted to generate sounds in response to signals received from the playback buffer and the playback buffer having a delay associated therewith sufficient to synchronize the visual image with the time segment.

9. The system of claim 6 wherein said second process further:

utilizes the energy characteristic of the captured segment to generate a conditioned control signal having an amplitude which responds rapidly to a rapid increase in magnitude of energy characteristic; and

when the conditioned control signal amplitude increases rapidly in magnitude, initiates a deformation of the first object to be rendered in the current frame.

10. The system of claim 7 further comprising:

a user input device;

and wherein said second process responds to said user input device to implement user controlled mode switching during rendering of the visual representation.

11. A computer program product that controls a computer to vary representations in accordance with an audio signal, comprising:

for a current frame, code for capturing a frame'"'"'s worth of said audio signal to obtain a captured segment of the audio signal and extracting an energy characteristic of the captured segment of the audio signal;

code for utilizing the energy characteristics of the captured segment to generate a conditioned control signal which responds rapidly to a rapid increase in magnitude of energy characteristic;

code for initiating a deformation of a first object to be rendered in the current frame when the conditioned control signal increases rapidly in magnitude;

code for delaying the conversion of the captured segment of the audio signal, corresponding to the current frame, into a sound signal until the current frame is rendered and displayed to synchronize the occurrence of increase in magnitude of the energy characteristic to the visual initiation of the deformation.

12. The computer program product as recited in claim 11 wherein the code to extract characteristics of the audio signal includes code to divide the signal into a plurality of audio streams.

13. The computer program product recited in claim 11 further comprising:

code for filtering and normalizing said energy characteristic to generate a smoothed conditioned control signal with an amplitude that increases rapidly when the maximum energy characteristic of the audio signal increases rapidly and decreases slowly from its maximum regardless of whether the energy then decreases rapidly;

code for calculating the magnitude of a shape deformation having a deformation magnitude modulated by the amplitude of the smoothed conditioned control signal based on the amplitude of the smoothed conditioned control signal; and

code for rendering the frame with a deformation magnitude modulated by the smoothed conditioned control signal amplitude.

14. A method of driving a computer generated animation with an audio signal, with the animation comprising a number of frames rendered sequentially at a frame rate, with each frame for displaying at least a first object, the method comprising the steps of:

for a current frame, capturing a frame'"'"'s worth of said audio signal to obtain a captured segment of the audio signal and extracting an energy characteristic of the captured segment of the audio signal;

utilizing the energy characteristic of the captured segment to generate a first conditioned control signal which responds rapidly to a rapid increase in magnitude of energy characteristic;

when the conditioned control signal increases rapidly in magnitude, initiating a deformation of the first object to be rendered in the current frame;

filtering and normalizing said energy characteristic to generate a smoothed conditioned control signal with an amplitude that increases rapidly when the maximum energy characteristic of the audio signal increases rapidly and decreases slowly from its maximum regardless of whether the energy then decreases rapidly;

based on the amplitude of the smoothed conditioned control signal, calculating the magnitude a shape deformation having a deformation magnitude modulated by the amplitude of the smoothed conditioned control signal;

rendering the frame with the firs object deformed according to a deformation magnitude modulated by the smoothed conditioned control signal amplitude; and

delaying the conversion of the captured segment of the audio signal, corresponding to the current frame, into a sound signal until the current frame is rendered and displayed to synchronize the occurrence of increase in magnitude of the energy characteristic to the display of the initiation of the deformation.

15. The method of claim 14 further comprising the step of:

responding to user input to implement user controlled mode switching during rendering of the current frame.

16. A method of driving a computer generated animation with an audio signal, with the animation comprising a number of frames rendered sequentially at a frame rate, with each frame for displaying at least a first object, the method comprising the steps of:

for a current frame, capturing a frame'"'"'s worth of said audio signal to obtain a captured segment of the audio signal and extracting an energy characteristic of the captured segment of the audio signal;

utilizing the energy characteristic of the captured segment to generate a conditioned control signal having an amplitude which responds rapidly to a rapid increase in magnitude of energy characteristic;

providing a function generator for providing function generator output signal amplitude;

when the conditioned control signal increases rapidly in magnitude, triggering an animation function control signal having an amplitude equal to the amplitude of the conditioned control signal when the conditioned control signal is increasing in magnitude;

setting the animation function control signal amplitude to the function generator output amplitude when the conditioned control signal amplitude begins decreasing; and

utilizing the animation function control signal to initiate and control characteristics of a deformation to the first object.

17. The method of claim 16 further comprising the steps of:

selecting a minimum time period;

if the conditioned control signal, subsequent to decreasing, suddenly increases in magnitude before expiration of the minimum time period, setting the amplitude of a current animation function control signal equal to the amplitude of the conditioned control signal;

if the conditioned control signal, subsequent to decreasing, suddenly increases in magnitude after expiration of the minimum time period, triggering a new animation function control signal and setting the amplitude of the new animation function control signal equal to the amplitude of the conditioned control signal.

18. A system of driving a computer generated animation with an audio signal, with the animation comprising a number of frames rendered sequentially at a frame rate, with each frame for displaying at least a first object, the method comprising the steps of:

means for capturing a current frame'"'"'s worth of said audio signal to obtain a captured segment of the audio signal and extracting an energy characteristic of the captured segment of the audio signal;

means, coupled to receive the captured segment, for utilizing the energy characteristic of the captured segment to generate a first conditioned control signal which responds rapidly to a rapid increase in magnitude of an energy characteristic;

means, responsive to the first conditioned control signal, for initiating a deformation of the first object to be rendered in the current frame when the conditioned control signal increases rapidly in magnitude;

means for filtering and normalizing said energy characteristic to generate a smoothed conditioned control signal with an amplitude that increases rapidly when the maximum energy characteristic of the audio signal increases rapidly and decreases slowly from its maximum regardless of whether the energy then decreases rapidly;

means for calculating the magnitude a shape deformation having a deformation magnitude modulated by the amplitude of the smoothed conditioned control signal based on the amplitude of the smoothed conditioned control signal;

means for rendering the frame with the first object deformed according to a deformation magnitude modulated by the smoothed conditioned control signal amplitude; and

means for delaying the conversion of the captured segment of the audio signal, corresponding to the current frame, into a sound signal until the current frame is rendered and displayed to synchronize the occurrence of increase in magnitude of the energy characteristic to the display of the initiation of the deformation.