Methods and systems for synchronizing visualizations with audio streams

• US 7,596,582 B2
• Filed: 01/24/2005
• Issued: 09/29/2009
• Est. Priority Date: 03/26/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Harris, Geoffrey Howard, Novak, Michael J., Olson, Kipley J., Feller, Chris, Dideriksen, Tedd
• Primary Examiner(s): Le; Miranda
• Application Number:
  US11/041,441
• Publication Number:
  US 20050137861A1
• Time in Patent Office: 1,709 Days
• Field of Search: 707/104.1, 707/201, 709/203, 715/716
• US Class Current: 1/1
• CPC Class Codes: H04S 3/00 Systems employing more than two channels, e.g. quadraphonic H04S5/00, H04S7/00 take precedence
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users

Abstract

Methods and systems assist media players in rendering visualizations and synchronizing the visualizations with audio samples. In one example, a system includes one or more audio sources configured to provide audio samples that are to be rendered by a media player. An audio sample pre-processor is communicatively linked with the one or more audio sources and is configured to receive and to pre-process audio samples before the samples are rendered. The pre-processing provides characterizing data associated with each sample and derived from the audio samples. One or more effects are configured to receive the characterizing data and use the characterizing data to render the visualization that is synchronized with an audio sample that is being rendered by the media player.

Claims

1. A system for synchronizing a visualization with audio samples comprising:

a memory;

one or more audio sources configured to provide audio samples that are to be rendered by a media player;

an audio sample pre-processor coupled to the memory communicatively linked with the one or more audio sources and configured to receive and pre-process audio samples before the audio samples are rendered, the pre processing providing characterizing data associated with each sample, wherein the characterizing data is derived from the audio samples, the audio sample pre-processor comprises a Fast Fourier Transform to process the audio samples to provide frequency data associated with the audio samples;

the audio sample pre-processor to provide a timestamp for each audio sample, wherein each timestamp is maintained by a data structure associated with the audio sample, and wherein each timestamp is assigned based upon when the audio sample is calculated to be rendered by the media player;

an audio rendering object called by the media player to render visualizations corresponding to the audio samples, wherein the audio rendering object has an associated target frame rate defining how frequently the visualizations are drawn, redrawn, or painted; and

one or more effects associated with the audio rendering object, the one or more effects configured to receive the characterizing data and an instruction to render a visualization from the audio rendering object, the one or more effects rendering the visualization using the characterizing data, wherein the visualization is synchronized with an audio sample that is being rendered by the media player and the rendering corresponds to the associated target frame rate.

2. The system of claim 1, wherein the audio sample pre-processor is configured to maintain data structures associated with each audio sample.

3. The system of claim 1, wherein said characterizing data comprises frequency data.

4. A media player comprising:

a memory;

an audio sample pre-processor coupled to the memory configured to receive and to pre-process audio samples before the audio samples are rendered by the media player, the pre-processing providing frequency data associated with each audio sample, wherein the frequency data is derived from the audio samples, the audio sample pre-processor pre-processes the audio samples by using a Fast Fourier Transform to provide the frequency data associated with the audio samples;

the audio sample pre-processor configured to provide a timestamp for each audio sample, wherein each timestamp is maintained by a data structure associated with an audio sample, and wherein each timestamp is assigned based upon when the audio samples is calculated to be rendered by the media player;

an audio rendering object called by the media player to render visualizations corresponding to the audio samples, wherein the audio rendering abject has an associated target frame rate defining how frequently the visualizations are drawn, redrawn, or painted; and

one or more effects associated with the audio rendering object, the one or more effects configured to receive the frequency data and an instruction to render a visualization from the audio rendering object, the one or more effects rendering the visualization using the frequency data, wherein the visualization is synchronized with an audio sample that is being rendered by the media player and the rendering corresponds to the associated target frame rate.

5. A system for synchronizing a visualization with audio samples comprising:

a memory;

an audio sample pre-processor coupled to the memory configured to receive and to preprocess audio samples before the audio samples are rendered by a renderer that comprises part of a media player to a speaker, the audio sample pre-processor comprising at least one of a timestamp module, a spectrum analyzer module, a waveform analysis module, or a stream state module, for preprocessing the audio samples to provide characterizing data derived from each audio sample, the characterizing data comprising a timestamp associated with each audio sample, the timestamp being assigned when the audio sample is calculated to be rendered by the renderer;

the audio sample pre-processor to preprocess the audio samples by using a Fast Fourier Transform to provide frequency data associated with the audio samples;

multiple data structures configured to hold the characterizing data, each data structure being associated with an audio sample;

an audio rendering object called by the media player to render visualizations corresponding to the audio samples, wherein the audio rendering object has an associated target frame rate defining how frequently the visualizations are drawn, redrawn, or painted, wherein the audio rendering object is configured to receive a draw or a paint call from the media player;

the audio rendering object configured to call the audio sample pre-processor after receiving the draw or the paint call to ascertain the characterizing data associated with an audio sample that is currently being rendered by the renderer;

the audio sample pre-processor being configured to ascertain the characterizing data by querying the renderer for a time associated with the currently-rendered audio sample, and then using the time associated with the currently rendered audio sample to identify a data structure having a timestamp that is nearest in value to the time associated with the currently rendered audio sample; and

one or more effects associated with the audio rendering object, the one or more effects configured to receive the characterizing data that is associated with the data structure having the timestamp that is nearest in value to the time and an instruction to render the visualization from the audio rendering object, the one or more effects rendering the visualization using the characterizing data, wherein the visualization is synchronized with the audio sample that is being rendered by the renderer and the rendering corresponds to the associated target frame rate.

6. The system of claim 5, wherein the characterizing data comprises frequency data.

7. The system of claim 5, wherein the visualization is rendered in a rendering area in which other media types can be rendered.

8. The system of claim 7, wherein the other media types comprise a video type.

9. The system of claim 7, wherein the other media types comprise a skin type.

10. The system of claim 7, wherein the other media types comprise a HTML type.

11. The system of claim 7, wherein the other media types comprise an animation type.