Methods and systems for synchronizing visualizations with audio streams

• US 7,620,656 B2
• Filed: 01/24/2005
• Issued: 11/17/2009
• Est. Priority Date: 03/26/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Novak, Michael J., Olson, Kipley J., Harris, Geoffrey Howard, Feller, Chris, Dideriksen, Tedd
• Primary Examiner(s): Le; Miranda
• Application Number:
  US11/041,444
• Publication Number:
  US 20050188012A1
• Time in Patent Office: 1,758 Days
• Field of Search: 707/1, 707/100, 707/104.1, 707200-201, 709/231, 709/203, 725/37, 725/91, 715/716, 715/203
• US Class Current: 1/1
• CPC Class Codes: H04S 3/00 Systems employing more than two channels, e.g. quadraphonic H04S5/00, H04S7/00 take precedence
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users

Abstract

Methods and systems provide a tool for assisting media players in rendering visualizations and synchronizing those visualizations with audio samples. In one embodiment, visualizations are synchronized with an audio stream using a technique that builds and maintains various data structures. Each data structure can maintain data that is associated with a particular pre-processed audio sample. The maintained data can include a timestamp that is associated with a time when the audio sample is to be rendered. The maintained data can also include various characteristic data that is associated with the audio stream. When a particular audio sample is being rendered, its timestamp is used to locate a data structure having characteristic data. The characteristic data is then used in a visualization rendering process to render a visualization.

Claims

1. A system for synchronizing a visualization with audio samples comprising:

a processor; and

computer-readable storage media having instructions stored thereon, that if executed by the processor, cause the processor to perform a method comprising;

means for receiving and preprocessing audio samples before the samples are rendered by a renderer that comprises part of a media player, to provide characterizing data derived from each sample, the characterizing data comprising a timestamp associated with each audio sample, the timestamp being assigned in accordance with how many, if any, additional audio samples are scheduled to be rendered and when the audio sample is calculated to be rendered by the renderer, wherein the audio samples are preprocessed by a Fast Fourier Transform to provide frequency data associated with the audio samples wherein the characterizing data further comprises the frequency data;

means for holding the characterizing data using a storage medium associated with an audio sample;

means for ascertaining the characterizing data associated with an audio sample that is currently being rendered by the renderer;

said receiving and preprocessing further comprising ascertaining said characterizing data by querying the renderer for a time associated with the currently-rendered audio sample, and then using said time to identify a data structure having a timestamp that is nearest in value to said time;

means for receiving characterizing data that is associated with the storage medium, having the timestamp that is nearest in value to said time, and using the characterizing data to render a visualization that is synchronized with the audio sample that is being rendered by the renderer, wherein the frequency data is used to render the visualization, wherein the visualization is rendered in a rendering area in which other media types can be rendered; and

means for defining a frame rate at which the visualization is to be rendered, setting a threshold associated with an amount of time that is to be spent rendering the visualization, monitoring the time associated with rendering the visualization, determining whether the visualization rendering time exceeds the threshold, and providing an effective frame rate for rendering the visualization that is longer than the defined frame rate if the determined visualization rendering time exceeds the threshold.

2. The system of claim 1, wherein the other media types comprise a video type.

3. The system of claim 1, wherein the other media types comprise a skin type.

4. The system of claim 1, wherein the other media types comprise a HTML type.

5. The system of claim 1, wherein the other media types comprise an animation type.

6. A system for providing a visualization comprising:

a processor; and

computer-readable storage media having instructions stored thereon, that if executed by the processor, cause the processor to perform a method comprising;

means for receiving multiple audio samples;

means for pre-processing the audio samples before they are rendered by a media player renderer, the pre-processing deriving characterizing data from each sample, wherein the characterizing data comprises frequency data that is associated with each audio sample and a timestamp associated with the audio sample, the timestamp being provided based upon how many, if any, additional audio samples are scheduled to be rendered and when the audio sample is calculated to be rendered by the media player renderer;

wherein said means for preprocessing comprises means for using a Fast Fourier Transform to provide frequency data associated with the samples;

means for maintaining characterizing data for each audio sample in a data structure associated with each audio sample;

means for determining when an audio sample is being rendered by the media player renderer, wherein said means for determining comprises;

means for ascertaining a time associated with a currently-rendered audio sample;

means for selecting a data structure having a timestamp that is nearest the time; and

means for providing characterizing data associated with the selected data structure to a component configured to provide the visualization;

means for using the characterizing data that is associated with the audio sample that is being rendered, including the frequency data, to provide a visualization, wherein the frequency data is used to render the visualization; and

means for defining a frame rate at which the visualization is to be rendered, setting a threshold associated with an amount of time that is to be spent rendering the visualization, monitoring the time associated with rendering the visualization, determining whether the visualization rendering time exceeds the threshold, and providing an effective frame rate for rendering the visualization that is longer than the defined frame rate if the determined visualization rendering time exceeds the threshold.