Methods and systems for synchronizing visualizations with audio streams

• US 7,072,908 B2
• Filed: 03/26/2001
• Issued: 07/04/2006
• Est. Priority Date: 03/26/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Novak, Michael J., Harris, Geoffrey, Olson, Kipley J., Feller, Chris, Dideriksen, Tedd
• Primary Examiner(s): Donovan, Greta
• Assistant Examiner(s): Le, Miranda
• Application Number:
  US09/817,902
• Publication Number:
  US 20020172377A1
• Time in Patent Office: 1,926 Days
• Field of Search: 707/1, 707/100, 707/101, 707/104.1, 707/200, 381/98, 715/500.1, 345/418, 345/419, 345/716, 709/231, 725/37, 375/240.28
• US Class Current: 1/1
• CPC Class Codes: H04S 3/00 Systems employing more than two channels, e.g. quadraphonic H04S5/00, H04S7/00 take precedence
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users

Abstract

Methods and systems are described that assist media players in rendering visualizations and synchronizing those visualizations with audio samples. In one embodiment, visualizations are synchronized with an audio stream using a technique that builds and maintains various data structures. Each data structure can maintain data that is associated with a particular pre-processed audio sample. The maintained data can include a timestamp that is associated with a time when the audio sample is to be rendered. The maintained data can also include various characteristic data that is associated with the audio stream. When a particular audio sample is being rendered, its timestamp is used to locate a data structure having characteristic data. The characteristic data is then used in a visualization rendering process to render a visualization.

Claims

1. A system for synchronizing visualization with audio samples comprising:

one or more audio sources configured to provide audio samples that are to be rendered by a media player;

an audio sample pre-processor communicatively linked with the one or more audio sources and configured to receive and pre-process audio samples before the samples are rendered, the pre-processing providing characterizing data associated with each sample, wherein the characterizing data is derived from the audio samples, wherein said audio sample pre-processor comprises a Fast Fourier Transform that it utilizes to process the audio samples to provide frequency data associated with the audio samples, wherein the audio sample pre-processor comprises a timestamp module that provides a timestamp for each audio sample, each timestamp being maintained by a data structure associated with the audio sample, wherein the audio sample pre-processor is configured to;

query a media player audio sample renderer for a time associated with an audio sample that is being currently rendered, and

use the time to ascertain a timestamp of an associated audio sample, the audio sample pre-processor further being configured to provide characterizing data of the associated audio sample so that the characterizing data can be used to render the visualization;

one or more effects configured to receive the characterizing data and use the characterizing data to render a visualization that is synchronized with an audio sample that is being rendered by the media player; and

multiple data structures configured to hold the characterizing data, each data structure being associated with an audio sample.

2. The system of claim 1, wherein the audio sample pre-processor is configured to maintain the data structures.

3. The system of claim 1, wherein the timestamp is assigned by the timestamp module based upon when the audio sample is calculated to be rendered by the media player.

4. The system of claim 1, wherein said characterizing data comprises frequency data.

5. A media player comprising:

an audio sample pre-processor configured to receive and pre-process audio samples before the samples are rendered by the media player, the pre-processing providing frequency data associated with each sample, wherein the frequency data is derived from the audio samples, wherein the audio sample pre-processor pre-processes the audio samples by using a Fast Fourier Transform to provide the frequency data, wherein the audio sample pre-processor comprises a timestamp module that provides a timestamp for each audio sample, each timestamp being maintained by a data structure associated with the audio sample, and further wherein the audio sample pre-processor is configured to;

query a media player audio sample renderer for a time associated with an audio sample that is being currently rendered, and

use the time to ascertain a timestamp of an associated audio sample, the audio sample pre-processor further being configured to provide frequency data of the associated audio sample to the one or more effects so that the frequency data can be used to render the visualization;

one or more effects configured to receive the frequency data and use the frequency data to render a visualization that is synchronized with an audio sample that is being rendered by the media player; and

multiple data structures configured to hold the frequency data, each data structure being associated with an audio sample.

6. A system for synchronizing a visualization with audio samples comprising:

an audio sample pre-processor configured to receive and preprocess audio samples before the samples are rendered by a renderer that comprises part of a media player, the audio sample preprocessor preprocessing the samples to provide characterizing data derived from each sample, the characterizing data comprising a timestamp associated with each audio sample, the timestamp being assigned in accordance with when the audio sample is calculated to be rendered by the renderer, wherein the audio sample pre-processor comprises a Fast Fourier Transform that it utilizes to process the audio samples to provide frequency data associated with the audio samples;

multiple data structures configured to hold the characterizing data, each data structure being associated with an audio sample;

an audio rendering object configured to call the audio sample pre-processor to ascertain the characterizing data associated with an audio sample that is currently being rendered by the renderer;

the audio sample pre-processor being configured to ascertain said characterizing data by querying the renderer for a time associated with the currently-rendered audio sample, and then using said time to identify a data structure having a timestamp that is nearest in value to said time; and

one or more effects configured to receive characterizing data that is associated with the data structure having the timestamp that is nearest in value to said time, and use the characterizing data to render a visualization that is synchronized with the audio sample that is being rendered by the renderer.

7. The system of claim 6, wherein the characterizing data comprises frequency data.

8. The system of claim 6, wherein the visualization is rendered in a rendering area in which other media types can be rendered.

9. The system of claim 8, wherein the other media types comprise a video type.

10. The system of claim 8, wherein the other media types comprise a skin type.

11. The system of claim 8, wherein the other media types comprise a HTML type.

12. The system of claim 8, wherein the other media types comprise an animation type.

13. A system for processing audio samples comprising:

a timestamp module for assigning timestamps to audio samples that are to be rendered by a media player renderer

a spectrum analyzer for processing the audio samples to derive frequency data from the audio samples, wherein the spectrum analyzer comprises a Fast Fourier Transform that is utilized to provide the frequency data;

a multiple data structures each of which being associated with an audio sample, the data structures each containing timestamp data and frequency data for its associated audio sample; and

the system being configured to use the timestamp data to ascertain a data structure associated with an audio sample that is currently being rendered by the media player renderer and provide the frequency data associated with that audio sample so that the frequency data can be used to render a visualization associated with that audio sample.

14. A method of providing a visualization comprising:

receiving multiple audio samples;

pre-processing the audio samples before they are rendered by a media player renderer, the pre-processing deriving characterizing data for each sample, wherein said pre-processing comprises using a Fast Fourier Transform to provide frequency data associated with the samples, wherein the characterizing data comprises a timestamp associated with the audio sample, the timestamp being provided based upon when the audio sample is calculated to be rendered by the media player renderer;

maintaining characterizing data for each audio sample in a data structure associated with each audio sample;

determining when an audio sample is being rendered by the media player renderer, wherein said determining comprises;

ascertaining a time associated with a currently-rendered audio sample;

selecting a data structure having a timestamp that is nearest the time; and

providing characterizing data associated with the selected data structure to a component configured to provide the visualization; and

responsive to said determining, using the characterizing data that is associated with the audio sample that is being rendered to provide a visualization.

15. The method of claim 14, wherein the characterizing data comprises frequency data associated with each sample.

16. A method of providing a visualization comprising:

receiving multiple audio samples;

pre-processing the audio samples before they are rendered by a media player renderer, the pre-processing comprising at least (1) using a Fast Fourier Transform to derive frequency data from the samples, and (2) associating a timestamp with each sample;

maintaining frequency data and a timestamp for each sample in a data structure;

determining when an audio sample is being rendered by a media player renderer by;

ascertaining a time associated with a currently-rendered sample; and

selecting a data structure having a timestamp that is nearest the time; and

providing frequency data associated with the selected data structure to a component configured to use the frequency data to render the visualization.

17. One or more computer-readable media having computer-readable instructions thereon which, when executed by a computer, cause the computer to implement the method of claim 16.

18. A method of providing a visualization comprising:

calling an audio sample pre-processor for characterizing data that has been derived from an associated audio sample that is currently being rendered by a media player renderer, wherein the characterizing data comprises frequency data associated with the audio samples and generated by pre-processing the audio samples using a Fast Fourier Transform;

calling the media player renderer for a time associated with a currently-rendered audio sample;

using the time to select a data structure containing characterizing data associated with the currently-rendered audio sample; and

providing the characterizing data to a component for rendering a visualization.

19. One or more computer-readable media having computer-readable instructions thereon which, when executed by a computer, cause the computer to:

pre-process audio samples using a Fast Fourier Transform to derive from the audio samples frequency data, the audio samples being pre-processed before they are rendered by a media player renderer;

query for frequency data that is associated with an audio sample that is currently being rendered by the media player renderer;

query for a time associated with the currently-rendered audio sample;

use the time to select a data structure containing frequency data associated with the currently-rendered audio sample; and

provide the frequency data to a component that uses the frequency data for rendering a visualization.

20. A method of providing a visualization comprising:

defining a frame rate at which visualization frames of a visualization are to be rendered, the visualization frames being rendered from characterizing data that is computed from audio sample and which is used to create the visualization, wherein the characterizing data comprises frequency data associated with the audio samples and generated by pre-processing the audio samples using a Fast Fourier Transform;

setting a threshold associated with an amount of time that is to be spent rendering a visualization frame;

monitoring the time associated with rendering individual visualization frames;

determining whether a visualization frame rendering time exceeds the threshold; and

providing an effective frame rate for rendering visualization frames that is longer than the defined frame rate if the determined visualization frame rendering time exceeds the threshold.

21. The method of claim 20, wherein said providing comprises increasing a call interval associated with calls that are made to a visualization-rendering component.

22. The method of claim 20, further comprising modifying the effective frame rate so that the visualization frames are rendered at the defined frame rate.

23. set a threshold associated with an amount of time that is to be spent rendering a visualization frame for a given frame rate, said visualization frame being associated with a visualization that is rendered using characterizing data that is computed from audio samples, which characterizing data is used to create the visualization, wherein the characterizing data comprises frequency data associated with the audio samples and generated by pre-processing the audio samples using a Fast Fourier Transform;

monitor the time associated with rendering individual visualization frames;

determine whether a visualization frame rendering time exceeds the threshold;

provide an effective frame rate for rendering the visualization that is longer than the defined frame rate if the determined visualization frame rendering time exceeds the threshold.