Systems and Methods of Rendering a Textual Animation

• US 20110273455A1
• Filed: 12/03/2010
• Published: 11/10/2011
• Est. Priority Date: 05/04/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Shazam Entertainment Limited (Apple Inc.)
• Inventors: Powar, Rahul, Wang, Avery Li-Chun
• Granted Patent: US 9,159,338 B2
• Time in Patent Office: Days
• US Class Current: 345/473
• CPC Class Codes: G06F 16/638 Presentation of query results
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications
• CPC Class Codes: G10H 2240/251 Mobile telephone transmission, i.e. transmitting, accessing or controlling music data wirelessly via a wireless or mobile telephone receiver, analog or digital, e.g. DECT GSM, UMTS smartphone, PDA or palmtop used for musical purposes G10H2230/015; mobile ringtone G10H2230/021
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: G10L 21/18 Details of the transformation process
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

Systems and methods of rendering a textual animation are provided. The methods include receiving an audio sample of an audio signal that is being rendered by a media rendering source. The methods also include receiving one or more descriptors for the audio signal based on at least one of a semantic vector, an audio vector, and an emotion vector. Based on the one or more descriptors, a client device may render the textual transcriptions of vocal elements of the audio signal in an animated manner. The client device may further render the textual transcriptions of the vocal elements of the audio signal to be substantially in synchrony to the audio signal being rendered by the media rendering source. In addition, the client device may further receive an identification of a song corresponding to the audio sample, and may render lyrics of the song in an animated manner.

Claims

1. A method of rendering a textual animation, comprising:

receiving an audio sample of an audio signal comprising at least one of audio elements and vocal elements, the audio signal being rendered by a media rendering source;

sending the audio sample to a server;

in response to sending the audio sample to the server, receiving one or more descriptors for the audio signal based on at least one of a semantic vector, an audio vector, and an emotion vector, wherein the semantic vector indicates a semantic content of corresponding textual transcriptions of vocal elements of the audio signal as a function of time with respect to a length of the audio signal, wherein the audio vector indicates an audio content of audio elements of the audio signal as a function of time with respect to a length of the audio signal, and wherein the emotion vector indicates an emotional content of audio elements of the audio signal as a function of time with respect to a length of the audio signal; and

based on the one or more descriptors, a client device rendering the textual transcriptions of vocal elements of the audio signal in an animated manner.

2. The method of claim 1, further comprising based on the one or more descriptors, the client device rendering the textual transcriptions of the vocal elements of the audio signal to be substantially in synchrony to the audio signal being rendered by the media rendering source.

3. The method of claim 2, wherein the audio sample is associated with a timestamp corresponding to a beginning time of the audio sample, and the method further comprises:

receiving a time offset indicating a time position in the audio signal corresponding to the beginning time of the audio sample;

determining a real-time offset using a real-time timestamp, the timestamp of the audio sample, and the time offset, wherein the real-time timestamp indicates a present time;

receiving the textual transcriptions of the vocal elements; and

the client device rendering the textual transcriptions of the vocal elements of the audio signal at a time corresponding to the real-time offset and to be substantially in synchrony to the audio sample being rendered by the media rendering source.

4. The method of claim 2, wherein the audio sample is associated with a timestamp corresponding to a beginning time of the audio sample, and the method further comprises:

receiving a time offset indicating a time position in the audio signal corresponding to the beginning time of the audio sample;

determining a real-time offset using a real-time timestamp, the timestamp of the audio sample, and the time offset, wherein the real-time timestamp indicates a present time;

receiving a numeric skew indicating a speed at which the audio signal is being rendered by the media rendering source with reference to a given rendering speed of the audio signal;

receiving the textual transcriptions of the vocal elements; and

the client device rendering the textual transcriptions of the vocal elements of the audio signal at a time corresponding to the real-time offset, at the speed indicated by the numeric skew, and to be substantially in synchrony to the audio sample being rendered by the media rendering source.

5. The method of claim 1, further comprising:

receiving an identification of a song corresponding to the audio sample; and

receiving the textual transcriptions of the vocal elements of the song.

6. The method of claim 1, further comprising receiving the one or more descriptors for the audio signal based on the identification of the song.

7. The method of claim 1, wherein receiving the one or more descriptors for the audio signal comprises the client device receiving the one or more descriptors from a server.

8. The method of claim 1, further comprising:

in response to sending the audio sample to the server;

receiving an identification of a song corresponding to the audio sample; and

receiving the one or more descriptors for the audio signal.

9. The method of claim 1, further comprising:

receiving graphics or images; and

based on the one or more descriptors, the client device rendering the graphics or images in an animated manner.

10. The method of claim 1, further comprising the client device performing the method in real time.

11. The method of claim 1, further comprising:

the client device receiving the one or more descriptors for the audio signal;

storing in memory the one or more descriptors for the audio signal;

subsequently receiving the audio sample, the audio sample being rendered by the media rendering source;

the client device retrieving the one or more descriptors for the audio signal from the memory; and

based on the one or more descriptors, the client device rendering the textual transcriptions of the vocal elements of the audio signal in the animated manner.

12. The method of claim 1, further comprising:

based on at least one of the semantic vector, the audio vector, or the emotion vector, selecting an animation theme selected from the group consisting of colors, types of animation, backgrounds, typefaces of font, speed of animation, saturation, and brightness; and

the client device rendering the textual transcriptions of the vocal elements of the audio signal in the animated manner based on the selected animation theme.

13. The method of claim 1, wherein the audio signal is a song and the textual transcriptions of the vocal elements are lyrics of the song, and the method further comprises:

receiving background graphics; and

displaying the lyrics and the background graphics in a dynamic animation based on the one or more descriptors, the animation of the lyrics being independent of the animation of the background graphics.

14. The method of claim 1, wherein the client device performs the steps of:

receiving the audio sample of the audio signal and rendering the textual transcriptions of the vocal elements of the audio signal in an animated manner.

15. The method of claim 1, further comprising:

determining textual transcriptions of the vocal elements of the audio sample using speech recognition; and

based on the one or more descriptors, the client device rendering the textual transcriptions of the vocal elements of the audio sample in an animated manner.

16. The method of claim 1, wherein the one or more descriptors includes a time stamp that indicates a time position in the audio signal at which to apply specified animation.

17. The method of claim 1, further comprising the client device rendering the textual transcriptions of vocal elements of the audio signal in a dynamic animation, wherein the dynamic animation changes based on the one or more descriptors.

18. A method comprising:

receiving an audio sample;

determining an identification of a song corresponding to the audio sample, the song comprising at least one of audio elements and vocal elements;

retrieving one or more descriptors for the song based on at least one of a semantic vector, an audio vector, and an emotion vector, wherein the semantic vector indicates a semantic content of corresponding textual transcriptions of the vocal elements as a function of time with respect to a length of the song, wherein the audio vector indicates an audio content of the audio elements as a function of time with respect to a length of the song, and wherein the emotion vector indicates an emotional content of the audio elements as a function of time with respect to a length of the song; and

sending to a client device the one or more descriptors.

19. The method of claim 18, further comprising:

based on the one or more descriptors, selecting a textual animation or a background animation for the song; and

sending to the client device the textual animation or the background information.

20. The method of claim 19, further comprising associating a time stamp with the textual animation or the background animation for the song, wherein the time stamp indicates a time position in the song at which to apply the textual animation or the background animation.

21. The method of claim 20, further comprising:

selecting multiple textual animations or multiple background animations for the song; and

associating a given time stamp with each textual animation or background animation for the song.

22. The method of claim 18, further comprising associating a time stamp with each descriptor in the one or more descriptors such that animation of the textual transcriptions of the vocal elements is substantially in synchrony with the audio sample when rendered.

23. The method of claim 18, further comprising determining the semantic content of corresponding textual transcriptions of the vocal elements to form the semantic vector.

24. The method of claim 18, further comprising determining musical content of the audio elements to form the audio vector.

25. The method of claim 18, further comprising determining the emotional content based on a musical key or tone of the audio elements to form the emotion vector.

26. The method of claim 18, further comprising determining the emotional content based on lyrics of the audio sample.

27. The method of claim 18, further comprising wirelessly receiving a request for the one or more descriptors from a mobile client device.

28. The method of claim 18, wherein the audio sample is associated with a timestamp corresponding to a beginning time of the song, and the method further comprises:

determining a time offset indicating a time position in the song corresponding to the beginning time of the audio sample;

sending to the client device the textual transcriptions of the vocal elements and the time offset to enable the client device to render the textual transcriptions of the vocal elements at a time corresponding to the time offset and to be substantially in synchrony to the song as being rendered by a media rendering source.

29. The method of claim 18, wherein the audio sample is associated with a timestamp corresponding to a beginning time of the song, and the method further comprises:

determining a time offset indicating a time position in the song corresponding to the beginning time of the audio sample;

determining a numeric skew indicating a speed at which the song is being rendered by a media rendering source with reference to a given rendering speed of the song; and

sending to the client device the textual transcriptions of the vocal elements and the time offset to enable the client device to render the textual transcriptions of the vocal elements at a time corresponding to the time offset, at the speed indicated by the numeric skew, and to be substantially in synchrony to the song as being rendered by the media rendering source.

30. A non-transitory computer readable storage medium having stored therein instructions executable by a computing device to cause the computing device to perform functions of:

receiving an audio sample of an audio signal comprising at least one of audio elements and vocal elements, the audio signal being rendered by a media rendering source;

sending the audio sample to a server;

in response to sending the audio sample to the server, receiving one or more descriptors for the audio signal based on at least one of a semantic vector, an audio vector, and an emotion vector, wherein the semantic vector indicates a semantic content of corresponding textual transcriptions of vocal elements of the audio signal as a function of time with respect to a length of the audio signal, wherein the audio vector indicates an audio content of the audio elements of the audio signal as a function of time with respect to a length of the audio signal, and wherein the emotion vector indicates an emotional content of the audio elements of the audio signal as a function of time with respect to a length of the audio signal; and

based on the set of descriptors, rendering the textual transcriptions of vocal elements of the audio signal in an animated manner.

31. A method of rendering a textual animation, comprising:

receiving an audio sample of an audio signal comprising at least one of audio elements and vocal elements, the audio signal being rendered by a media rendering source;

determining an identification of a song corresponding to the audio sample and lyrics corresponding to the vocal elements;

receiving a set of descriptors for the song based on at least one of a semantic vector, an audio vector, and an emotion vector, wherein the semantic vector indicates a semantic content of the lyrics as a function of time with respect to a length of the song, wherein the audio vector indicates an audio content of audio elements of the song as a function of time with respect to a length of the song, and wherein the emotion vector indicates an emotional content of audio elements of the song as a function of time with respect to a length of the song;

receiving a time offset indicating a time position in the audio signal corresponding to a beginning time of the audio sample;

determining a real-time offset using a real-time timestamp, a beginning time of the audio sample, and the time offset, wherein the real-time timestamp indicates a present time; and

based on the set of descriptors, a client device rendering the lyrics in an animated manner at a time corresponding to the real-time offset and substantially in synchrony to the audio signal being rendered by the media rendering source.

32. The method of claim 31, wherein determining the identification of the song corresponding to the audio sample comprises:

sending the audio sample to an audio identification system; and

receiving information indicating the identification of the song.

33. The method of claim 31, further comprising:

receiving a numeric skew indicating a speed at which the audio signal is being rendered by the media rendering source with reference to a given rendering speed of the audio signal; and

the client device rendering the lyrics at a time corresponding to the real-time offset, at the speed indicated by the numeric skew, and substantially in synchrony to the audio sample being rendered by the media rendering source.