Systems and methods for personalized karaoke

• US 20050123886A1
• Filed: 11/26/2003
• Published: 06/09/2005
• Est. Priority Date: 11/26/2003
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hua, Xian-Sheng, Zhang, Hong-Jiang, Lu, Lie
• Application Number:
  US10/723,049
• Publication Number:
  US 20050123886A1
• Time in Patent Office: Days
• US Class Current: 434/307.A
• CPC Class Codes: G10H 1/361 Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications

Abstract

Systems and methods are described that implement personalized karaoke, wherein a user'"'"'s personal home video and photographs are used to form a background for the lyrics during a karaoke performance. An exemplary karaoke apparatus is configured to segment visual content to produce a plurality of sub-shots and to segment music to produce a plurality of music sub-clips. Having produced the visual content sub-shots and music sub-clips, the exemplary karaoke apparatus shortens some of the plurality of sub-shots to a length of a corresponding music sub-clip from within the plurality of music sub-clips. The plurality of sub-shots is then displayed as a background to lyrics associated with the music, thereby adding interest to a karaoke performance.

Claims

1. A processor-readable medium comprising processor-executable instructions for personalizing karaoke, the processor-executable instructions comprising instructions for:

segmenting visual content to produce a plurality of sub-shots;

segmenting music to produce a plurality of music sub-clips; and

displaying at least some of the plurality of sub-shots as a background to lyrics associated with the plurality of music sub-clips.

2. The processor-readable medium as recited in claim 1, additionally comprising instructions for:

shortening some of the plurality of sub-shots to a length of a corresponding music sub-clip from within the plurality of music sub-clips.

3. The processor-readable medium as recited in claim 1, wherein segmenting the visual content comprises instructions for:

dividing a shot into two sub-shots at a maximum peak of a frame difference curve; and

repeating the dividing to result in sub-shots shorter than a maximum sub-shot length.

4. The processor-readable medium as recited in claim 1, additionally comprising instructions for:

filtering the plurality of sub-shots according to importance; and

filtering the plurality of sub-shots according to quality.

5. The processor-readable medium as recited in claim 4, wherein filtering the plurality of sub-shots according to quality comprises instructions for:

examining color entropy within each of the plurality of sub-shots for indications of diffusion of color; and

if color entropy is low, analyzing each of the plurality of sub-shots to detect motion more that a threshold indicating interest and less than a threshold indicating low camera and/or object movement;

selecting sub-shots having acceptable motion and/or color entropy scores.

6. The processor-readable medium as recited in claim 4, wherein filtering the plurality of sub-shots according to importance comprises instructions for:

evaluating frames within a sub-shot according to attention indices; and

averaging the attention indices for the frames to determine if the sub-shot should be included or excluded.

7. The processor-readable medium as recited in claim 4, wherein filtering the sub-shots according to importance comprises instructions for:

analyzing for camera motion, for object motion and for specific objects within the sub-shots;

filtering the sub-shots according to the analysis.

8. The processor-readable medium as recited in claim 1, wherein the instructions for segmenting visual content segment video.

9. The processor-readable medium as recited in claim 8, additionally comprising instructions for:

selecting important sub-shots from within the plurality of sub-shots; and

selecting sub-shots such that they are uniformly distributed within the video.

10. The processor-readable medium as recited in claim 9, wherein selecting important sub-shots comprises instructions for:

evaluating color entropy, camera motion, object motion and object detection; and

selecting the important sub-shots based on the evaluation.

11. The processor-readable medium as recited in claim 9, wherein selecting uniformly distributed sub-shots comprises instructions for:

evaluating normalized entropy of the sub-shots along a time line of video from which the sub-shots were obtained.

12. The processor-readable medium as recited in claim 1, wherein the instructions for segmenting visual content includes instructions for assigning photographs to be sub-shots.

13. The processor-readable medium as recited in claim 12, wherein the instructions for assigning photographs includes instructions for:

rejecting photographs having problems with quality; and

rejecting photographs within a group of very similar photographs wherein a photo within the group has been selected.

14. The processor-readable medium as recited in claim 12, wherein the instructions for assigning photographs includes instructions for:

converting at least one of the photographs to video.

15. The processor-readable medium as recited in claim 1, wherein the visual content comprises home video and photographs in digital formats.

16. The processor-readable medium as recited in claim 1, wherein segmenting the music comprises instructions for:

establishing boundaries for the music sub-clips at beat positions within the music.

17. The processor-readable medium as recited in claim 1, wherein segmenting music into the plurality of music sub-clips comprises instructions for bounding music sub-clip length according to:

minimum length=min {max {2*tempo,2},4} and
maximum length=minimum+2.

18. The processor-readable medium as recited in claim 1, wherein segmenting the music comprises instructions for:

establishing music sub-clips'"'"' length within a range of 3 to 5 seconds.

19. The processor-readable medium as recited in claim 18, wherein segmenting the music comprises instructions for:

establishing boundaries for the music sub-clips at sentence breaks.

20. The processor-readable medium as recited in claim 1, additionally comprising instructions for:

obtaining the lyrics from a file; and

coordinating delivery of the lyrics with the music using timing information contained within the file.

21. A processor-readable medium as recited in claim 20, wherein obtaining the lyrics comprises instructions for sending the file over a network to a karaoke device as a part of a pay-for-play service.

22. The processor-readable medium as recited in claim 1, additionally comprising instructions for:

querying a database of songs by humming a portion of a desired song; and

selecting the desired song from among a number of possibilities suggested by an interface to the database.

23. A processor-readable medium comprising processor-executable instructions for providing lyrics for integration with music suitable for karaoke, the processor-executable instructions comprising instructions for:

receiving a request for a file associated with a specified song, wherein the file;

associates each syllable contained within the lyrics with timing values; and

associates each sentence contained within the lyrics with timing values; and

fulfilling the request for the file by sending the file associated with the specified song.

24. A processor-readable medium as recited in claim 23, wherein obtaining the lyrics comprises instructions for sending the file over a network to a karaoke device.

25. A personalized karaoke device, comprising:

a music analyzer configured to create music sub-clips of varying lengths according to a song;

a visual content analyzer configured to define and select visual content sub-shots;

a lyric formatter configured to time delivery of syllables of lyrics of the song; and

a composer configured to assemble the music-sub clips with the visual content sub-shots, and configured to adjust length of the sub-shots to correspond to the music sub-clips, and configured to superimpose the syllables of the lyrics of the song over the sub-shots.

26. The personalized karaoke device of claim 25, wherein the music analyzer is configured to segment the song with a strong onset between each of the music sub-clips.

27. The personalized karaoke device of claim 25, wherein the music analyzer is configured to segment the song with a beat between each of the music sub-clips.

28. The personalized karaoke device of claim 25, wherein the music analyzer is configured to segment the song automatically into sub-clips, each having a duration that is a function of song tempo.

29. The personalized karaoke device of claim 25, wherein the visual content analyzer is configured to segment video into sub-shots.

30. The personalized karaoke device of claim 25, wherein the visual content analyzer is configured to access folders of home video and photographs containing content from which the sub-shots are derived.

31. The personalized karaoke device of claim 25, wherein the visual content analyzer is configured to assemble still photographs, each of which is a sub-shot.

32. The personalized karaoke device of claim 25, wherein the visual content analyzer is configured to select from among sub-shots according to ranked importance, wherein importance is gauged by detection of color entropy, detection of object motion within the sub-shot, detection of camera motion during the sub-shot, and/or detection of a face within the sub-shot.

33. The personalized karaoke device of claim 25, wherein the visual content analyzer is configured to filter out sub-shots having low image quality as measured by low entropy and low motion intensity.

34. The personalized karaoke device of claim 25, wherein the visual content analyzer is configured to select sub-shots of greater importance consistent with creating a uniform distribution of the sub-shots over a runtime of a source video.

35. The personalized karaoke device of claim 25, wherein the visual content analyzer is configured to reject photographs of low quality by detecting over and under exposure, overly homogeneous images and blurred images.

36. The personalized karaoke device of claim 25, wherein the visual content analyzer is configured to organize photographs by date of exposure and by scene, thereby obtaining photographs having a relationship.

38. The personalized karaoke device of claim 25, wherein the visual content analyzer is configured to:

detect an attention area within a photograph; and

create a photo to video sub-shot based on the attention area, wherein the video includes panning and/or zooming.

39. The personalized karaoke device of claim 25, wherein the lyric formatter is configured to consume a file detailing timing of each syllable and each sentence of the lyrics.

37. The personalized karaoke device of claim 37, wherein the visual content analyzer is configured to reject photographs which are members within a group of very similar photographs, wherein one of the group has already been selected.

40. An apparatus, comprising:

means for creating music sub-clips of varying lengths according to a song;

means for defining and selecting visual content sub-shots;

means for timing delivery of syllables of lyrics of the song; and

means for assembling the music sub-clips with the visual content sub-shots, and to adjust length of the sub-shots to correspond to length of the music sub-clips, and to superimpose the syllables of the lyrics of the song over the sub-shots.

41. The apparatus of claim 40, wherein the means for defining and selecting visual content sub-shots is a video analyzer configured to segment video into sub-shots.

42. The apparatus of claim 40, wherein the means for defining and selecting visual content sub-shots is a video analyzer configured to access folders of home video and photographs containing content from which the sub-shots are derived.

43. The apparatus of claim 40, wherein the means for defining and selecting visual content sub-shots is a video analyzer configured for:

detecting an attention area within a photograph; and

creating a photo to video sub-shot based on the attention area, wherein the video includes panning and zooming.

44. The apparatus of claim 40, wherein the means for timing delivery of syllables of lyrics of the song is a lyric formatter configured for consuming a file detailing timing of each syllable and each sentence of the lyrics and for rendering the lyrics syllable by syllable.