Interactive tool and appertaining method for creating a graphical music display

• US 7,601,904 B2
• Filed: 08/03/2006
• Issued: 10/13/2009
• Est. Priority Date: 08/03/2005
• Status: Expired due to Fees

• Current Assignee: John Kiehl, Richard Dreyfuss, Roger Luke Dubois
• Original Assignee: John Kiehl, Richard Dreyfuss, Roger Luke Dubois
• Inventors: Dubois, Roger Luke, Kiehl, John, Dreyfuss, Richard
• Primary Examiner(s): Warren; David S.
• Application Number:
  US11/499,834
• Publication Number:
  US 20080314228A1
• Time in Patent Office: 1,167 Days
• Field of Search: 84600-602
• US Class Current: 84/600
• CPC Class Codes: G09B 15/00 Teaching music metronomes G04F5/02
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2220/005 Non-interactive screen display of musical or status data graphical user interfaces specifically adapted for electrophonic musical instruments G10H2220/091; fingering displays G10H2220/041

Abstract

The invention provides apparatus and a method for creating a visual representation of music that conveys the musical form, structure, event surface, orchestration, and sonic framework of the piece being visualized. The representation comprises visual characters/sprites that move in time according to instructions from the metadata synchronized to the actual musical recording. These characters accurately portray the rhythm, pitch, and timbre of the individual elements in the musical surface (individual lines or performers) and have a hierarchy of representation, e.g. thematic elements that are divided into sub-elements (gestures or phrases) that are further divided into individual notes. The visual representation corresponds to the structure of the music as heard, where simultaneous harmonic events are parsed into individual notes which are then tied together into a rhythmic stream of linked events. Two streams of data (the “musical” and the “sonic”) are synchronized to the musical piece to drive the visualization.

Claims

1. A system for producing a time-based representation of a recorded musical work, comprising:

a sound recording of the recorded musical work;

a musical meta-score algorithm that transforms non-audio musical source material into a musical meta-score that is synchronized with the sound recording based on a combination of harmonic, melodic, and rhythmic characteristics derived from the musical source material;

a sonic meta-score algorithm that transforms the sound recording into a sonic meta-score comprising information related to sound characteristics of the recording that is synchronized with the sound recording;

a synchronization algorithm that combines the musical meta-score and the sonic meta-score and maps data in the combination to visual descriptors; and

a computer display system that displays a sequential series of images comprising renderings of the visual descriptors that are choreographed to the mapped combination of data.

2. The system according to claim 1, wherein the musical source material is selected from the group consisting of a musical score, a MIDI file or data stream, and a recording transcription.

3. The system according to claim 1, wherein the musical meta-score comprises digital encodings of pitches, rhythms, dynamics and arrangement of the sound recording.

4. The system according to claim 1, wherein the musical meta-score comprises time stamps accurate to the display of the sequential series of images.

5. The system according to claim 4, wherein the time stamps occur at a resolution utilized by known display timing or refresh standards.

6. The system according to claim 5, wherein the time stamps occur at a resolution of 30 frames/second.

7. The system according to claim 1, wherein the harmonic characteristics comprise data related to duration, key areas and chord vocabulary.

8. The system according to claim 1, wherein the melodic characteristics comprise data related to major themes, motifs and orchestration shifts.

9. The system according to claim 1, wherein the rhythmic characteristics comprise data related to meters, beat, measure division and patterns.

10. The system according to claim 1, wherein the sonic meta score comprises amplitude, pitch, and timbral envelope information.

11. The system according to claim 1, wherein the recording is a multi-part or multi-track recording comprising either digital or analog data.

12. The system according to claim 11, wherein each part of the multi-part recording represents either a single instrument or single voice.

13. The system according to claim 1, wherein the sound characteristics of the recording comprise pitch, timbre and volume.

14. The system according to claim 1, further comprising a configuration file comprising settings utilized to provide the sequential series of images.

15. The system according to claim 1, further comprising a user input with controls via which a user adjusts characteristics of the display of the sequential series of images.

16. The system according to claim 15, wherein the user input controls comprise controls for selectively isolating voices, freezing, zooming, panning around within the display, or adjusting a user viewpoint.

17. The system according to claim 1, further comprising an algorithm for mapping specific musical parts into a series of visual descriptors in a virtual scene on the computer display.

18. The system according to claim 17, wherein the visual descriptors are two- or three-dimensional visual descriptors or sprites.

19. The system according to claim 17, further comprising an algorithm for morphing the visual descriptors among a set of fixed geometric shapes according to pitch classes they represent.

20. The system according to claim 17, further comprising an algorithm for transforming the form of the visual descriptors based on at least one of pitch, volume, and salience.

21. The system according to claim 17, further comprising an algorithm for moving the visual descriptors based on information contained within the meta-scores.

22. The system according to claim 1, further comprising at least one of Fourier analysis algorithms and Linear Predictive Coding algorithms that are utilized in producing the sonic meta-score from the multi-part audio recording.

23. The system according to claim 1, wherein each visual descriptor corresponds to a single part of the multi-part recording.

24. The system according to claim 1, further comprising an algorithm for dividing the sound recording into thematic and structural elements, dividing the thematic and structural elements into sub-elements that are gestures or phrases, and dividing the sub-elements into individual notes, and for associating the thematic and structural elements, sub-elements, and individual notes into the sequential series of images.

25. The system according to claim 24, further comprising an algorithm for performing a hierarchical analysis in order to extrapolate the thematic and structural elements.

26. The system according to claim 1, further comprising an algorithm for providing lines or other trail markers to connect the visual descriptors and sequential series of images that represent melodic fragments, motifs and themes represented.

27. The system according to claim 1, further comprising an algorithm to apply data in the sonic meta-score to modify contour and shape of the visual descriptors.

28. The system according to claim 1, further comprising an event queue that comprises information combined from the musical meta-score and the sonic meta-score by image or frame.

29. The system according to claim 28, further comprising a mapping system that maps events in the event queue to visual descriptors.

30. The system according to claim 1, wherein the computer display system is a real-time display system.

31. The system according to claim 1, wherein the visual descriptors represent three-dimensional objects in three-dimensional space presented on the display system either in actual three-dimensional form or two-dimensional projected form.

32. The system according to claim 1, further comprising a storage area for storing the visual descriptors.

33. A method for producing a visual display that accurately represents a recorded musical work, comprising:

transforming musical source material related to a multi-part sound recording into a musical meta-score that is synchronized with the sound recording based on a combination of harmonic, melodic, and rhythmic characteristics derived from the musical source material;

transforming the multi-part sound recording into a sonic meta-score comprising information related to sound characteristics of the recording that is synchronized with the sound recording;

combining the musical meta-score and the sonic meta-score and mapping data in the combination to visual descriptors; and

displaying a sequential series of images comprising renderings of the visual descriptors as choreographed to the mapped combination of data.

34. The method according to claim 33, wherein the musical source material is selected from the group consisting of a musical score, a MIDI file or data stream, and a recording transcription.

35. The method according to claim 33, further comprising generating digital encodings of pitches, rhythms, dynamics and arrangement of the sound recording in the musical meta-score.

36. The method according to claim 33, further comprising generating time stamps accurate to the display of the sequential series of images in the musical meta-score.

37. The method according to claim 36, further comprising providing the time stamps at a resolution utilized by known display timing or refresh standards.

38. The method according to claim 37, wherein the resolution is 30 frames/second.

39. The method according to claim 33, wherein the harmonic characteristics comprise data related to duration, key areas and chord vocabulary.

40. The method according to claim 33, wherein the melodic characteristics comprise data related to major themes, motifs and orchestration shifts.

41. The method according to claim 33, wherein the rhythmic characteristics comprise data related to meters, beat, measure division and patterns.

42. The method according to claim 33, wherein the sonic meta-score comprises amplitude, pitch, and timbral envelope information.

43. The method according to claim 33, wherein the recording is a multi-part or multi-track recording comprising either digital or analog data.

44. The method according to claim 43, wherein each part of the multi-part recording represents either a single instrument or single voice.

45. The method according to claim 33, wherein the sound characteristics of the recording comprise pitch, timbre and volume.

46. The method according to claim 33, further comprising utilizing a configuration file to provide the sequential series of images.

47. The method according to claim 33, further comprising utilizing controls of a user input to adjust characteristics of the display of the sequential series of images.

48. The method according to claim 47, further comprising selectively isolating voices, freezing, zooming, panning around within the display, or adjusting a user viewpoint with the user input controls.

49. The method according to claim 33, wherein the displaying of a sequential series of images comprises displaying the sequential series of images on a computer display, and wherein the method further comprises mapping specific musical parts into a series of visual descriptors in a virtual scene on the computer display.

50. The method according to claim 49, wherein the visual descriptors are two- or three-dimensional visual descriptors or sprites.

51. The method according to claim 49, further comprising morphing the visual descriptors among a set of fixed geometric shapes according to pitch classes they represent.

52. The method according to claim 49, further comprising transforming the form of the visual descriptors based on at least one of pitch, volume, and salience.

53. The method according to claim 52, wherein the volume is a dynamic volume.

54. The method according to claim 49, further comprising moving the visual descriptors based on information contained within the meta-scores.

55. The method according to claim 33, further comprising utilizing at least one of Fourier analysis algorithms and Linear Predictive Coding algorithms in producing the sonic meta-score from the multi-part audio recording.

56. The method according to claim 33, wherein each visual descriptor corresponds to a single part of the multi-part recording.

57. The method according to claim 33, further comprising:

dividing the sound recording into thematic and structural elements;

dividing the thematic and structural elements into sub-elements that are gestures or phrases;

dividing the sub-elements into individual notes; and

associating the thematic and structural elements, sub-elements, and individual notes into the sequential series of images.

58. The method according to claim 57, further comprising performing a hierarchical analysis in order to extrapolate the thematic and structural elements.

59. The method according to claim 33, further comprising providing lines or other trail markers to connect the visual descriptors and sequential series of images that represent melodic fragments, motifs and themes represented.

60. The method according to claim 33, further comprising applying data in the sonic meta-score to modify contour and shape of the visual descriptors.

61. The method according to claim 33, further comprising producing an event queue that comprises information combined from the musical meta-score and the sonic meta-score by image or frame.

62. The method according to claim 61, further comprising mapping events in the event queue to visual descriptors.

63. The method according to claim 33, wherein the displaying of a sequential series of images comprises displaying the sequential series of images on a computer display system, and wherein the computer display system is a real-time display system.

64. The method according to claim 33, wherein the visual descriptors represent three-dimensional objects in three-dimensional space presented on the display system either in actual three-dimensional form or two-dimensional projected form.

65. The method according to claim 33, further comprising storing the visual descriptors in a storage area.