Method and apparatus for generating visual images based on musical compositions

• US 7,589,727 B2
• Filed: 01/18/2006
• Issued: 09/15/2009
• Est. Priority Date: 01/18/2005
• Status: Active Grant

• Current Assignee: Eric P. Haeker
• Original Assignee: Eric P. Haeker
• Inventors: Haeker, Eric P.
• Primary Examiner(s): PAPPAS, PETER
• Application Number:
  US11/334,029
• Publication Number:
  US 20060156906A1
• Time in Patent Office: 1,336 Days
• Field of Search: None
• US Class Current: 345/440
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/0066 using a MIDI interface
• CPC Class Codes: G10H 2210/086 for transcription of raw audio or music data to a displayed or printed staff representation or to displayable MIDI-like note-oriented data, e.g. in pianoroll format
• CPC Class Codes: G10H 2220/005 Non-interactive screen display of musical or status data graphical user interfaces specifically adapted for electrophonic musical instruments G10H2220/091; fingering displays G10H2220/041
• CPC Class Codes: G10H 2220/401 3D sensing, i.e. three-dimensional (x, y, z) position or movement sensing. movement pattern or gesture sensing G10H2220/201; geolocation sensing G10H2220/355, 3D sensing with accelerometer G10H2220/395

Abstract

The present invention generates 3D moving images representing various aspects of a musical performance that can be synchronized, as necessary, to the changing tempo of a live or recorded performance, either automatically, or with live-controlled user input, and either with or without a score.

Claims

1. A method of producing a graphical representation of a musical work comprising a plurality of individual musical lines comprising notes, said method comprising the steps of:

(1) obtaining an electronic version of said musical work;

(2) translating using a processor said notes of each individual musical line of said electronic version into a separate x, y graph in which a y value of said notes is representative of a pitch of said note and an x value is representative of a relative time of said note in said musical work and a duration of said note;

(3) importing using said processor each said graph into three-dimensional animation software;

(4) generating using said processor a visual object corresponding to each individual musical line of said musical work; and

(5) applying using said processor at least one three-dimensional animation technique to each said object, said animation technique being a function of said corresponding graph.

2. The method of claim 1 further comprising the step of:

(6) displaying said object using a visual display device.

3. The method of claim 2 further comprising the steps of:

(7) for each graph, generating a smooth mathematical function representative of a curve traced by said graph; and

(8) importing said smooth mathematical function and said graph into three-dimensional animation software;

wherein said animation technique a function of said mathematical function.

4. The method of claim 3 wherein step (7) comprises generating a piece wise smooth linear function that has a finite derivative at all points.

5. The method of claim 2 wherein there is a one-to-one correspondence between the animation and the music such that a written traditional musical score can be derived from the animation.

6. The method of claim 2 wherein step (1) comprises converting a standard musical score into a MIDI score.

7. The method of claim 6 wherein step (2) comprises generating a MIDI graph of said musical work.

8. The method of claim 2 wherein step (1) comprises:

(1.1) receiving an audible musical composition;

(1.2) converting said audible musical composition into said electronic version in real time.

9. The method of claim 2 wherein each said object is assigned to a different position within a virtual three-dimensional space.

10. The method of claim 9 wherein the relative positions of said objects in said virtual three-dimensional space correspond to the relative positions of instruments in an musical ensemble producing the musical lines corresponding to said objects.

11. The method of claim 2 wherein step (5) comprises depicting an object for each individual musical line in said musical work, each said object assigned to a different position within a virtual three-dimensional space.

12. The method of claim 11 wherein the relative positions of said objects in said virtual three-dimensional space correspond to the relative positions of instruments in a musical ensemble producing the musical lines corresponding to said objects.

13. A method of producing a graphical representation of a musical work comprising a plurality of notes, said method comprising the steps of:

(1) obtaining an electronic version of said musical work;

(2) translating using a processor said notes of said electronic version into an x, y graph in which a y value of said notes is representative of a pitch of said note and said x value is representative of a relative time of said note in said musical work and a duration of said note;

(3) importing using said processor said graph into three-dimensional animation software;

(4) selecting using said processor a frame rate, said frame rate being a number of frames per musical time unit in said musical work;

(5) providing to said animation software a tempo of said musical work;

(6) synchronizing said frame rate to said tempo;

(7) generating using said processor a visual object; and

(8) applying using said processor at least one three-dimensional animation technique to said object, said animation technique being a function of said graph.

14. The method of claim 13 further comprising the step of:

(9) displaying said object using a visual display device.

15. The method of claim 14 further comprising the step of:

(10) generating using said processor a smooth mathematical function representative of a curve traced by said graph; and

wherein, in step (8), said animation technique is a function of said smooth mathematical function.

16. The method of claim 14 wherein step (4) comprises selecting a frame rate such that every note in said musical work corresponds to an integer number of frames.

17. The method of claim 16 wherein said musical time unit is a quarter note of said musical work.

18. The method of claim 14 wherein step (5) comprises a user manually providing said tempo to said animation software.

19. The method of claim 18 wherein step (5) comprises said user activating an input device in temporal correspondence with said tempo of said musical piece.

20. The method of claim 19 wherein step (6) further comprises automatically correcting for a missed activation by said user.

21. The method of claim 20 further comprising the step of:

(11) permitting said user to disable said automatic correction in step (6), whereby said automatic correction will not interfere with a sudden tempo change in said musical work.

22. The method of claim 14 wherein step (5) comprises:

(5.1) inputting said musical work via a digital audio input to automatic pitch and rhythm tracking software; and

(5.2) using said automatic pitch and rhythm tracking software to determine said tempo.

23. The method of claim 22 wherein step (5.1) comprises inputting each melodic line of said musical work via a separate digital audio input.

24. A method of producing a graphical representation of a musical work comprising a plurality of individual musical lines comprising notes, said method comprising the steps of:

(1) obtaining an electronic version of said musical work;

(2) translating said notes of each individual musical line of said electronic version into a separate x, y graph in which a y value of said notes is representative of a pitch of said note and an x value is representative of a relative time of said note in said musical work and a duration of said note;

(3) analyzing said electronic version of said musical work to identify discrete coherent musical phrases within said work;

(4) importing using a processor each said graph into three-dimensional animation software; and

(5) generating using said processor a visual display depicting an object for each individual music line and applying at least one three-dimensional animation technique to said object, at least one of each said object and said animation technique being a function of said graph and said musical phrases.

25. The method of claim 24 further comprising the step of:

(6) displaying said object using a visual display device.

26. The method of claim 25 wherein said musical phrases are at least one of keys, musical tension/release sequences, musical themes, and musical antecedent/consequent sequences.

27. The method of claim 25 wherein there is a one-to-one correspondence between the animation and the music such that a written traditional musical score can be derived from the animation.

28. The method of claim 25 wherein step (1) comprises converting a standard musical score into a MIDI score.

29. The method of claim 28 wherein step (2) comprises generating a MIDI graph.

30. The method of claim 25 wherein each said object is assigned to a different position within a virtual three-dimensional space.

31. The method of claim 30 wherein the relative positions of said objects in said virtual three-dimensional space correspond to the relative positions of instruments in a musical ensemble producing the melodic lines corresponding to said objects.

32. The method of claim 25 further comprising the steps of:

(6) generating a smooth mathematical function representative of a curve traced by said graph; and

(7) importing said smooth mathematical function into said three-dimensional animation software.

33. The method of claim 32 wherein, in step (5), at least one of said object and said animation technique is further a function of said mathematical function.