Music instruction system

• US 8,629,342 B2
• Filed: 05/07/2010
• Issued: 01/14/2014
• Est. Priority Date: 07/02/2009
• Status: Active Grant

• Current Assignee: Guitar Center Stores Incorporated (Guitar Center Holdings Incorporated)
• Original Assignee: The Way of H, Inc.
• Inventors: Lee, Harold, Dannenberg, Roger B.
• Primary Examiner(s): FLETCHER, MARLON T
• Application Number:
  US12/775,665
• Publication Number:
  US 20110003638A1
• Time in Patent Office: 1,348 Days
• Field of Search: None
• US Class Current: 84/610
• CPC Class Codes: A63F 11/0051 Indicators of values, e.g. score counters
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8047 Music games
• CPC Class Codes: G09B 15/00 Teaching music metronomes G04F5/02
• CPC Class Codes: G09B 15/023 Electrically operated
• CPC Class Codes: G09B 15/04 with sound emitters
• CPC Class Codes: G09B 5/065 Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/0016 Means for indicating which keys, frets or strings are to be actuated, e.g. using lights or leds
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/051 for extraction or detection of onsets of musical sounds or notes, i.e. note attack timings
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/091 for performance evaluation, i.e. judging, grading or scoring the musical qualities or faithfulness of a performance, e.g. with respect to pitch, tempo or other timings of a reference performance
• CPC Class Codes: G10H 2220/005 Non-interactive screen display of musical or status data
• CPC Class Codes: G10H 2220/051 Fret indicator, e.g. for playing guidance on a string instrument or string instrument emulator
• CPC Class Codes: G10H 2220/056 Hand or finger indicator, e.g. for indicating which hand or which specific finger should be used
• CPC Class Codes: G10H 2220/076 String indicator, e.g. on a stringed musical instrument for indicating which string is to be played, plucked or bowed
• CPC Class Codes: G10H 2220/126 for graphical editing of individual notes, parts or phrases represented as variable length segments on a 2D or 3D representation, e.g. graphical edition of musical collage, remix files or pianoroll representations of MIDI-like files
• CPC Class Codes: G10H 2220/151 Musical difficulty level setting or selection
• CPC Class Codes: G10H 3/186 Means for processing the signal picked up from the strings filtering G10H1/12

Abstract

A method includes receiving a user selection of a musical piece; providing performance cues to a user to perform musical events on a musical instrument, wherein the performance cues are synchronized to expert performance data of the musical piece; receiving audio data corresponding to musical events performed by the user on the musical instrument; detecting fundamental frequencies associated with the user-performed musical events; determining an extent to which the user-performed musical events have been correctly or incorrectly performed; providing real-time or near real-time audio feedback and/or visual feedback indicating the extent to which the user-performed musical events have been correctly or incorrectly performed; and using the expert performance data as real-time or near real-time audio and/or video feedback by controlling an output level of the expert performance data output to the user during a session.

Claims

1. A method comprising:

receiving a user selection of a musical piece;

providing performance cues to a user to perform musical events on a musical instrument, wherein the performance cues are synchronized to expert performance data of the musical piece, wherein the expert performance data includes a pre-recorded audio track;

providing the expert performance data to the user;

receiving audio data corresponding to musical events performed by the user on the musical instrument;

detecting fundamental frequencies associated with the user-performed musical events;

determining an extent to which the user-performed musical events have been correctly or incorrectly performed;

providing real-time or near real-time audio feedback, visual feedback and/or tactile feedback indicating the extent to which the user-performed musical events have been correctly or incorrectly performed; and

using the expert performance data as real-time or near real-time audio or real-time or near real-time audio/video feedback by controlling an output level of the expert performance data that includes the pre-recorded audio track that is output to the user during a session.

2. The method of claim 1, further comprising:

automatically adjusting a difficulty level of the musical piece during the session based on the extent to which user-performed musical events have been correctly or incorrectly performed.

3. The method of claim 1, wherein, when using the expert performance data, the user-performed musical events are muted and the output level of the expert performance data is increased when the user-performed musical events have been correctly performed.

4. The method of claim 1, wherein the detecting further comprises:

detecting onsets of the user-performed musical events; and

performing fundamental frequency detection on a set of contiguous blocks in the audio data that do not include the onsets.

5. The method of claim 4, wherein the detecting further comprises:

forming blocks of audio data having a first size to detect the onsets; and

forming blocks of audio data having a second size to detect the fundamental frequencies, wherein the first size is smaller than the second size.

6. The method of claim 4, wherein the detecting further comprises:

calculating a weighted sum of magnitudes associated with frequencies included in the audio data, wherein the weighted sum is calculated based on multipliers having values corresponding to indexes that are multiplied by the magnitudes.

7. The method of claim 1, wherein the determining comprises:

assigning one or more musical characteristic tolerances with respect to musical events associated with the musical piece, wherein each musical characteristic tolerance includes a range of values corresponding to a musical characteristic associated with the musical events; and

determining the extent to which the user-performed musical events are correctly or incorrectly performed based on the musical characteristic tolerances.

8. The method of claim 7, wherein the musical characteristic relates to one of frequency, amplitude, onset time, release time, or duration, and wherein the range of values assigned are based on a difficulty level of the session.

9. The method of claim 1, further comprising:

providing a video image of the user during a performance of the musical piece.

10. The method of claim 1, further comprising:

displaying a scrolling waveform corresponding to an expert performance of the musical piece; and

displaying a scrolling waveform corresponding to the user'"'"'s performance of the musical piece.

11. The method of claim 1, wherein the providing of performance cues comprises:

displaying fingering positions for performing musical events, wherein the displaying includes a representation of a human hand.

12. The method of claim 1, further comprising:

providing score information to the user, wherein the score information includes error tendencies associated with the user'"'"'s performance of the musical piece.

13. A device comprising:

one or more memories to store instructions; and

one or more processors to execute the instructions in the one or more memories to;

receive a user selection of a musical piece;

provide performance cues to a user to perform musical events associated with the musical piece on a musical instrument, wherein the performance cues are synchronized to expert performance data of the musical piece, and wherein the expert performance data includes a pre-recorded audio track;

provide the expert performance data to the user;

receive audio data from the musical instrument, wherein the audio data comprises user-performed musical events;

detect fundamental frequencies associated with the user-performed musical events;

determine an extent to which the user-performed musical events have been correctly or incorrectly performed based on one or more musical characteristic tolerances associated with the musical events; and

provide real-time or near real-time visual feedback and/or audio feedback to the user, by controlling an output level of the expert performance data that includes the pre-recorded audio track, that indicates the extent to which the user-performed musical events have been correctly or incorrectly performed based on the determining.

14. The device of claim 13, wherein the performance cues are synchronized to expert performance data corresponding to an expert performance of the musical piece.

15. The device of claim 14, wherein the one or more processors are to execute instructions to:

provide the expert performance data as visual feedback to the user, wherein a visual clarity of a displaying of the expert performance data is based on the extent to which the user-performed musical events have been played correctly or not, wherein the expert performance data includes a pre-recorded visual track.

16. The device of claim 13, wherein the one or more processors are to execute instructions to:

introduce one or more first signal processing audio effects to an audio output of the user-performed musical events when it is determined that the user-performed musical events have been correctly performed; and

introduce one or more second signal processing audio effects to the output of the user-performed musical events when it is determined that the user-performed musical events have not been correctly performed, wherein the one or more first signal processing audio effects enhance the user-performed musical events and the one or more second signal processing audio effects diminish the user-performed musical events.

17. The device of claim 13, wherein the device comprises at least one of a game system, a computer, or a portable communication device.

18. The device of claim 13, wherein the one or more processors are to execute instructions to:

display scrolling tablatures that include the performance cues, wherein the scrolling tablatures include a graphical representation of the musical instrument and the performance cues include visual cues corresponding to the musical events.

19. The device of claim 13, wherein when detecting the fundamental frequencies, the one or more processors are to execute instructions to:

form first blocks of audio data having a first size to detect onsets of the user-performed musical events; and

form second blocks of audio data having a second size to detect fundamental frequencies, wherein the first size is smaller than the second size, and the second blocks include a set of contiguous first blocks that do not include the onsets.

20. The device of claim 13, wherein each musical characteristic tolerance includes a range of values corresponding to a musical characteristic associated with the musical events, wherein the musical characteristic includes at least one of frequency, amplitude, onset time, release time, rhythm, or duration, and each range of values includes a value corresponding to an expert performance of the musical events, and wherein when determining whether the user-performed musical events have been correctly or incorrectly performed, the one or more processors are to execute instructions to:

evaluate the musical characteristics associated with the user-performed musical events;

compare values corresponding to the musical characteristics associated with the user-performed musical events to the range of values; and

determine an extent to which the user-performed musical events have been correctly or incorrectly performed based on the comparing.

21. The device of claim 13, wherein the one or more processors are to execute instructions to:

map notes performed on the musical instrument as input commands to allow the user to navigate and interact with user interfaces associated with the device.

22. The device of claim 13, wherein the one or more processors are to execute instructions to:

calculate a score corresponding to whether the user-performed musical events have been correctly or incorrectly performed, wherein the score is calculated based on a difficulty level assigned to the musical piece; and

display the score to the user.

23. The device of claim 13, wherein the one or more processors are to execute instructions to:

output to a display one or more of a virtual mentor, an instructor, or a coach that provides audio feedback and visual feedback to the user during a session or a video image of the user during the session.

24. The device of claim 13, wherein the one or more processors are to execute instructions to:

calculate a weighted sum of magnitudes associated with frequencies included with the user-performed musical events, wherein the weighted sum is calculated based on multipliers having values corresponding to indexes that are multiplied by the magnitudes.

25. A computer-readable medium containing instructions executable by at least one processor, the computer-readable medium storing instructions to:

receive a user selection of a musical piece;

provide performance cues to a user to perform musical events associated with the musical piece on a musical instrument, wherein the performance cues are synchronized to expert performance data of the musical piece, wherein the expert performance data includes a pre-recorded audio track;

provide the expert performance data to the user;

receive audio data from the musical instrument, wherein the audio data comprises user-performed musical events;

determine whether to detect fundamental frequencies associated with the user-performed musical events based on at least one of the audio data or the musical instrument;

determine an extent to which the user-performed musical events have been correctly or incorrectly performed; and

provide feedback to the user indicating the extent to which the user-performed musical events have been correctly performed based on the determining, wherein the feedback includes controlling an output level of the expert performance data that includes the pre-recorded audio track that is output to the user during a session.

26. The computer-readable medium of claim 25, further comprising one or more instructions to:

detect fundamental frequencies associated with the user-performed musical events;

calculate a weighted sum of magnitudes associated with frequencies included with the user-performed musical events;

detect onsets of the user-performed musical events;

perform fundamental frequency detection on a set of contiguous blocks of the audio data that do not include the onsets; and

calculate peak estimations associated with frequencies included in the set of contiguous blocks.

27. The computer-readable medium of claim 26, further comprising one or more instructions to:

calculate the weighted sum based on multipliers having values corresponding to indexes that are multiplied by the magnitudes.

28. The computer-readable medium of claim 25, further comprising one or more instructions to:

store performance statistics associated with the user according to a genre of the musical piece.

29. The computer-readable medium of claim 25, further comprising one or more instructions to:

provide different levels of difficulties for performing the musical piece;

receive a user selection of one of the levels of difficulties;

automatically increase a difficulty level of the musical piece during a session when the extent to which the user-performed musical events have been correctly performed exceeds a first threshold value;

automatically decrease a difficulty level of the musical piece during the session when the extent to which the user-performed musical events been incorrectly performed is below a second threshold value; and

automatically end the session when the extent to which the user-performed musical events have been incorrectly performed is below a stoppage threshold value.

30. The computer-readable medium of claim 25, further comprising one or more instructions to:

store a database that includes data related to the user'"'"'s progress occurring over multiple sessions.