System and method for synchronizing a live musical performance with a reference performance

• US 20050252362A1
• Filed: 05/14/2004
• Published: 11/17/2005
• Est. Priority Date: 05/14/2004
• Status: Active Grant

• Current Assignee: Konami Digital Entertainment Incorporated (Konami Holdings Corp.)
• Original Assignee: Konami Digital Entertainment Incorporated (Konami Holdings Corp.)
• Inventors: McHale, Mike, Egozy, Eran B.
• Granted Patent: US 7,164,076 B2
• Time in Patent Office: Days
• US Class Current: 84/616
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/091 for performance evaluation, i.e. judging, grading or scoring the musical qualities or faithfulness of a performance, e.g. with respect to pitch, tempo or other timings of a reference performance
• CPC Class Codes: G10H 2220/141 Games on or about music, i.e. based on musical knowledge, e.g. musical multimedia quizzes
• CPC Class Codes: G10H 3/125 Extracting or recognising the pitch or fundamental frequency of the picked up signal

Abstract

In a music video game, a player'"'"'s performance is digitally sampled while the player performs a musical composition. The player'"'"'s performance is compared with a reference performance of the musical composition provided by the music video game. Performance feedback is presented to the player based on the results of the comparison. In some embodiments, sample times associated with digital samples of the player'"'"'s live vocal performance are compared against timestamps of data records embedded or otherwise accompanying the reference performance audio track. Pitch and rhythm information is retrieved from the data record having a timestamp that most closely matches the sample time of interest. The pitch and rhythm data is used to compute pitch and rhythm errors, which are used to generate performance evaluation data. The performance evaluation data is used to present performance feedback to the player while the player is performing the musical composition.

Claims

1. A method of synchronizing a live musical performance with a reference performance, comprising:

retrieving a set of records corresponding to a reference musical performance, the set of records including reference pitches and timestamps for determining positions of the reference pitches in the musical performance;

storing the records;

retrieving a sample and corresponding sample time of a live vocal performance;

determining a pitch value from the sample;

comparing the sample time with the timestamps of the records;

selecting a reference pitch from a record having a timestamp that most closely matches the sample time;

comparing the pitch value with the selected reference pitch; and

scoring the live musical performance based on the results of the comparison.

2. The method of claim 1, wherein the live musical performance is a live vocal performance.

3. The method of claim 1, wherein determining a pitch value further comprises:

determine a periodicity component from a set of input samples;

converting the periodicity component to a frequency component; and

converting the frequency component into a semitone value or index representative of a pitch in the live musical performance.

4. The method of claim 1, further comprising:

determining a pitch error from the pitch value and reference pitch;

comparing the pitch error with a target range;

scoring the live musical performance positively if the pitch error is less than the target range; and

scoring the live musical performance negatively if the pitch error exceeds the target range.

5. The method of claim 4, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

6. The method of claim 4, wherein the target range is based at least in part on a player profile associated with the live musical performance.

7. The method of claim 1, further comprising:

determining a rhythm error based at least in part on the sample time;

comparing the rhythm error with a target range;

scoring the live musical performance positively if the rhythm error is less than the target range; and

scoring the live musical performance negatively if the rhythm error exceeds the target range.

8. The method of claim 7, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

9. The method of claim 7, wherein the target range is based at least in part on a profile of a player associated with the live musical performance.

10. The method of claim 1, further comprising;

adjusting the pitch value by one or more octaves;

comparing the adjusted pitch value with a target threshold;

scoring the live musical performance positively if the adjusted pitch value is less than the target threshold; and

scoring the live musical performance negatively if the adjusted pitch value exceeds the target threshold.

14. The system of claim 1, wherein the digital processor determines a pitch error from the pitch value and reference pitch, the compare module compares the pitch error with a target range, and the performance evaluation module scores the live musical performance positively if the pitch error is less than the target range, and scores the live musical performance negatively if the pitch error exceeds the target range.

15. The system of claim 14, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

16. The system of claim 14, wherein the target range is based at least in part on a player profile associated with the live musical performance.

11. A system for synchronizing a live musical performance with a reference performance, comprising:

a data extractor for extracting from a data stream a set of records corresponding to a reference musical performance, the set of records including reference pitches and timestamps for determining positions of the reference pitches in the musical performance;

a buffer coupled to the data extractor and configured for storing the set of records;

a digital processor adapted to receive samples of a live musical performance, the digital processor configured to determine a pitch value from the samples;

a compare module coupled to the digital processor and configured to compare the sample time with the timestamps of the records, select a reference pitch from a record having a timestamp that most closely matches the sample time, and compare the pitch value with the selected reference pitch; and

a performance evaluation module coupled to the compare module and configured to score the live musical performance based on the results of the comparison.

12. The system of claim 11, wherein the live musical performance is a live vocal performance.

13. The system of claim 11, wherein the digital processor determines a pitch value from a periodicity component associated with the samples, converts the periodicity component into a frequency component, and converts the frequency component into a semitone value representative of a pitch in the live musical performance.

17. The system of claim 11, wherein the compare module determines a rhythm error based at least in part on the sample time and a time window, compares the rhythm error with a target range, scores the live musical performance positively if the rhythm error is less than the target range, and scores the live musical performance negatively if the rhythm error exceeds the target range.

18. The system of claim 17, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

19. The system of claim 17, wherein the target range is based at least in part on a profile of a player associated with the live musical performance.

20. The system of claim 11, further comprising:

an octave analyzer coupled to the digital processor and configured to adjust the selected pitch value by one or more octaves and compare the adjusted, selected pitch value with a target threshold.

21. A computer-readable medium having stored thereon instructions, which, when executed by a processor, causes the processor to perform the operations of:

retrieving a set of records corresponding to a reference musical performance, the set of records including reference pitches and timestamps for determining positions of the reference pitches in the musical performance;

storing the records;

retrieving a sample and corresponding sample time of a live vocal performance;

determining a pitch value from the sample;

comparing the sample time with the timestamps of the records;

selecting a reference pitch from a record having a timestamp that most closely matches the sample time;

comparing the pitch value with the selected reference pitch; and

scoring the live musical performance based on the results of the comparison.

22. The computer-readable medium of claim 21, wherein the live musical performance is a live vocal performance.

23. The computer-readable medium of claim 21, wherein determining a pitch value further comprises:

determine a periodicity component from a set of input samples;

converting the periodicity component to a frequency component; and

converting the periodicity component into a semitone value representative of a pitch in the live musical performance.

24. The computer-readable medium of claim 21, further comprising:

determining a pitch error from the pitch value and reference pitch;

comparing the pitch error with a target range;

scoring the live musical performance positively if the pitch error does not exceed the target range; and

scoring the live musical performance negatively if the pitch error exceeds the target range.

25. The computer-readable medium of claim 24, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

26. The computer-readable medium of claim 24, wherein the target range is based at least in part on a player profile associated with the live musical performance.

27. The computer-readable medium of claim 21, further comprising:

determining a rhythm error based at least in part on the sample time;

comparing the rhythm error with a target range;

scoring the live musical performance positively if the rhythm error is less than the target range; and

scoring the live musical performance negatively if the rhythm error exceeds the target range.

28. The computer-readable medium of claim 27, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

29. The computer-readable medium of claim 27, wherein the target range is based at least in part on a profile of a player associated with the live musical performance.

30. The computer-readable medium of claim 21, further comprising;

adjusting the pitch value by one or more octaves;

comparing the adjusted pitch value with a target threshold; and

scoring the live musical performance positively if the adjusted pitch value is less than the target threshold; and

scoring the live musical performance negatively if the adjusted pitch value exceeds the target threshold.