System and method for synchronizing a live musical performance with a reference performance

• US 7,164,076 B2
• Filed: 05/14/2004
• Issued: 01/16/2007
• Est. Priority Date: 05/14/2004
• Status: Active Grant

• Current Assignee: Konami Digital Entertainment Incorporated (Konami Holdings Corp.)
• Original Assignee: Konami Digital Entertainment Incorporated (Konami Holdings Corp.)
• Inventors: McHale, Mike, Egozy, Eran B.
• Primary Examiner(s): Donovan; Lincoln
• Assistant Examiner(s): RUSSELL, CHRISTINA MARIE
• Application Number:
  US10/846,366
• Publication Number:
  US 20050252362A1
• Time in Patent Office: 977 Days
• Field of Search: 846/16, 846/10, 846/45, 846/37, 844/77.R, 846/09, 704/270, 434/307.A, 702/182, 369/53
• US Class Current: 84/616
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/091 for performance evaluation, i.e. judging, grading or scoring the musical qualities or faithfulness of a performance, e.g. with respect to pitch, tempo or other timings of a reference performance
• CPC Class Codes: G10H 2220/141 Games on or about music, i.e. based on musical knowledge, e.g. musical multimedia quizzes
• CPC Class Codes: G10H 3/125 Extracting or recognising the pitch or fundamental frequency of the picked up signal

Abstract

In a music video game, a player'"'"'s performance is digitally sampled while the player performs a musical composition. The player'"'"'s performance is compared with a reference performance of the musical composition provided by the music video game. Performance feedback is presented to the player based on the results of the comparison. In some embodiments, sample times associated with digital samples of the player'"'"'s live vocal performance are compared against timestamps of data records embedded or otherwise accompanying the reference performance audio track. Pitch and rhythm information is retrieved from the data record having a timestamp that most closely matches the sample time of interest. The pitch and rhythm data is used to compute pitch and rhythm errors, which are used to generate performance evaluation data. The performance evaluation data is used to present performance feedback to the player while the player is performing the musical composition.

Claims

1. A method of synchronizing a live musical performance with a reference performance, comprising:

retrieving a set of records corresponding to a reference musical performance, the set of records including reference pitches and timestamps for determining positions of the reference pitches in the reference performance;

sampling the live musical performance to produce a sequence of samples, each sample having an associated sample time and pitch value, wherein the sample times of successive samples in the sequence of samples are independent of the timestamps of the records of the reference musical performance;

comparing the sample times of a sequence of respective samples of the live musical performance with the timestamps of the records of the reference performance, including, for each of a plurality of respective samples in the sequence of respective samples, comparing the sample time of the respective sample with a plurality of the timestamps of the records, including a timestamp that is earlier than the sample time of the respective sample and a timestamp that is later than the sample time of the respective sample;

for each respective sample in the plurality of respective samples;

selecting a reference pitch from a record having a timestamp that most closely matches the sample time of the respective sample;

comparing the pitch value of the respective sample with the selected reference pitch; and

scoring the live musical performance based on the results of the comparison.

2. The method of claim 1, wherein the live musical performance is a live vocal performance.

3. The method of claim 1, wherein determining a pitch value further comprises:

determine a periodicity component from a set of input samples;

converting the periodicity component to a frequency component; and

converting the frequency component into a semitone value or index representative of a pitch in the live musical performance.

4. The method of claim 1, further comprising:

determining a itch error from the pitch value of a respective sample and a corresponding selected reference pitch;

comparing the pitch error with a target range;

scoring the live musical performance positively if the pitch error is less than the target range; and

scoring the live musical performance negatively if the pitch error exceeds the target range.

5. The method of claim 4, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

6. The method of claim 4, wherein the target range is based at least in part on a player profile associated with the live musical performance.

7. The method of claim 1, further comprising:

determining a rhythm error based at least in part on the sample time of a respective sample;

comparing the rhythm error with a target range;

scoring the live musical performance positively if the rhythm error is less than the target range; and

scoring the live musical performance negatively if the rhythm error exceeds the target range.

8. The method of claim 7, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

9. The method of claim 7, wherein the target range is based at least in part on a profile of a player associated with the live musical performance.

10. The method of claim 1, further comprising;

adjusting the pitch value of a respective sample by one or more octaves;

comparing the adjusted pitch value with a target threshold;

scoring the live musical performance positively if the adjusted pitch value is less than the target threshold; and

scoring the live musical performance negatively if the adjusted pitch value exceeds the target threshold.

11. A system for synchronizing a live musical performance with a reference performance, comprising:

memory for storing a set of records corresponding to the reference performance, the set of records including reference pitches and timestamps for determining positions of the reference pitches in the reference performance;

a digital processor adapted to receive a sequence of samples of the live musical performance, each sample having an associated sample time and pitch value, wherein the sample times of successive samples in the sequence of samples are independent of the timestamps of the records of the reference musical performance;

a compare module coupled to the digital processor and configured to compare the sample times of a sequence of respective samples of the live musical performance with the timestamps of the records of the reference performance, including, for each of a plurality of respective samples in the sequence of respective samples, comparing the sample time of the respective sample with a plurality of the timestamps of the records, including a timestamp that is earlier than the sample time of the respective sample and a timestamp that is later than the sample time of the respective sample;

the compare module further configured to process each respective sample in the plurality of respective samples by selecting a reference pitch from a record having a timestamp that most closely matches the sample time of the respective sample, and comparing the pitch value of the respective sample with the selected reference pitch; and

a performance evaluation module coupled to the compare module and configured to score the live musical performance based on the results of comparisons performed by the compare module.

12. The system of claim 11, wherein the live musical performance is a live vocal performance.

13. The system of claim 11, wherein the digital processor determines a pitch value from a periodicity component associated with the samples, converts the periodicity component into a frequency component, and converts the frequency component into a semitone value representative of a pitch in the live musical performance.

14. The system of claim 11, wherein the digital processor determines a pitch error from the pitch value of a respective sample and a corresponding selected reference pitch, the compare module compares the pitch error with a target range, and the performance evaluation module scores the live musical performance positively if the pitch error is less than the target range, and scores the live musical performance negatively if the pitch error exceeds the target range.

15. The system of claim 14, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

16. The system of claim 14, wherein the target range is based at least in part on a player profile associated with the live musical performance.

17. The system of claim 11, wherein the compare module determines a rhythm error based at least in part on the sample time of a respective sample and a time window, compares the rhythm error with a target range, scores the live musical performance positively if the rhythm error is less than the target range, and scores the live musical performance negatively if the rhythm error exceeds the target range.

18. The system of claim 17, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

19. The system of claim 17, wherein the target range is based at least in part on a profile of a player associated with the live musical performance.

20. The system of claim 11, further comprising:

an octave analyzer coupled to the digital processor and configured to adjust the pitch value of a respective sample by one or more octaves and compare the adjusted, selected pitch value with a target threshold.

21. A computer-readable medium having stored thereon instructions, which, when executed by a processor, causes the processor to perform the operations of:

retrieving a set of records corresponding to a reference musical performance, the set of records including reference pitches and timestamps for determining positions of the reference pitches in the reference performance;

sampling the live musical performance to produce a sequence of samples, each sample having an associated sample time and pitch value, wherein the sample times of successive samples in the sequence of samples are independent of the timestamps of the records of the reference musical performance;

comparing the sample times of a sequence of respective samples of the live musical performance with the timestamps of the records of the reference performance, including, for each of a plurality of respective samples in the sequence of respective samples, comparing the sample time of the respective sample with a plurality of the timestamps of the records, including a timestamp that is earlier than the sample time of the respective sample and a timestamp that is later than the sample time of the respective sample;

for each respective sample in the plurality of respective samples;

selecting a reference pitch from a record having a timestamp that most closely matches the sample time of the respective sample;

comparing the pitch value of the respective sample with the selected reference pitch; and

scoring the live musical performance based on the results of the comparison.

22. The computer-readable medium of claim 21, wherein the live musical performance is a live vocal performance.

23. The computer-readable medium of claim 21, wherein determining a pitch value further comprises:

determine a periodicity component from a set of input samples;

converting the periodicity component to a frequency component; and

converting the periodicity component into a semitone value representative of a pitch in the live musical performance.

24. The computer-readable medium of claim 21, further comprising:

determining a pitch error from the pitch value of a respective sample and a corresponding selected reference pitch;

comparing the pitch error with a target range;

scoring the live musical performance positively if the pitch error does not exceed the target range; and

scoring the live musical performance negatively if the pitch error exceeds the target range.

25. The computer-readable medium of claim 24, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

26. The computer-readable medium of claim 24, wherein the target range is based at least in part on a player profile associated with the live musical performance.

27. The computer-readable medium of claim 21, further comprising:

determining a rhythm error based at least in part on the sample time of a respective sample;

comparing the rhythm error with a target range;

scoring the live musical performance positively if the rhythm error is less than the target range; and

scoring the live musical performance negatively if the rhythm error exceeds the target range.

28. The computer-readable medium of claim 27, wherein the target range is based at least in part on a level of difficulty associated with the reference performance.

29. The computer-readable medium of claim 27, wherein the target range is based at least in part on a profile of a player associated with the live musical performance.

30. The computer-readable medium of claim 21, further comprising;

adjusting the pitch value of a respective sample by one or more octaves;

comparing the adjusted pitch value with a target threshold; and

scoring the live musical performance positively if the adjusted pitch value is less than the target threshold; and

scoring the live musical performance negatively if the adjusted pitch value exceeds the target threshold.