Pitch-correction of vocal performance in accord with score-coded harmonies

• US 9,852,742 B2
• Filed: 10/17/2014
• Issued: 12/26/2017
• Est. Priority Date: 04/12/2010
• Status: Active Grant

• Current Assignee: Smule, Inc.
• Original Assignee: Smule, Inc.
• Inventors: Cook, Perry R., Lazier, Ari, Lieber, Tom, Kirk, Turner Evan
• Primary Examiner(s): He, Jialong
• Application Number:
  US14/517,647
• Publication Number:
  US 20150170636A1
• Time in Patent Office: 1,166 Days
• CPC Class Codes: G10H 1/366 with means for modifying or correcting the external signal, e.g. pitch correction, reverberation, changing a singer's voice
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/331 Note pitch correction, i.e. modifying a note pitch or replacing it by the closest one in a given scale
• CPC Class Codes: G10H 2240/251 Mobile telephone transmission, i.e. transmitting, accessing or controlling music data wirelessly via a wireless or mobile telephone receiver, analog or digital, e.g. DECT GSM, UMTS smartphone, PDA or palmtop used for musical purposes G10H2230/015; mobile ringtone G10H2230/021
• CPC Class Codes: G10L 13/0335 Pitch control
• CPC Class Codes: G10L 21/013 Adapting to target pitch
• CPC Class Codes: Y10S 84/04 Chorus; ensemble; celeste

Abstract

Despite many practical limitations imposed by mobile device platforms and application execution environments, vocal musical performances may be captured and continuously pitch-corrected for mixing and rendering with backing tracks in ways that create compelling user experiences. In some cases, the vocal performances of individual users are captured on mobile devices in the context of a karaoke-style presentation of lyrics in correspondence with audible renderings of a backing track. Such performances can be pitch-corrected in real-time at a portable computing device (such as a mobile phone, personal digital assistant, laptop computer, notebook computer, pad-type computer or netbook) in accord with pitch correction settings. In some cases, pitch correction settings include a score-coded melody and/or harmonies supplied with, or for association with, the lyrics and backing tracks. Harmonies notes or chords may be coded as explicit targets or relative to the score coded melody or even actual pitches sounded by a vocalist, if desired.

Claims

1. A method comprising:

using a first portable computing device for vocal performance capture, the portable computing device having a display, a microphone interface and a communications interface;

retrieving via the communications interface, a vocal score temporally synchronizable with a corresponding backing track and lyrics, the vocal score encoding (i) a sequence of notes for a vocal melody and (ii) at least a first set of harmony notes for at least some portions of the vocal melody;

at the first portable computing device, audibly rendering the backing track and concurrently presenting corresponding portions of the lyrics on the display in temporal correspondence therewith;

at the first portable computing device, capturing and pitch correcting a vocal performance of a first user in accord with the score-encoded vocal melody to produce a first version of the first user'"'"'s vocal performance;

pitch shifting at least some portions of the first user'"'"'s captured vocal performance in accord with the score-encoded harmony notes to produce at least a second version of the first user'"'"'s vocal performance; and

mixing either or both of first and second versions of the user'"'"'s vocal performance with the backing track, wherein a second user'"'"'s vocal performance is captured and pitch corrected at a remote second portable computing device prior to audibly rendering the backing track at the first portable computing device, and the backing track includes the second user'"'"'s vocal performance.

2. The method of claim 1, further comprising:

retrieving the backing track from a remote content server via a data communications interface.

3. A method for use in connection with vocal performance capture, the method comprising:

retrieving a computer readable media encoding of a vocal score temporally synchronizable with a corresponding backing track and lyrics, the vocal score encoding (i) a sequence of notes for a vocal melody and (ii) at least a first set of harmony notes for at least some portions of the vocal melody;

audibly rendering the backing track and concurrently presenting corresponding portions of the lyrics on a display in temporal correspondence with the audible rendering;

capturing a vocal performance of a user and pitch correcting the captured performance in accord with the score-encoded vocal melody to produce a first version of the user'"'"'s vocal performance;

pitch shifting at least some portions of the user'"'"'s captured vocal performance in accord with the score-encoded harmony notes to produce at least a second version of the user'"'"'s vocal performance; and

adding a temporal delay to the second version of the user'"'"'s vocal performance, wherein the audible rendering is in real-time correspondence with the user'"'"'s vocal performance and mixes either or both of the first and temporally delayed second versions of the user'"'"'s vocal performance with the backing track.

4. The method of claim 3, further comprising:

mixing at least the first and temporally delayed second versions of the user'"'"'s vocal performance with the backing track, wherein the resulting mixed performance includes both pitch corrected vocal melody and accompanying pitch shifted vocal harmony versions of the user'"'"'s vocal performance.

5. The method of claim 4, wherein for at least some portions of the vocal melody, the vocal score encodes a second set of harmony notes, the method further comprising:

pitch shifting at least some portions of the user'"'"'s captured vocal performance in accord with the second set of score-encoded harmony notes to produce at least a third version of the user'"'"'s vocal performance,

wherein the resulting mixed performance further includes the third version of the user'"'"'s vocal performance as an additional pitch corrected vocal harmony.

6. The method of claim 5,

wherein one or more of (i) the pitch shifting to produce a second version, (ii) the pitch shifting to produce a third version and (iii) the mixing of versions of the user'"'"'s vocal performance to produce a resulting mixed performance are performed using a remote service platform physically separated from the user, but communicatively coupled to computational implementations at a portable computing device of the vocal performance capture and local audible rendering.

7. The method of claim 4, further comprising:

transmitting to a remote content server via a communications interface, an audio encoding of one or more of (i) the captured vocal performance of the user, (ii) a pitch corrected vocal melody or harmony version of the user'"'"'s vocal performance, and (iii) the mixed performance including both pitch corrected vocal melody and accompanying pitch corrected vocal harmony versions of the user'"'"'s vocal performance.

8. The method of claim 7, further comprising:

geocoding the transmitted audio encoding to, in correspondence with a remote audible rendering of the transmitted audio encoding or a derivative mix thereof, identify a geographic origin of the user'"'"'s vocal performance.

9. The method of claim 8,

wherein the identification of geographic origin is by display animation suggestive of a performance emanating from a particular location on a globe.

10. The method of claim 8, further comprising:

capturing and conveying back to the remote server one or more of (i) listener comment on and (ii) ranking of a mixed performance for inclusion as metadata in association with subsequent supply and rendering thereof.

11. The method of claim 3,

wherein at least the vocal capture, pitch-correction to vocal melody and the audible rendering in real-time correspondence are performed at a portable computing device.

12. The method of claim 11, wherein the portable computing device is selected from the group of:

a mobile phone;

a personal digital assistant;

a laptop computer, notebook computer, tablet computer or netbook.

13. The method of claim 3,

wherein the pitch correcting and pitch shifting are based on continuous time-domain estimation of pitch for the user'"'"'s captured vocal performance.

14. The method of claim 13,

wherein the continuous time-domain pitch estimation includes computing, for a current block of a sampled signal corresponding to the user'"'"'s captured vocal performance, a lag-domain periodogram.

15. The method of claim 14, wherein the lag-domain periodogram computation includes, for an analysis window of the sampled signal, at least one of:

evaluations of an average magnitude difference function (AMDF) for a range of lags; and

evaluations of an autocorrelation function for a range of lags.

16. The method of claim 3, further comprising:

evaluating throughout the user'"'"'s vocal performance whether the user'"'"'s current vocals more closely correspond to the score-encoded vocal melody or to a score-encoded harmony; and

based on the evaluation, synthesizing either remaining portions of a score-coded chord as pitch-shifted variants of the captured vocal performance or a harmonically correct set of notes rooted on corrected pitch of the user'"'"'s vocal performance.

17. The method of claim 3, further comprising:

retrieving the backing track from a remote content server via a data communications interface.

18. The method of claim 3,

wherein the backing track is locally stored, and

wherein the retrieving identifies the vocal score temporally synchronizable with the corresponding backing track and lyrics using an identifier ascertainable from the locally stored backing track.

19. A vocal performance capture and processing system comprising:

a portable computing device having a display;

a microphone interface;

an audio transducer interface;

a data communications interface;

user interface code executable on the portable computing device to capture user interface gestures selective for a backing track and to initiate retrieval of at least a vocal score corresponding thereto, the vocal score encoding (i) a sequence of notes for a vocal melody and (ii) at least a first set of harmony notes for at least some portions of the vocal melody;

the user interface code further executable to capture user interface gestures to initiate (i) audible rendering of the backing track, (ii) concurrent presentation lyrics on the display and (iii) capture of the user'"'"'s vocal performance using the microphone interface;

first pitch correction code executable on the portable computing device to, concurrent with said audible rendering, continuously pitch correct the user'"'"'s vocal performance in accord with the score-encoded vocal melody to produce a first version of the user'"'"'s vocal performance;

second pitch correction code executable to continuously pitch shift at least some portions of the user'"'"'s vocal performance in accord with the score-encoded harmony notes to produce at least a second version of the user'"'"'s vocal performance;

third pitch correction code executable to add a temporal delay to the second version of the user'"'"'s vocal performance; and

a local rendering pipeline executable on the portable computing device to mix either or both of first and temporally delayed second versions of the user'"'"'s vocal performance with the backing track and render a resulting mixed performance via the audio transducer interface in real-time correspondence with the user'"'"'s vocal performance.

20. The vocal performance capture and processing system of claim 19,

wherein the second pitch correction code is executable using a remote service platform physically separated from the user but communicatively coupled to receive from the portable computing device a signal encoding the user'"'"'s vocal performance.

21. The vocal performance capture and processing system of claim 19,

wherein the second pitch correction code is executable on the portable computing device.

22. The vocal performance capture and processing system of claim 19, further comprising:

a rendering pipeline executable using a remote service platform physically separated from the user but communicatively coupled to receive from the portable computing device a signal encoding the user'"'"'s vocal performance and to supply a resulting mixed performance, the rendering pipeline executable to mix at least the first and temporally delayed second versions of the user'"'"'s vocal performance with the backing track, such that the resulting mixed performance includes the user'"'"'s own vocal performance captured in correspondence with the lyrics and backing track, but pitch-corrected and harmonized in accord with the vocal score.

23. The vocal performance capture and processing system of claim 19,

wherein the pitch correction code includes a time-domain implementation of pitch estimation.

24. The vocal performance capture and processing system of claim 23,

wherein the time-domain implementation of pitch estimation includes code executable to compute, for a current block of a sampled signal corresponding to the user'"'"'s captured vocal performance, a lag-domain periodogram.

25. The vocal performance capture and processing system of claim 24, wherein the lag-domain periodogram computation includes, for an analysis window of the sampled signal, at least one of:

evaluations of an average magnitude difference function (AMDF) for a range of lags; and

evaluations of an autocorrelation function for a range of lags.

26. The vocal performance capture and processing system of claim 19, further comprising:

code executable on the portable computing device (i) to evaluate throughout the user'"'"'s vocal performance whether the user'"'"'s current vocals more closely correspond to the score-encoded vocal melody or to a score-encoded harmony and (ii) based on the evaluation, to synthesize either remaining portions of a score-coded chord as pitch-shifted variants of the captured vocal performance or a harmonically correct set of notes rooted on corrected pitch of the users vocal performance.

27. The vocal performance capture and processing system of claim 19,

wherein the portable computing device further includes local storage,

wherein the initiated retrieval includes checking instances, if any, of the vocal score information in the local storage against instances available from a remote server and retrieving from the remote server if instances in local storage are unavailable or out-of-date.

28. A computer program product encoded in one or more media, the computer program product including instructions executable on a processor of the portable computing device to cause the portable computing device to:

retrieve via a communications interface, a vocal score temporally synchronizable with a corresponding backing track and lyrics, the vocal score encoding (i) a sequence of notes for a vocal melody and (ii) at least a first set of harmony notes for at least some portions of the vocal melody;

audibly render the backing track and present in temporal correspondence therewith corresponding portions of the lyrics on a display of the portable computing device;

capture and pitch correct a vocal performance of the user in accord with the score-encoded vocal melody to produce a first version of the user'"'"'s vocal performance;

at least initiate pitch shift of at least some portions of the user'"'"'s captured vocal performance in accord with the score-encoded harmony notes to produce at least a second version of the user'"'"'s vocal performance; and

add a temporal delay to the second version of the user'"'"'s vocal performance,

wherein the audible rendering is in real-time correspondence with the user'"'"'s vocal performance and mixes either or both of first and temporally delayed second versions of the user'"'"'s vocal performance with the backing track.

29. The computer program product of claim 28, the instructions encoded therein being executable on the processor of the portable computing device to further cause the portable computing device to:

mix at least the first and temporally delayed second versions of the user'"'"'s vocal performance with the backing track, wherein the resulting mixed performance includes both pitch corrected vocal melody and accompanying pitch shifted vocal harmony versions of the user'"'"'s vocal performance.

30. The computer program product of claim 28,

wherein the pitch correcting and pitch shifting are provided using a subset of the instructions executable on the processor of the portable computing device to provide continuous time-domain estimation of pitch for the user'"'"'s captured vocal performance.

31. The computer program product of claim 28,

wherein the pitch shifting to produce at least the second version of the user'"'"'s vocal performance is initiated from the portable computing device and performed, at least in part, using code executed on a remote service platform physically separated from the portable computing device but responsive to the initiation.