CONTINUOUS SCORE-CODED PITCH CORRECTION

• US 20110144982A1
• Filed: 09/04/2010
• Published: 06/16/2011
• Est. Priority Date: 12/15/2009
• Status: Active Grant

• Current Assignee: Smule, Inc.
• Original Assignee: Smule, Inc.
• Inventors: Cook, Perry R., Smith, Jeffrey C., Ljungström, Mattias, Fiebrink, Rebecca A., Wang, Ge, Salazar, Spencer
• Granted Patent: US 9,147,385 B2
• Time in Patent Office: Days
• US Class Current: 704/207
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G10H 1/366 with means for modifying or correcting the external signal, e.g. pitch correction, reverberation, changing a singer's voice
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/251 Chorus, i.e. automatic generation of two or more extra voices added to the melody, e.g. by a chorus effect processor or multiple voice harmonizer, to produce a chorus or unison effect, wherein individual sounds from multiple sources with roughly the same timbre converge and are perceived as one
• CPC Class Codes: G10H 2210/331 Note pitch correction, i.e. modifying a note pitch or replacing it by the closest one in a given scale
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications
• CPC Class Codes: G10H 2240/125 Library distribution, i.e. distributing musical pieces from a central or master library
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/211 Wireless transmission, e.g. of music parameters or control data by radio, infrared or ultrasound beam G10H2220/405
• CPC Class Codes: G10H 2240/251 Mobile telephone transmission, i.e. transmitting, accessing or controlling music data wirelessly via a wireless or mobile telephone receiver, analog or digital, e.g. DECT GSM, UMTS smartphone, PDA or palmtop used for musical purposes G10H2230/015; mobile ringtone G10H2230/021
• CPC Class Codes: G10L 21/013 Adapting to target pitch
• CPC Class Codes: G10L 21/0356 for synchronising with other signals, e.g. video signals

Abstract

Vocal musical performances may be captured and continuously pitch-corrected at a mobile device for mixing and rendering with backing tracks in ways that create compelling user experiences. In some cases, the vocal performances of individual users are captured in the context of a karaoke-style presentation of lyrics in correspondence with audible renderings of a backing track. Such performances can be pitch-corrected in real-time at the mobile device in accord with pitch correction settings. In some cases, such pitch correction settings code a particular key or scale for the vocal performance or for portions thereof. In some cases, pitch correction settings include a score-coded melody sequence of note targets supplied with, or for association with, the lyrics and/or backing track. In some cases, pitch correction settings are dynamically variable based on gestures captured at a user interface.

Claims

1. A method comprising:

using a portable computing device for vocal performance capture, the portable computing device having a display, a microphone interface and a data communications interface;

retrieving via the data communications interface both (i) lyrics and (ii) pitch correction settings, the retrieved lyrics and pitch correction settings respectively coding temporal correspondence of particular lyrics and of particular variations in pitch correction settings with particular points in a backing track;

at the portable computing device, audibly rendering a first encoding of the backing track and concurrently presenting the lyrics on the display in accord with the temporal correspondence coding thereof;

at the portable computing device, capturing and continuously pitch correcting a vocal performance of a user in accord with the coded temporal variations in pitch correction settings; and

mixing the pitch-corrected vocal performance into the audible rendering of the backing track at the portable computing device, wherein the resulting mixed performance includes the user'"'"'s own vocal performance captured in correspondence with the lyrics and backing track, but pitch-corrected in accord with the coded temporal variations in pitch correction settings.

2. The method of claim 1, further comprising:

transmitting from the portable computing device to a remote server via the data communications interface, an audio encoding of the pitch-corrected vocal performance for mixing with a second encoding of the backing track.

3. The method of claim 2, further comprising:

at the remote server, receiving and mixing the pitch-corrected vocal performance with the second encoding of the backing track to produce a second mixed performance; and

transcoding the second mixed performance to a storage or transmission coding format suitable for retrieval by, or interchange with, devices of other users in a community thereof.

4. The method of claim 2,

wherein the mixing with the second encoding of the backing track is performed at a remote device at which the resulting mixed performance is audibly rendered.

5. The method of claim 2,

wherein the backing track includes either or both of instrumentals and backing vocals, and

wherein the first encoding of the backing track audibly rendered at the portable computing device in correspondence with the lyrics is a monophonic scratch version, and the second encoding of the backing track is a polyphonic version of higher quality or fidelity than the first encoding.

6. The method of claim 1, further comprising:

retrieving via the data communications interface (iii) the first encoding of the backing track.

7. The method of claim 1, further comprising:

transmitting from the portable computing device to a remote content server via the data communications interface, an audio encoding of one or more of (i) the captured vocal performance of the user and (ii) the pitch-corrected vocal performance of the user.

8. The method of claim 7, further comprising:

transmitting to the remote content server via the data communications interface an identifier for the backing track audibly rendered at the portable computing device.

9. The method of claim 7, further comprising:

varying the pitch correction settings dynamically during the vocal performance capture in correspondence with user interface gestures by the user; and

transmitting to the remote content server via the data communications interface the temporally coded pitch correction settings as dynamically varied by the user.

10. The method of claim 1,

wherein the retrieved pitch correction settings include a vocal score that encodes one or more keys in which respective portions of the vocals are to be performed.

11. The method of claim 1,

wherein the retrieved pitch correction settings include a vocal score that encodes a sequence of note targets to which the user'"'"'s vocal performance is to be pitch-corrected.

12. The method of claim 1, wherein the portable computing device is selected from the group of:

a mobile phone;

a personal digital assistant;

a laptop computer, notebook computer, pad-type device or netbook.

13. The method of claim 1, further comprising:

audibly rendering a second mixed performance at the portable computing device, wherein the second mixed performance includes an encoding of a pitch-corrected vocal performance captured and pitch-corrected at a second remote device and mixed with the backing track.

14. The method of claim 2, further comprising:

geocoding the transmitted audio encoding; and

displaying a geographic origin for, and in correspondence with audible rendering of, a third mixed performance of a pitch-corrected vocal performance captured and pitch-corrected at a third remote device and mixed with the backing track, the third mixed performance received via the communications interface directly or indirectly from a third remote device.

15. The method of claim 14,

wherein the display of geographic origin is by display animation suggestive of a performance emanating from a particular location on a globe.

16. The method of claim 14, further comprising:

capturing and conveying back to the remote server one or more of (i) listener comment on and (ii) ranking of the third mixed performance for inclusion as metadata in association with subsequent supply and rendering thereof.

17. The method of claim 2, further comprising:

capturing and transmitting with the audio encoding, video that is performance synchronized with the captured pitch-corrected vocals.

18. A portable computing device comprising:

a display;

a microphone interface;

an audio transducer interface;

a data communications interface;

user interface code executable on the portable computing device to capture user interface gestures selective for a backing track and to initiate retrieval of at least (i) lyrics and (ii) pitch correction settings corresponding thereto, the retrieved lyrics and pitch correction settings respectively coding temporal correspondence of particular lyrics and of particular variations in pitch correction settings with particular points in the backing track;

the user interface code further executable to capture user interface gestures to initiate (i) audible rendering a first encoding of the backing track, (ii) concurrent presentation the lyrics on the display and (iii) capture of the user'"'"'s vocal performance using the microphone interface;

pitch correction code executable on the portable computing device to, concurrent with said audible rendering, continuously pitch correct the user'"'"'s vocal performance in correspondence with the retrieved, temporally-coded variations in pitch correction settings; and

a rendering pipeline that mixes the user'"'"'s pitch-corrected vocal performance into the audible rendering of the backing track, such that the resulting mixed performance includes the user'"'"'s own vocal performance captured in correspondence with the lyrics and backing track, but pitch-corrected in accord with the temporally-coded variations in pitch correction settings.

19. The portable computing device of claim 18, further comprising:

transmit code executable on the portable computing device to transmit an audio encoding of the pitch-corrected vocal performance to a remote server, via the data communications interface, for mixing with a second encoding of the backing track.

20. The portable computing device of claim 18,

further comprising local storage,

wherein the initiated retrieval includes checking instances, if any, of the lyrics and pitch correction settings in the local storage against instances available from a remote server and retrieving from the remote server if instances in local storage are unavailable or out-of-date.

21. The portable computing device of claim 18,

the user interface code further executable to initiate retrieval of the first encoding of the backing track.

22. A computer program product encoded in one or more media, the computer program product including instructions executable on a processor of the portable computing device to cause the portable computing device to:

retrieve via the data communications interface both (i) lyrics and (ii) pitch correction settings, the retrieved lyrics and pitch correction settings respectively coding temporal correspondence of particular lyrics and of particular variations in pitch correction settings with particular points in a backing track;

audibly render a first encoding of the backing track and concurrently present the lyrics on the display in accord with the temporal correspondence coding thereof;

capture and continuously pitch correct a vocal performance of a user in accord with the coded temporal variations in pitch correction settings; and

mix the pitch-corrected vocal performance into the audible rendering of the backing track at the portable computing device, wherein the resulting mixed performance includes the user'"'"'s own vocal performance captured in correspondence with the lyrics and backing track, but pitch-corrected in accord with the coded temporal variations in pitch correction settings.

23. The computer program product of claim 22, the instructions encoded therein being executable on a processor of the portable computing device to further cause the portable computing device to transmit to a remote server via the data communications interface, an audio encoding of the pitch-corrected vocal performance for mixing with a second encoding of the backing track.

24. A method comprising:

from a content server, responsive to a user selection, supplying a first portable computing device with (i) a first encoding of a backing track, (ii) associated lyrics and (iii) pitch correction settings, the associated lyrics and pitch correction settings respectively coding temporal correspondence of particular lyrics and of particular variations in pitch correction settings with particular points in the backing track;

at the content server, receiving an encoding of a captured and pitch-corrected vocal performance of the user, wherein the pitch-corrected vocal performance corresponds to the supplied backing track, lyrics and pitch correction settings;

mixing the received pitch-corrected vocal performance with a second encoding of the backing track; and

thereafter supplying the resulting mixed performance for audible rendering at a second portable computing device.

25. The method of claim 24, further comprising:

mixing the received pitch corrected vocal performance with a third encoding of the backing track as an alternative mixed performance, the third encoding being of higher quality or fidelity than the second; and

supplying the alternative mixed performance for audible rendering at a third remote device.

26. The method of claim 25, further comprising:

selecting one of the resulting mixed performance and the alternative mixed performance for supply to a fourth remote device based on characteristics of communications channel thereto.

27. The method of claim 24, further comprising:

receiving in correspondence with the captured and pitch-corrected vocal performance a geocode for inclusion as metadata in association with subsequent supply and rendering of the resulting mixed performance.

28. The method of claim 24, further comprising:

receiving at the content server one or more of (i) listener comment on and (ii) ranking of the resulting mixed performance for inclusion as metadata in association with subsequent supply and rendering thereof.