CONTINUOUS PITCH-CORRECTED VOCAL CAPTURE DEVICE COOPERATIVE WITH CONTENT SERVER FOR BACKING TRACK MIX

• US 20110144981A1
• Filed: 09/04/2010
• Published: 06/16/2011
• Est. Priority Date: 12/15/2009
• Status: Active Grant

• Current Assignee: Smule, Inc.
• Original Assignee: Smule, Inc.
• Inventors: Smith, Jeffrey C., Cook, Perry R., Ljungström, Mattias, Fiebrink, Rebecca A., Wang, Ge, Salazar, Spencer
• Granted Patent: US 9,058,797 B2
• Time in Patent Office: Days
• US Class Current: 704/207
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G10H 1/366 with means for modifying or correcting the external signal, e.g. pitch correction, reverberation, changing a singer's voice
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/251 Chorus, i.e. automatic generation of two or more extra voices added to the melody, e.g. by a chorus effect processor or multiple voice harmonizer, to produce a chorus or unison effect, wherein individual sounds from multiple sources with roughly the same timbre converge and are perceived as one
• CPC Class Codes: G10H 2210/331 Note pitch correction, i.e. modifying a note pitch or replacing it by the closest one in a given scale
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications
• CPC Class Codes: G10H 2240/125 Library distribution, i.e. distributing musical pieces from a central or master library
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/211 Wireless transmission, e.g. of music parameters or control data by radio, infrared or ultrasound
• CPC Class Codes: G10H 2240/251 Mobile telephone transmission, i.e. transmitting, accessing or controlling music data wirelessly via a wireless or mobile telephone receiver, analogue or digital, e.g. DECT, GSM, UMTS
• CPC Class Codes: G10L 21/013 Adapting to target pitch
• CPC Class Codes: G10L 21/0356 for synchronising with other signals, e.g. video signals

Abstract

Techniques have been developed to facilitate (1) the capture and pitch correction of vocal performances on handheld or other portable computing devices and (2) the mixing of such pitch-corrected vocal performances with backing tracks for audible rendering on targets that include such portable computing devices and as well as desktops, workstations, gaming stations, even telephony targets. Implementations of the described techniques employ signal processing techniques and allocations of system functionality that are suitable given the generally limited capabilities of such handheld or portable computing devices and that facilitate efficient encoding and communication of the pitch-corrected vocal performances (or precursors or derivatives thereof) via wireless and/or wired bandwidth-limited networks for rendering on portable computing devices or other targets.

Claims

1. A method comprising:

using a portable computing device for vocal performance capture, the handheld computing device having a display, a microphone interface and a data communications interface;

responsive to user selection of a backing track, retrieving via the data communications interface at least lyrics and timing information corresponding to the backing track;

at the portable computing device, audibly rendering a first encoding of the backing track and concurrently presenting the retrieved lyrics on the display in accord with the timing information;

at the portable computing device, capturing and pitch correcting a vocal performance of the user in accord with pitch correction settings; and

transmitting from the portable computing device to a remote server via the communications interface, an audio encoding of the pitch-corrected vocal performance for mixing with a second encoding of the backing track.

2. The method of claim 1,

wherein the second encoding is of higher quality or fidelity than the first encoding.

3. The method of claim 1, further comprising:

responsive to the user selection, also retrieving the first encoding of the backing track via the data communications interface.

4. The method of claim 1,

wherein the first encoding of the backing track resides in storage local to the portable computing device, and

wherein both the retrieving of lyrics and timing information and the transmitting of the audio encoding to the remote server identify the backing track using an identifier ascertainable from the first encoding of the backing track.

5. The method of claim 1, further comprising:

the mixing with the second encoding of the backing track, wherein the mixing is performed at the remote server.

6. The method of claim 1, further comprising:

the mixing with the second encoding of the backing track, wherein the mixing is performed at a remote device at which the resulting mixed performance is audibly rendered.

7. The method of claim 1, further comprising:

mixing the pitch-corrected vocal performance into the audible rendering of the backing track at the portable computing device, wherein the resulting mixed performance includes the user'"'"'s own vocal performance captured in correspondence with the lyrics and backing track, but pitch-corrected in accord with the pitch correction settings.

8. The method of claim 1, wherein the pitch correction settings code one or more of:

settings characteristic of a particular artist or a particular vocal performance corresponding to the backing track;

a musical key or scale; and

temporal variations that are performance synchronized with timings of particular lyrics to provide note targets.

9. The method of claim 1, further comprising:

responsive to the user selection, also retrieving the pitch correction settings via the communications interface.

10. The method of claim 1,

wherein the backing track encodes either or both of instrumentals and backing vocals.

11. The method of claim 1, further comprising:

receiving from the remote server, and audibly rendering at the portable computing device, a second mixed performance, wherein the second mixed performance is an encoding of the user'"'"'s own pitch-corrected vocal performance mixed with the second encoding of the backing track.

12. The method of claim 11, further comprising:

audibly rendering the second mixed performance on a remote device.

13. The method of claim 12, wherein the remote device and the portable computing device are both selected from the group of:

a mobile phone;

a personal digital assistant; and

a laptop computer, notebook computer pad-type device or netbook.

14. The method of claim 1, further comprising:

audibly rendering a third mixed performance at the portable computing device, wherein the third mixed performance is an encoding of another pitch-corrected vocal performance captured and pitch-corrected at a second remote device and mixed with the backing track.

15. The method of claim 1, further comprising:

geocoding the transmitted audio encoding of the pitch-corrected vocal performance; and

displaying a geographic origin for, and in correspondence with audible rendering of, a third mixed performance of a pitch-corrected vocal performance captured and pitch-corrected at a third remote device and mixed with the backing track, the third mixed performance received via the data communications interface directly or indirectly from a third remote device.

16. The method of claim 15,

wherein the display of geographic origin is by display animation suggestive of a performance emanating from a particular location on a globe.

17. The method of claim 15, further comprising:

capturing and conveying back to the remote server one or more of (i) listener comment on and (ii) ranking of the third mixed performance for inclusion as metadata in association with subsequent supply and rendering thereof.

18. A portable computing device comprising:

a display;

a microphone interface;

an audio transducer interface;

a data communications interface;

user interface code executable on the portable computing device to capture user interface gestures selective for a backing track and to initiate retrieval of at least lyrics and timing information corresponding thereto;

the user interface code further executable to capture user interface gestures to initiate (i) audible rendering a first encoding of the backing track, (ii) concurrent presentation the lyrics on the display and (iii) capture of the user'"'"'s vocal performance using the microphone interface;

pitch correction code executable on the portable computing device to, concurrent with said audible rendering, continuously pitch correct the user'"'"'s vocal performance in accord with pitch correction settings; and

transmit code executable on the portable computing device to transmit to a remote server via the data communications interface, an audio encoding of the pitch-corrected vocal performance for mixing with a second encoding of the backing track.

19. The portable computing device of claim 18,

wherein the second encoding is of higher quality or fidelity than the first encoding.

20. The portable computing device of claim 18, further comprising:

a rendering pipeline that mixes the user'"'"'s pitch-corrected vocal performance into the audible rendering of the backing track at the portable computing device, such that the resulting mixed performance includes the user'"'"'s own vocal performance captured in correspondence with the lyrics and backing track, but pitch-corrected in accord with the pitch correction settings.

21. The portable computing device of claim 18,

further comprising local storage,

wherein the initiated retrieval includes checking instances, if any, of the lyrics and timing information in the local storage against instances available from a remote server and retrieving from the remote server if instances in local storage are unavailable or out-of-date.

22. The portable computing device of claim 18,

the user interface code further executable to initiate retrieval of the first encoding of the backing track.

23. The portable computing device of claim 18, the user interface code further executable to initiate retrieval of the pitch correction settings, wherein the pitch correction settings code one or more of:

settings characteristic of a particular artist or a particular vocal performance corresponding to the backing track;

a musical key or scale; and

temporal variations that are performance synchronized with timings of particular lyrics to provide note targets.

24. The portable computing device of claim 18,

the user interface code further executable to allow the user to dynamically alter the pitch correction settings during the course of the vocal performance.

25. A computer program product encoded in one or more media, the computer program product including instructions executable on a processor of the portable computing device to cause the portable computing device to:

retrieve via the data communications interface both lyrics and timing information corresponding to a backing track;

audibly render a first encoding of the backing track and concurrently present the lyrics on the display in accord with the timing information;

capture and continuously pitch correct a vocal performance of a user in accord with pitch correction settings; and

transmit to a remote server via the data communications interface, an audio encoding of the pitch-corrected vocal performance for mixing with a second encoding of the backing track.

26. The computer program product of claim 25,

wherein the second encoding is of higher quality or fidelity than the first encoding.

27. The computer program product of claim 25, the instructions encoded therein being executable on the processor of the portable computing device to further cause the portable computing device to mix the pitch-corrected vocal performance into the audible rendering of the backing track at the portable computing device, wherein the resulting mixed performance includes the user'"'"'s own vocal performance captured in correspondence with the lyrics and backing track, but pitch-corrected in accord with the pitch correction settings.

28. A method comprising:

from a content server, responsive to a user selection, supplying a first portable computing device with (i) a first encoding of a backing track and (ii) associated lyrics and timing information;

at the content server, receiving an encoding of a captured and pitch-corrected vocal performance of the user, wherein the pitch-corrected vocal performance corresponds to the backing track, lyrics and timing information; and

at the content server, mixing the received pitch-corrected vocal performance with a second encoding of the backing track and thereafter supplying the resulting mixed performance for audible rendering at a first remote device.

29. The method of claim 28, further comprising:

supplying the mixed performance for audible rendering at the portable computing device.

30. The method of claim 28,

wherein the second encoding is of higher quality or fidelity than the first encoding.

31. The method of claim 30, further comprising:

mixing the received pitch-corrected vocal performance with a third encoding of the backing track as an alternative mixed performance, the third encoding being of still higher quality or fidelity than the second; and

supplying the alternative mixed performance for audible rendering at a second remote device.

32. The method of claim 31, further comprising:

selecting one of the resulting mixed performance and the alternative mixed performance for supply to a third remote device based on characteristics of the third remote device or of a communications channel thereto.

33. The method of claim 28, further comprising:

receiving at the content server one or more of (i) a geocode for the pitch-corrected vocal performance and (ii) listener comment on or ranking of the resulting mixed performance for inclusion as metadata in association with subsequent supply and rendering thereof.