Continuous pitch-corrected vocal capture device cooperative with content server for backing track mix

• US 9,058,797 B2
• Filed: 09/04/2010
• Issued: 06/16/2015
• Est. Priority Date: 12/15/2009
• Status: Active Grant

• Current Assignee: Smule, Inc.
• Original Assignee: Smule, Inc.
• Inventors: Salazar, Spencer, Fiebrink, Rebecca A., Wang, Ge, Ljungstrm, Mattias, Smith, Jeffrey C., Cook, Perry R.
• Primary Examiner(s): He, Jialong
• Application Number:
  US12/876,131
• Publication Number:
  US 20110144981A1
• Time in Patent Office: 1,746 Days
• Field of Search: 704/205, 704/207, 704/270, 704/278, 846/09, 846/10
• US Class Current: 1/1
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G10H 1/366 with means for modifying or correcting the external signal, e.g. pitch correction, reverberation, changing a singer's voice
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/251 Chorus, i.e. automatic generation of two or more extra voices added to the melody, e.g. by a chorus effect processor or multiple voice harmonizer, to produce a chorus or unison effect, wherein individual sounds from multiple sources with roughly the same timbre converge and are perceived as one
• CPC Class Codes: G10H 2210/331 Note pitch correction, i.e. modifying a note pitch or replacing it by the closest one in a given scale
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications
• CPC Class Codes: G10H 2240/125 Library distribution, i.e. distributing musical pieces from a central or master library
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/211 Wireless transmission, e.g. of music parameters or control data by radio, infrared or ultrasound
• CPC Class Codes: G10H 2240/251 Mobile telephone transmission, i.e. transmitting, accessing or controlling music data wirelessly via a wireless or mobile telephone receiver, analogue or digital, e.g. DECT, GSM, UMTS
• CPC Class Codes: G10L 21/013 Adapting to target pitch
• CPC Class Codes: G10L 21/0356 for synchronising with other signals, e.g. video signals

Abstract

Techniques have been developed to facilitate (1) the capture and pitch correction of vocal performances on handheld or other portable computing devices and (2) the mixing of such pitch-corrected vocal performances with backing tracks for audible rendering on targets that include such portable computing devices and as well as desktops, workstations, gaming stations, even telephony targets. Implementations of the described techniques employ signal processing techniques and allocations of system functionality that are suitable given the generally limited capabilities of such handheld or portable computing devices and that facilitate efficient encoding and communication of the pitch-corrected vocal performances (or precursors or derivatives thereof) via wireless and/or wired bandwidth-limited networks for rendering on portable computing devices or other targets.

Claims

1. A method comprising:

using a portable computing device for vocal performance capture, the portable computing device having a display, a microphone interface and a data communications interface;

responsive to user selection of a backing track, retrieving via the data communications interface at least lyrics and timing information corresponding to the backing track;

at the portable computing device, audibly rendering a first encoding of the backing track and concurrently presenting the retrieved lyrics on the display in accord with the timing information;

at the portable computing device, capturing and pitch correcting a vocal performance of the user in accord with pitch correction settings;

transmitting from the portable computing device to a remote server via the communications interface, an audio encoding of the pitch-corrected vocal performance for mixing with a second encoding of the backing track; and

receiving from the remote server, and audibly rendering at the portable computing device, a mixed performance, wherein the mixed performance is an encoding of the user'"'"'s own pitch-corrected vocal performance mixed with the second encoding of the backing track.

2. The method of claim 1,

wherein the second encoding is of higher quality or fidelity than the first encoding.

3. The method of claim 1, further comprising:

responsive to the user selection, also retrieving the first encoding of the backing track via the data communications interface.

4. The method of claim 1,

wherein the first encoding of the backing track resides in storage local to the portable computing device, and

wherein both the retrieving of lyrics and timing information and the transmitting of the audio encoding to the remote server identify the backing track using an identifier ascertainable from the first encoding of the backing track.

5. The method of claim 1, further comprising:

the mixing with the second encoding of the backing track, wherein the mixing is performed at the remote server.

6. The method of claim 1, further comprising:

the mixing with the second encoding of the backing track, wherein the mixing is performed at a remote device at which the resulting mixed performance is audibly rendered.

7. The method of claim 1, further comprising:

mixing the pitch-corrected vocal performance with the audible rendering of the backing track at the portable computing device, wherein the resulting mixed performance includes the user'"'"'s own vocal performance captured in correspondence with the lyrics and backing track, but pitch-corrected in accord with the pitch correction settings.

8. The method of claim 1, wherein the pitch correction settings code one or more of:

settings characteristic of a particular artist or a particular vocal performance corresponding to the backing track;

a musical key or scale; and

temporal variations that are performance synchronized with timings of particular lyrics to provide note targets.

9. The method of claim 1, further comprising:

responsive to the user selection, also retrieving the pitch correction settings via the communications interface.

10. The method of claim 1,

wherein the backing track encodes either or both of instrumentals and backing vocals.

11. The method of claim 1, further comprising:

audibly rendering the second mixed performance on a remote device.

12. The method of claim 11, wherein the remote device and the portable computing device are both selected from a group of:

a mobile phone;

a personal digital assistant; and

a laptop computer, notebook computer pad-type device or netbook.

13. The method of claim 1, further comprising:

audibly rendering a second mixed performance at the portable computing device, wherein the second mixed performance is an encoding of another pitch-corrected vocal performance captured and pitch-corrected at a remote device and mixed with the backing track.

14. The method of claim 1, further comprising:

geocoding the transmitted audio encoding of the pitch-corrected vocal performance; and

displaying a geographic origin for, and in correspondence with audible rendering of, a second mixed performance of a pitch-corrected vocal performance captured and pitch-corrected at a remote device and mixed with the backing track, the second mixed performance received via the data communications interface directly or indirectly from the remote device.

15. The method of claim 14,

wherein the display of geographic origin is by display animation suggestive of a performance emanating from a particular location on a globe.

16. The method of claim 14, further comprising:

capturing and conveying back to the remote server one or more of (i) listener comment on and (ii) ranking of the second mixed performance for inclusion as metadata in association with subsequent supply and rendering thereof.

32. The method of claim 1,

wherein the first and second encodings of the backing track are respective versions of the same underlying audio source material, and the first encoding is different from the second encoding.

17. A portable computing device comprising:

a display;

a microphone interface;

an audio transducer interface;

a data communications interface;

user interface code executable on the portable computing device to capture user interface gestures selective for a backing track and to initiate retrieval of at least lyrics and timing information corresponding thereto;

the user interface code further executable to capture user interface gestures to initiate (i) audible rendering of a first encoding of the backing track, (ii) concurrent presentation of the lyrics on the display and (iii) capture of the user'"'"'s vocal performance using the microphone interface;

pitch correction code executable on the portable computing device to, concurrent with said audible rendering, continuously pitch correct the user'"'"'s vocal performance in accord with pitch correction settings;

transmit code executable on the portable computing device to transmit to a remote server via the data communications interface, an audio encoding of the pitch-corrected vocal performance for mixing with a second encoding of the backing track; and

audible render code executable on the portable computing device to receive from the remote server, and audibly render at the portable computing device, a mixed performance, wherein the mixed performance is an encoding of the user'"'"'s own pitch-corrected vocal performance mixed with the second encoding of the backing track.

18. The portable computing device of claim 17,

wherein the second encoding is of higher quality or fidelity than the first encoding.

19. The portable computing device of claim 17, further comprising:

a rendering pipeline that mixes the user'"'"'s pitch-corrected vocal performance into the audible rendering of the backing track at the portable computing device, such that the resulting mixed performance includes the user'"'"'s own vocal performance captured in correspondence with the lyrics and backing track, but pitch-corrected in accord with the pitch correction settings.

20. The portable computing device of claim 17,

further comprising local storage,

wherein the initiated retrieval includes checking instances, if any, of the lyrics and timing information in the local storage against instances available from a remote server and retrieving from the remote server if instances in local storage are unavailable or out-of-date.

21. The portable computing device of claim 17,

the user interface code further executable to initiate retrieval of the first encoding of the backing track.

22. The portable computing device of claim 17, the user interface code further executable to initiate retrieval of the pitch correction settings, wherein the pitch correction settings code one or more of:

settings characteristic of a particular artist or a particular vocal performance corresponding to the backing track;

a musical key or scale; and

temporal variations that are performance synchronized with timings of particular lyrics to provide note targets.

23. The portable computing device of claim 17,

the user interface code further executable to allow the user to dynamically alter the pitch correction settings during the course of the vocal performance.

24. A computer program product encoded in one or more non-transitory media, the computer program product including instructions executable on a processor of the portable computing device to cause the portable computing device to:

retrieve via the data communications interface both lyrics and timing information corresponding to a backing track;

audibly render a first encoding of the backing track and concurrently present the lyrics on the display in accord with the timing information;

capture and continuously pitch correct a vocal performance of a user in accord with pitch correction settings;

transmit to a remote server via the data communications interface, an audio encoding of the pitch-corrected vocal performance for mixing with a second encoding of the backing track; and

receive from the remote server, and audibly render at the portable computing device, a mixed performance, wherein the mixed performance is an encoding of the user'"'"'s own pitch-corrected vocal performance mixed with the second encoding of the backing track.

25. The computer program product of claim 24,

wherein the second encoding is of higher quality or fidelity than the first encoding.

26. The computer program product of claim 24, the instructions encoded therein being executable on the processor of the portable computing device to further cause the portable computing device to mix the pitch-corrected vocal performance into the audible rendering of the backing track at the portable computing device, wherein the resulting mixed performance includes the user'"'"'s own vocal performance captured in correspondence with the lyrics and backing track, but pitch-corrected in accord with the pitch correction settings.

27. A method comprising:

from a content server, responsive to a user selection, supplying a first portable computing device with (i) a first encoding of a backing track and (ii) associated lyrics and timing information;

at the content server, receiving an encoding of a captured and pitch-corrected vocal performance of the user, wherein the pitch-corrected vocal performance corresponds to the backing track, lyrics and timing information;

at the content server, mixing the received pitch-corrected vocal performance with a second encoding of the backing track and thereafter supplying the resulting mixed performance for audible rendering at a first remote device;

mixing the received pitch-corrected vocal performance with a third encoding of the backing track as an alternative mixed performance, the third encoding being of still higher quality or fidelity than the second encoding; and

supplying the alternative mixed performance for audible rendering at a second remote device.

28. The method of claim 27, further comprising:

supplying the mixed performance for audible rendering at the first portable computing device.

29. The method of claim 27,

wherein the second encoding is of higher quality or fidelity than the first encoding.

30. The method of claim 29, further comprising:

selecting one of the resulting mixed performance and the alternative mixed performance for supply to a third remote device based on characteristics of the third remote device or of a communications channel thereto.

31. The method of claim 27, further comprising:

receiving at the content server one or more of (i) a geocode for the pitch-corrected vocal performance and (ii) listener comment on or ranking of the resulting mixed performance for inclusion as metadata in association with subsequent supply and rendering thereof.

33. The method of claim 27,

wherein the first and second encodings of the backing track are respective versions of the same underlying audio source material, and the first encoding is different from the second encoding.