Continuous pitch-corrected vocal capture device cooperative with content server for backing track mix

• US 10,685,634 B2
• Filed: 09/01/2017
• Issued: 06/16/2020
• Est. Priority Date: 12/15/2009
• Status: Active Grant

• Current Assignee: Smule, Inc.
• Original Assignee: Smule, Inc.
• Inventors: Salazar, Spencer, Fiebrink, Rebecca A., Wang, Ge, Ljungstrom, Mattias, Smith, Jeffrey C., Cook, Perry R.
• Primary Examiner(s): He, Jialong
• Application Number:
  US15/693,946
• Publication Number:
  US 20180182366A1
• Time in Patent Office: 1,019 Days
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G10H 1/366 with means for modifying or correcting the external signal, e.g. pitch correction, reverberation, changing a singer's voice
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/251 Chorus, i.e. automatic generation of two or more extra voices added to the melody, e.g. by a chorus effect processor or multiple voice harmonizer, to produce a chorus or unison effect, wherein individual sounds from multiple sources with roughly the same timbre converge and are perceived as one
• CPC Class Codes: G10H 2210/331 Note pitch correction, i.e. modifying a note pitch or replacing it by the closest one in a given scale
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications
• CPC Class Codes: G10H 2240/125 Library distribution, i.e. distributing musical pieces from a central or master library
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/211 Wireless transmission, e.g. of music parameters or control data by radio, infrared or ultrasound beam G10H2220/405
• CPC Class Codes: G10H 2240/251 Mobile telephone transmission, i.e. transmitting, accessing or controlling music data wirelessly via a wireless or mobile telephone receiver, analog or digital, e.g. DECT GSM, UMTS smartphone, PDA or palmtop used for musical purposes G10H2230/015; mobile ringtone G10H2230/021
• CPC Class Codes: G10L 21/013 Adapting to target pitch
• CPC Class Codes: G10L 21/0356 for synchronising with other signals, e.g. video signals

Abstract

Techniques have been developed to facilitate (1) the capture and pitch correction of vocal performances on handheld or other portable computing devices and (2) the mixing of such pitch-corrected vocal performances with backing tracks for audible rendering on targets that include such portable computing devices and as well as desktops, workstations, gaming stations, even telephony targets. Implementations of the described techniques employ signal processing techniques and allocations of system functionality that are suitable given the generally limited capabilities of such handheld or portable computing devices and that facilitate efficient encoding and communication of the pitch-corrected vocal performances (or precursors or derivatives thereof) via wireless and/or wired bandwidth-limited networks for rendering on portable computing devices or other targets.

Claims

1. A method comprising:

using a portable computing device for vocal performance capture, the portable computing device having a display, a microphone interface and a data communications interface;

responsive to a first user selection of a backing track, searching a music library resident on the portable computing device for the backing track;

responsive to a first encoding of the backing track being found in the music library, retrieving via the data communications interface at least lyrics and timing information corresponding to the backing track from a content server;

at the portable computing device, audibly rendering the first encoding of the backing track and concurrently presenting the retrieved lyrics on the display in accord with the timing information, the first encoding of the backing track being of equivalent or higher quality than a second encoding of the backing track stored at the content server;

at the portable computing device, receiving user selections of a first pitch correction setting and a second pitch correction setting, wherein the first pitch correction setting codes settings characteristic of a particular artist corresponding to the backing track, and the second pitch correction setting codes temporal variations that are performance synchronized with timings of particular lyrics to provide note targets; and

at the portable computing device, capturing and pitch correcting a vocal performance of a user in accord with the first and second pitch correction settings.

2. The method of claim 1, further comprising:

responsive to the first user selection, also retrieving the first encoding of the backing track via the data communications interface.

3. The method of claim 1, further comprising:

audibly rendering at the portable computing device, a mixed performance, wherein the mixed performance is an encoding of the user'"'"'s own pitch-corrected vocal performance mixed with the backing track.

4. The method of claim 1, wherein the first pitch correction setting codes settings characteristic of a particular vocal performance corresponding to the backing track.

5. The method of claim 1, wherein the first pitch correction setting codes a musical key.

6. The method of claim 1, wherein the first pitch correction setting codes a musical scale.

7. The method of claim 1, wherein the second pitch correction setting specifies a first note for use in conjunction with first individual lyrics timings and specifies a second note different from the first note for use in conjunction with second individual lyrics timings.

8. The method of claim 1, wherein the backing track encodes either or both of instrumentals and backing vocals.

9. A portable computing device comprising:

a display;

a microphone interface;

an audio transducer interface;

a data communications interface;

user interface code executable on the portable computing device to capture user interface gestures selective for a backing track, to search a music library resident on the portable computing device for the backing track, and in response to a first encoding of the backing track being found in the music library, to initiate retrieval of at least lyrics and timing information from a content server corresponding thereto;

the user interface code further executable to capture user interface gestures to initiate (i) audible rendering of the first encoding of the backing track, (ii) concurrent presentation of the lyrics on the display and (iii) capture of the user'"'"'s vocal performance using the microphone interface, wherein the first encoding of the backing track is of equivalent or higher quality than a second encoding of the backing track stored at a content server;

pitch correction code executable on the portable computing device to, concurrent with said audible rendering, continuously pitch correct the user'"'"'s vocal performance in accord with first and second pitch correction settings, wherein the first pitch correction setting codes settings characteristic of a particular artist corresponding to the backing track, and the second pitch correction setting codes temporal variations that are performance synchronized with timings of particular lyrics to provide note targets.

10. The portable computing device of claim 9, wherein the first encoding of the backing track resides in storage local to the portable computing device.

11. The portable computing device of claim 9, wherein the first pitch correction setting codes settings characteristic of a particular vocal performance corresponding to the backing track.

12. The portable computing device of claim 9, wherein the first pitch correction setting codes a musical key.

13. The portable computing device of claim 9, wherein the first pitch correction setting codes a musical scale.

14. The portable computing device of claim 9, wherein the second pitch correction setting specifies a first note for use in conjunction with first individual lyrics timings and specifies a second note different from the first note for use in conjunction with second individual lyrics timings.

15. The portable computing device of claim 9, wherein the backing track encodes either or both of instrumentals and backing vocals.

16. A computer program product encoded in one or more non-transitory media, the computer program product including instructions executable on a processor of a portable computing device to cause the portable computing device to:

responsive to a first user selection of a backing track, search a music library resident on the portable computing device for the backing track;

responsive to a first encoding of the backing track being found in the music library, retrieve via a data communications interface both lyrics and timing information corresponding to the backing track from a content server;

audibly render the first encoding of the backing track and concurrently present the lyrics on the display in accord with the timing information, wherein the first encoding of the backing track is of equivalent or higher quality than a second encoding of the backing track stored at the content server;

receive user selections of a first pitch correction setting and a second pitch correction setting, wherein the first pitch correction setting codes settings characteristic of a particular artist corresponding to the backing track, and the second pitch correction setting codes temporal variations that are performance synchronized with timings of particular lyrics to provide note targets;

capture and continuously pitch correct a vocal performance of a user in accord with the first and second pitch correction settings.

17. The computer program product of claim 16, wherein the first pitch correction setting codes a musical key.

18. The computer program product of claim 16, wherein the first pitch correction setting codes a musical scale.

19. The computer program product of claim 16, wherein the second pitch correction setting specifies a first note for use in conjunction with first individual lyrics timings and specifies a second note different from the first note for use in conjunction with second individual lyrics timings.

20. The computer program product of claim 16, wherein the backing track encodes either or both of instrumentals and backing vocals.