Continuous score-coded pitch correction

• US 10,672,375 B2
• Filed: 09/01/2017
• Issued: 06/02/2020
• Est. Priority Date: 12/15/2009
• Status: Active Grant

• Current Assignee: Smule, Inc.
• Original Assignee: Smule, Inc.
• Inventors: Salazar, Spencer, Fiebrink, Rebecca A., Wang, Ge, Ljungstrom, Mattias, Smith, Jeffrey C., Cook, Perry R.
• Primary Examiner(s): He, Jialong
• Application Number:
  US15/694,461
• Publication Number:
  US 20180151164A1
• Time in Patent Office: 1,005 Days
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G10H 1/366 with means for modifying or correcting the external signal, e.g. pitch correction, reverberation, changing a singer's voice
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/251 Chorus, i.e. automatic generation of two or more extra voices added to the melody, e.g. by a chorus effect processor or multiple voice harmonizer, to produce a chorus or unison effect, wherein individual sounds from multiple sources with roughly the same timbre converge and are perceived as one
• CPC Class Codes: G10H 2210/331 Note pitch correction, i.e. modifying a note pitch or replacing it by the closest one in a given scale
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications
• CPC Class Codes: G10H 2240/125 Library distribution, i.e. distributing musical pieces from a central or master library
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/211 Wireless transmission, e.g. of music parameters or control data by radio, infrared or ultrasound beam G10H2220/405
• CPC Class Codes: G10H 2240/251 Mobile telephone transmission, i.e. transmitting, accessing or controlling music data wirelessly via a wireless or mobile telephone receiver, analog or digital, e.g. DECT GSM, UMTS smartphone, PDA or palmtop used for musical purposes G10H2230/015; mobile ringtone G10H2230/021
• CPC Class Codes: G10L 21/013 Adapting to target pitch
• CPC Class Codes: G10L 21/0356 for synchronising with other signals, e.g. video signals

Abstract

Vocal musical performances may be captured and continuously pitch-corrected at a mobile device for mixing and rendering with backing tracks in ways that create compelling user experiences. In some cases, the vocal performances of individual users are captured in the context of a karaoke-style presentation of lyrics in correspondence with audible renderings of a backing track. Such performances can be pitch-corrected in real-time at the mobile device in accord with pitch correction settings. In some cases, such pitch correction settings code a particular key or scale for the vocal performance or for portions thereof. In some cases, pitch correction settings include a score-coded melody sequence of note targets supplied with, or for association with, the lyrics and/or backing track. In some cases, pitch correction settings are dynamically variable based on gestures captured at a user interface.

Claims

1. A method comprising:

using a portable computing device for vocal performance capture, the portable computing device having a display, a microphone interface and a data communications interface;

retrieving via the data communications interface at least lyrics and timing information corresponding to a backing track;

at the portable computing device, audibly rendering a first encoding of the backing track and concurrently presenting the retrieved lyrics on the display in accord with the timing information;

at the portable computing device, capturing and pitch correcting a vocal performance of a user in accord with pitch correction settings; and

varying the pitch correction settings in correspondence with one or more user interface gestures by the user, wherein a user interface gesture selectively falls back to settings that cause sounded pitches to be corrected to nearest notes of a particular key or scale for the pitch correction of the vocal performance.

2. The method of claim 1, wherein a user interface gesture dynamically switches from melody note sets to harmony note sets as operant pitch correction settings.

3. The method of claim 1, wherein a user interface gesture dynamically switches from harmony note sets to melody note sets as operant pitch correction settings.

4. The method of claim 1, wherein the pitch correction settings include score-coded note targets, and a first user interface gesture switches off use of the score-coded note targets.

5. The method of claim 4, wherein a second user interface gesture switches on use of the score-coded note targets.

6. The method of claim 1, wherein the pitch correction settings include a vocal score that encodes one or more keys in which respective portions of the vocals are to be performed.

7. The method of claim 1, wherein the varying includes during the vocal performance capture, varying the pitch correction settings dynamically in correspondence with one or more user interface gestures by the user.

8. The method of claim 1, wherein the varying includes after the vocal performance capture, varying the pitch correction settings dynamically in correspondence with one or more user interface gestures by the user.

9. The method of claim 1, wherein the retrieved lyrics and pitch correction settings respectively code temporal correspondence of particular lyrics and of particular variations in pitch correction settings with particular points in the backing track.

10. The method of claim 1, wherein a user interface gesture pitch corrects the vocal performance capture to nearest notes of a particular key in accordance with gesture selected points in the vocal performance capture.

11. The method of claim 1, wherein a user interface gesture pitch corrects the vocal performance capture to nearest notes of a particular scale in accordance with gesture selected points in the vocal performance capture.

12. The method of claim 1, further comprising:

receiving metadata including a geocoded location of the vocal performance capture; and

displaying a visual animation that indicates the pitch-corrected vocal performance emanating from a particular location visually depicted on a globe, wherein the particular location corresponds to the geocoded location.

13. The method of claim 12, wherein the visual animation is subject to viewpoint manipulation in correspondence with user interface gestures captured at the display of the portable computing device.

14. A computing device comprising:

a display;

a microphone interface;

an audio transducer interface;

a data communications interface;

user interface code executable on the computing device to retrieve via the data communications interface at least lyrics and timing information corresponding to a backing track;

the user interface code further executable to audibly render a first encoding of the backing track and concurrently present the retrieved lyrics on the display in accord with the timing information;

the user interface code further executable to capture and pitch correct a vocal performance of a user in accord with pitch correction settings; and

the user interface code further executable to vary the pitch correction settings in correspondence with one or more user interface gestures by the user, wherein a user interface gesture selectively falls back to settings that cause sounded pitches to be corrected to nearest notes of a particular key or scale for the pitch correction of the vocal performance.

15. The computing device of claim 14, comprising:

second user interface code executable on the computing device to pitch correct the vocal performance capture to nearest notes of a particular key in accordance with gesture selected points in the vocal performance capture.

16. The computing device of claim 14, comprising:

second user interface code executable on the computing device to pitch correct the vocal performance capture to nearest notes of a particular scale in accordance with gesture selected points in the vocal performance capture.

17. The computing device of claim 14, wherein the user interface code is further executable to during the vocal performance capture, vary the pitch correction settings dynamically in correspondence with one or more user interface gestures by the user.

18. The computing device of claim 14, wherein the user interface code is further executable to after the vocal performance capture, vary the pitch correction settings dynamically in correspondence with one or more user interface gestures by the user.

19. The computing device of claim 14, further comprising:

geocode code executable on the computing device to receive metadata including a geocoded location of the vocal performance capture; and

geocode code further executable on the computing device to display a visual animation that indicates the pitch-corrected vocal performance emanating from a particular location visually depicted on a globe, wherein the particular location corresponds to the geocoded location, and wherein the visual animation is subject to viewpoint manipulation in correspondence with user interface gestures captured at the display of the portable computing device.

20. The computing device of claim 19, wherein the metadata further includes geocoded listener feedback indications, the computing device further comprising:

second geocode code further executable to present the geocoded listener feedback indications on the globe at positions consistent with the geocoded metadata, wherein the geocoded metadata includes geographic locations from which the corresponding listener feedback was transmitted.