Coordinating and mixing vocals captured from geographically distributed performers

• US 10,395,666 B2
• Filed: 07/31/2017
• Issued: 08/27/2019
• Est. Priority Date: 04/12/2010
• Status: Active Grant

• Current Assignee: Smule, Inc.
• Original Assignee: Smule, Inc.
• Inventors: Cook, Perry R., Lazier, Ari, Lieber, Tom, Kirk, Turner Evan
• Primary Examiner(s): He, Jialong
• Application Number:
  US15/664,659
• Publication Number:
  US 20180174596A1
• Time in Patent Office: 757 Days
• CPC Class Codes: G10H 1/366 with means for modifying or correcting the external signal, e.g. pitch correction, reverberation, changing a singer's voice
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/331 Note pitch correction, i.e. modifying a note pitch or replacing it by the closest one in a given scale
• CPC Class Codes: G10H 2240/251 Mobile telephone transmission, i.e. transmitting, accessing or controlling music data wirelessly via a wireless or mobile telephone receiver, analogue or digital, e.g. DECT, GSM, UMTS
• CPC Class Codes: G10L 13/0335 Pitch control
• CPC Class Codes: G10L 21/013 Adapting to target pitch
• CPC Class Codes: Y10S 84/04 Chorus; ensemble; celeste

Abstract

Despite many practical limitations imposed by mobile device platforms and application execution environments, vocal musical performances may be captured and continuously pitch-corrected for mixing and rendering with backing tracks in ways that create compelling user experiences. Based on the techniques described herein, even mere amateurs are encouraged to share with friends and family or to collaborate and contribute vocal performances as part of virtual “glee clubs.” In some implementations, these interactions are facilitated through social network- and/or eMail-mediated sharing of performances and invitations to join in a group performance. Using uploaded vocals captured at clients such as a mobile device, a content server (or service) can mediate such virtual glee clubs by manipulating and mixing the uploaded vocal performances of multiple contributing vocalists.

Claims

1. A system comprising:

plural geographically-distributed portable computing devices, each having a respective display, microphone interface, and communications interface;

a first one of the plural geographically-distributed portable computing devices configured to receive via its respective communications interface, a first vocal score temporally synchronizable with a backing track and with lyrics, and further configured to, responsive to a selection by a first user thereof, audibly render the backing track and concurrently present corresponding portions of the lyrics on its respective display in temporal correspondence the audible rendering;

audio processing code executable on the first portable computing device to capture and perform a first pitch correction to a vocal performance of the first user in accord with the first vocal score to constitute a first-part vocal performance;

a second one of the plural geographically-distributed portable computing devices configured to receive via its respective communications interface (i) a second backing track including the first-part vocal performance of the first user vocalist captured at the first portable computing device and (ii) a second vocal score temporally synchronizable with the second backing track and with the lyrics, and further configured to, responsive to a selection by a second user thereof, audibly render the second backing track and concurrently present corresponding portions of the lyrics on its respective display in temporal correspondence the audible rendering; and

audio processing code executable on the second portable computing device to capture and perform a second pitch correction to a vocal performance of the second user in accord with the second vocal score to constitute a second-part vocal performance,

wherein the second pitch correction includes pitch shifting the vocal performance of the second user to one of a vocal melody position and a harmony position determined based on a prominence of the second-part vocal performance in a coordinated vocal duet based on the first-part and second-part vocal performances.

2. The system of claim 1, further comprising:

an audio processing pipeline configured to accrete and render a coordinated vocal duet from the first-part and second-part vocal performances captured at respective first and second geographically-distributed portable computing devices.

3. The system of claim 1, further comprising:

a content server communicatively coupled to convey;

backing tracks, including the first and second backing tracks;

vocal scores, including the first and second vocal scores; and

captured vocal performances, including the first-part and second-part vocal performances,

to or from the geographically-distributed portable computing devices via respective ones of the communications interfaces.

4. The system of claim 3,

wherein the respective first-part and second-part vocal performances conveyed to or from the content server include pitch-corrected/shifted encodings thereof, dry vocal encodings thereof, or both pitch-corrected/shifted and dry vocal encodings thereof.

5. The system of claim 1,

wherein the audio processing code executable at respective first and second portable computing devices locally performs, in real-time, the pitch correction in accord with the respective first or second vocal score and further mixes the respective pitch-corrected first- or second-part vocal performance into the audible rendering at the respective first or second portable computing device.

6. The system of claim 1,

wherein vocal performances, including the first-part and second-part vocal performances, captured at respective of the plural geographically-distributed portable computing devices are conveyed via the respective communications interfaces to a content server in a form that includes dry vocal encodings, and

wherein the content server hosts an audio processing pipeline configured to pitch correct or shift respective ones of the conveyed vocal performance encodings and accrete the pitch-corrected or shifted vocal performances into a coordinated group performance including at least pitch-corrected or shifted versions of the first-part and second-part vocal performances.

7. The system of claim 6,

wherein the plural geographically-distributed portable computing devices include three or more mobile digital devices;

and

wherein the coordinated group performance accretes vocal performances captured at the three or more mobile digital devices.

8. The system of claim 1,

wherein the first and second vocal scores respectively code Part A and Part B duet portions of an overall vocal score.

9. The system of claim 1,

wherein the first and second vocal scores respectively code one or the other of melody and harmony portions of an overall vocal score.

10. The system of claim 1,

wherein the first and second vocal scores are represented as separable pitch tracks in a combined vocal score suppled to both the first and second geographically-distributed portable computing devices.

11. The system of claim 1,

wherein vocal performances, including the first-part and second-part vocal performances, captured at respective of the plural geographically-distributed portable computing devices are conveyed via the respective communications interfaces to a content server that hosts an audio processing pipeline configured to accrete the vocal performances into a coordinated group performance that features more prominently, at least at times, vocals of one or the other of the first-part and second-part vocal performances.

12. The system of claim 1,

wherein the plural geographically-distributed portable computing devices each include a camera suitable for capture of still images and/or video for association with the captured and pitch-corrected vocal performances.

13. The system of claim 12, further comprising:

a content server configured to accrete the captured and pitch-corrected vocal performances into a coordinated group performance that features more prominently, at least at times, one or the other of the first-part and second-part vocal performances.

14. A method of coordinating a multi-part, vocal performance that includes contributions captured at respective geographically-distributed portable computing devices, the method comprising:

using a first one of the geographically-distributed portable computing devices for vocal performance capture, the first portable computing device having a display, a microphone interface and a communications interface;

responsive to a user selection at the first portable computing device, retrieving via the communications interface

(i) a first-part vocal performance of at least one other vocalist captured at a remote second one of the geographically-distributed portable computing devices; and

(ii) a vocal score temporally synchronizable with the first-part vocal performance and with lyrics;

at the first portable computing device, audibly rendering a backing track including the first-part vocal performance and concurrently presenting corresponding portions of the lyrics on the display in temporal correspondence with the audible rendering;

at the first portable computing device, capturing and pitch correcting a second-part vocal performance of the user in accord with the vocal score,

wherein the pitch correction includes pitch shifting the second-part vocal performance to one of a vocal melody position and a harmony position determined based on a prominence of the second-part vocal performance in a coordinated vocal duet based on the first-part and second-part vocal performances; and

accreting and rendering the coordinated vocal duet from the first-part and second-part vocal performances captured at the respective first and second geographically-distributed portable computing devices.

15. The method of claim 14, further comprising:

transferring between the first geographically-distributed portable computing device and a content server the vocal score and the captured vocal performances, including the first-part and second-part vocal performances.

16. The method of claim 15,

wherein the respective first-part and second-part vocal performances are transferred to or from the content server and include pitch-corrected/shifted encodings thereof, dry vocal encodings thereof, or both pitch-corrected/shifted and dry vocal encodings thereof.

17. The method of claim 14,

wherein audio processing code executable at the first portable computing device locally performs, in real-time, the pitch correction in accord with the vocal score and further mixes the pitch-corrected second-part vocal performance into the audible rendering at the respective first portable computing device.

18. The method of claim 14, further comprising:

hosting, at a content server hosts an audio processing pipeline configured to pitch correct or shift respective vocal performance encodings captured at the first and second portable computing devices and to accrete the pitch-corrected or shifted vocal performances into a coordinated group performance including at least pitch-corrected or shifted versions of the first-part and second-part vocal performances.

19. The method of claim 14,

wherein the vocal score codes at least a Part A or Part B duet portion of an overall vocal score.

20. The method of claim 14,

wherein the vocal score codes at least one or the other of melody and harmony portions of an overall vocal score.

21. The method of claim 14, further comprising:

capturing still images and/or video at the first geographically-distributed portable computing device for association with at least the captured and pitch-corrected second-part vocal performances.

22. The method of claim 21, further comprising:

accreting into a coordinated group performance, at a content server, pitch-corrected vocal performances captured at the first and second geographically-distributed portable computing devices, the coordinated group performance featuring more prominently, at least at times, one or the other of the first-part and second-part vocal performances.

23. A first portable computing device comprising:

a display;

a microphone interface;

a communications interface;

a user interface of the portable computing device responsive to a user selection from a user and operable to retrieve, via the communications interface, (i) a first-part vocal performance of at least one other vocalist captured at a remote second portable computing device and (ii) a vocal score temporally synchronizable with the first-part vocal performance and with lyrics;

the user interface further operable to cause the first portable computing device to, responsive to the user selection, audibly render a backing track including the first-part vocal performance and concurrently present corresponding portions of the lyrics on the display in temporal correspondence with the audible rendering;

audio processing code executable on the first portable computing device configured to capture and pitch correct a second-part vocal performance of the user in accord with the vocal score,

wherein the pitch correction includes pitch shifting the second-part vocal performance to one of a vocal melody position and a harmony position determined based on a prominence of the second-part vocal performance in a coordinated vocal duet based on the first-part and second-part vocal performances; and

the audio processing code further configured to render the coordinated vocal duet from the first-part and second-part vocal performances captured at the respective first and second portable computing devices.

24. The first portable computing device, as recited in claim 23, communicatively coupled to a content server and to the second portable computing device via the communications interface.