System and method for capture and rendering of performance on synthetic musical instrument

• US 8,686,276 B1
• Filed: 06/25/2012
• Issued: 04/01/2014
• Est. Priority Date: 11/04/2009
• Status: Active Grant

• Current Assignee: Smule, Inc.
• Original Assignee: Smule, Inc.
• Inventors: Salazar, Spencer, Wang, Ge, Cook, Perry
• Primary Examiner(s): Donels, Jeffrey
• Application Number:
  US13/532,321
• Time in Patent Office: 645 Days
• Field of Search: 846/02, 847/23
• US Class Current: 84/723
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/0033 Recording/reproducing or transmission of music for electrophonic musical instruments
• CPC Class Codes: G10H 2220/096 using a touch screen touch screen note input, e.g. using a displayed keyboard G10H2220/241; personal digital assistant [PDA] G10H2230/015
• CPC Class Codes: G10H 2220/361 Mouth control in general, i.e. breath, mouth, teeth, tongue or lip-controlled input devices or sensors detecting, e.g. lip position, lip vibration, air pressure, air velocity, air flow or air jet angle
• CPC Class Codes: G10H 2220/395 Acceleration sensing or accelerometer use, e.g. 3D movement computation by integration of accelerometer data, angle sensing with respect to the vertical, i.e. gravity sensing. conductor baton movement sensing G10H2220/206, angle sensing without reference to gravity G10H2220/391; player body joint angle sensing G10H2220/321
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/251 Mobile telephone transmission, i.e. transmitting, accessing or controlling music data wirelessly via a wireless or mobile telephone receiver, analog or digital, e.g. DECT GSM, UMTS smartphone, PDA or palmtop used for musical purposes G10H2230/015; mobile ringtone G10H2230/021

Abstract

Techniques have been developed for capturing and rendering musical performances on handheld or other portable devices. The developed techniques facilitate the capture, encoding and use of gesture streams for rendering of a musical performance. In some embodiments, a gesture stream encoding facilitates audible rendering of the musical performance locally on the portable device on which the musical performance is captured, typically in real time. In some embodiments, a gesture stream efficiently codes the musical performance for transmission from the portable device on which the musical performance is captured to (or toward) a remote device on which the musical performance is (or can be) rendered. Indeed, is some embodiments, a gesture stream so captured and encoded may be rendered both locally and on remote devices using substantially identical or equivalent instances of a digital synthesis of the musical instrument executing on the local and remote devices.

Claims

1. A method comprising:

using a portable computing device as a musical instrument, the portable computing device having a communications interface, plural on-board sensors and a multi-sensor user-machine interface;

capturing user gestures from data sampled from plural of the sensors, the user gestures indicative of user manipulation of controls of the musical instrument;

encoding a gesture stream for a performance of the user by parameterizing at least a subset of events captured from the plural sensors;

audibly rendering the performance on the portable computing device using the encoded gesture stream as an input to a digital synthesis of the musical instrument executing on the portable computing device; and

transmitting the encoded gesture stream via the communications interface for rendering of the performance on a remote device.

2. The method of claim 1,

wherein the musical instrument is a synthetic wind instrument.

3. The method of claim 1,

wherein the portable computing device includes a multi-touch sensitive display and relative to the multi-touch sensitive display, the capturing includes recognizing at least transient presence of one or more fingers at respective display positions corresponding to a control of the synthetic musical instrument;

and

wherein at least some of the parameterized events encode respective pitch in correspondence with the recognized presence of one or more fingers.

4. The method of claim 1,

wherein the portable computing device includes a multi-touch sensitive display and relative to the multi-touch sensitive display, the capturing includes recognizing at least transient presence of a finger along a range of positions.

5. The method of claim 1,

wherein the multi-sensor user-machine interface further includes an accelerometer,

wherein relative to the accelerometer, the capturing includes recognizing movement-type ones of the user gestures, and

wherein the movement-type user gestures captured using the accelerometer are indicative of one or more of vibrato and timbre for the rendered performance.

6. The method of claim 1, wherein the digital synthesis includes a model of acoustic response for wind instrument.

7. The method of claim 1, further comprising:

rendering the performance on the remote device using the encoded gesture stream as an input to a second digital synthesis of the musical instrument on the remote device.

8. The method of claim 7, wherein the remote device and the portable computing device are both selected from the group of:

a mobile phone;

a personal digital assistant; and

a laptop computer, notebook computer or netbook.

9. The method of claim 7,

wherein the remote device includes a server from which the rendered performance is subsequently supplied as one or more audio encodings thereof.

10. The method of claim 1, further comprising:

audibly rendering a second performance on the portable computing device using a second gesture stream encoding received via the communications interface directly or indirectly from a second remote device, the second performance rendering using the received second gesture stream encoding as an input to the digital synthesis of the musical instrument.

12. A computer program product encoded in one or more media, the computer program product including instructions executable on a processor of the portable computing device to cause the portable computing device to perform the method of claim 1.

13. The computer program product of claim 12, wherein the one or more media are readable by the portable computing device or readable incident to a computer program product conveying transmission to the portable computing device.

11. A method comprising:

using a portable computing device as a musical instrument, the portable computing device having a communications interface, plural on-board sensors and a multi-sensor user-machine interface;

capturing user gestures from data sampled from plural of the sensors, the user gestures indicative of user manipulation of controls of the musical instrument;

encoding a gesture stream for a performance of the user by parameterizing at least a subset of events captured from the plural sensors;

audibly rendering the performance on the portable computing device using the encoded gesture stream as an input to a digital synthesis of the musical instrument executing on the portable computing device;

transmitting the encoded gesture stream via the communications interface for rendering of the performance on a remote device;

geocoding the transmitted gesture stream; and

displaying a geographic origin for, and in correspondence with audible rendering of, a third performance encoded as a third gesture stream received via the communications interface directly or indirectly from a third remote device.

14. A method of using a portable computing device as a musical instrument, the portable computing device having plural on-board sensors and a multi-sensor user-machine interface, the method comprising:

capturing user gestures from data sampled from the sensors, the user gestures indicative of user manipulation of controls of the musical instrument;

encoding a gesture stream for a performance of the user by parameterizing at least a subset of events captured from the plural sensors;

audibly rendering the performance on the portable computing device using the encoded gesture stream as an input to a local digital synthesis of the musical instrument; and

transmitting the encoded gesture stream via a communications interface for rendering of the performance on a remote device using the encoded gesture stream as an input to a digital synthesis of the musical instrument hosted thereon.

15. The method of claim 14,

wherein the encoded gesture stream effectively compresses the sampled data by substantially eliminating duplicative states maintained across multiple samples of user manipulation state and instead coding performance time elapsed between events of the parameterized subset.

16. The method of claim 14,

wherein the musical instrument is a synthetic wind instrument.

17. The method of claim 16,

wherein the portable computing device includes a multi-touch sensitive display and relative to the multi-touch sensitive display, the capturing includes recognizing at least transient presence of one or more fingers at respective display positions corresponding to a control of the synthetic musical instrument;

and

wherein at least some of the parameterized events encode respective pitch in correspondence with the recognized presence of one or more fingers.

18. The method of claim 16,

wherein the portable computing device includes a multi-touch sensitive display and relative to the multi-touch sensitive display, the capturing includes recognizing at least transient presence of a finger along a range of positions.

19. The method of claim 16,

wherein the multi-sensor user-machine interface further includes an accelerometer,

wherein relative to the accelerometer, the capturing includes recognizing movement-type ones of the user gestures, and

wherein the movement-type user gestures captured using the accelerometer are indicative of one or more of vibrato and timbre for the rendered performance.

20. The method of claim 14, further comprising:

rendering the performance on the remote device using the encoded gesture stream.

21. The method of claim 20,

wherein the rendering on the remote device is an audible rendering.

22. The method of claim 20,

wherein the rendering on the remote device is to an audio encoding.