Method and apparatus for remote real time collaborative acoustic performance and recording thereof

• US 20070140510A1
• Filed: 10/11/2006
• Published: 06/21/2007
• Est. Priority Date: 10/11/2005
• Status: Active Grant

• Current Assignee: eJamming, Inc.
• Original Assignee: eJamming, Inc.
• Inventors: Redmann, William
• Granted Patent: US 7,853,342 B2
• Time in Patent Office: Days
• US Class Current: 381/97
• CPC Class Codes: G10H 1/0058 Transmission between separate instruments or between individual components of a musical system G10H1/0083 takes precedence
• CPC Class Codes: G10H 2210/031 Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
• CPC Class Codes: G10H 2210/125 Medley, i.e. linking parts of different musical pieces in one single piece, e.g. sound collage, DJ mix
• CPC Class Codes: G10H 2210/141 Riff, i.e. improvisation, e.g. repeated motif or phrase, automatically added to a piece, e.g. in real time
• CPC Class Codes: G10H 2230/031 Use of cache memory for electrophonic musical instrument processes, e.g. for improving processing capabilities or solving interfacing problems
• CPC Class Codes: G10H 2240/115 Instrument identification, i.e. recognizing an electrophonic musical instrument, e.g. on a network, by means of a code, e.g. IMEI, serial number, or a profile describing its capabilities
• CPC Class Codes: G10H 2240/175 for jam sessions or musical collaboration through a network, e.g. for composition, ensemble playing or repeating; Compensation of network or internet delays therefor
• CPC Class Codes: G10H 2240/185 Error prevention, detection or correction in files or streams for electrophonic musical instruments
• CPC Class Codes: G10H 2240/301 Ethernet, e.g. according to IEEE 802.3
• CPC Class Codes: G10H 2240/305 Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes
• CPC Class Codes: G10H 2240/311 MIDI transmission G10H2240/056 takes precedence
• CPC Class Codes: G10H 2240/325 Synchronizing two or more audio tracks or files according to musical features or musical timings
• CPC Class Codes: G10H 2250/031 Spectrum envelope processing
• CPC Class Codes: G10H 2250/041 Delay lines applied to musical processing
• CPC Class Codes: G10H 2250/261 Window, i.e. apodization function or tapering function amounting to the selection and appropriate weighting of a group of samples in a digital signal within some chosen time interval, outside of which it is zero valued
• CPC Class Codes: G10H 3/181 Details of pick-up assemblies
• CPC Class Codes: G10H 7/00 Instruments in which the tones are synthesised from a data store, e.g. computer organs
• CPC Class Codes: H04R 2227/003 Digital PA systems using, e.g. LAN or internet
• CPC Class Codes: H04R 27/00 Public address systems circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; monitoring or testing arrangements for public address systems H04R29/007; amplifiers H03F
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field

Abstract

A method and apparatus are disclosed to permit real time, distributed acoustic performance by multiple musicians at remote locations. The latency of the communication channel is reflected in the audio monitor used by the performer. This allows a natural accommodation to be made by the musician. Simultaneous remote acoustic performances are played together at each location, though not necessarily simultaneously at all locations. This allows locations having low latency connections to retain some of their advantage. The amount of induced latency can be overridden by each musician. The method preferably employs a CODEC able to aesthetically synthesize packets missing from the audio stream in real time.

Claims

1. An audio processor for use by a performer, said audio processor comprising:

an audio input for accepting a local acoustic performance by said performer in real time, said audio input providing a local signal representative of said local acoustic performance;

a communication channel interface, said interface having access through a communication channel to at least one remote audio processor, said access to each of the at least one remote audio processor having an associated first latency, said interface substantially immediately sending said local signal to the at least one remote audio processor in real time, said interface further receiving from each of the at least one remote audio processor an inbound signal representative of a remote acoustic performance;

an audio output; and

,

a delay, said delay having a non-zero local delay value, said delay adding a second latency specified by the local delay value to said local signal, said delay providing said local signal to said audio output, said delay having a remote delay value associated with each of the at least one remote audio processor, said delay adding a third latency specified by said remote delay value to each corresponding at least one inbound signal, said delay providing the inbound signal to said audio output;

said audio output converting the local signal and the at least one inbound signal for said performer to hear with the effects of said delay.

2. The audio processor of claim 1, wherein said local acoustic performance provided to said audio input is captured by at least one feed selected from the group consisting of a microphone, a preamp, an electronic pickup, an effects box, a mixer.

3. The audio processor of claim 1, wherein said audio output drives at least one selected from the group consisting of a headphone, an earphone, and a speaker.

4. The audio processor of claim 1, wherein said local delay value is set manually.

5. The audio processor of claim 1, further comprising a timing control connected to said communication channel interface, said timing control directing a first timing signal to the remote audio processor, said timing control receiving a corresponding second timing signal in response from remote audio processor, whereby said timing control measures a round trip latency associated with the remote audio processor, said timing control setting said local delay value to substantially half of the round trip latency.

6. The audio processor of claim 1, wherein each remote delay value is a corresponding value of at least zero by which said local delay value exceeds the corresponding first latency.

7. The audio processor of claim 1, wherein said communications channel is a telephone network.

8. The audio processor of claim 1, wherein said local signal and said inbound signal are both digital.

9. The audio processor of claim 8, wherein said communications channel is the Internet.

10. The audio processor of claim 8, wherein said communications channel interface sends said local signal as a first plurality of packets and receives each of the at least one inbound signal as a second plurality of packets.

11. The audio processor of claim 10, wherein said communications channel interface further comprises an encoder and decoder, said encoder performing an encode process upon said local signal prior to sending, and said decoder performing a decode process upon the at least one inbound signal after receipt.

12. The audio processor of claim 11, further comprising a synthesizer, wherein said encode process comprises a transform of said local signal, said decode process comprises an inverse transform of said transform, said synthesizer interacting with said decoder to providing a patch for a late one of said second plurality of packets, said patch being computed from at least one of said second plurality of packets previously received.

13. The audio processor of claim 10 wherein said communications channel interface detects a gap in said second plurality of packets, said communication channel interface further comprising a synthesizer for constructing a patch to fill said gap.

14. The audio processor of claim 13 wherein said patch is selected from the group comprising silence, noise, a prior one of said second plurality of packets, and an inverse transform of a time-shifted result of a transform of the prior one of said second plurality of packets.

15. The audio processor of claim 1, further comprising a store, said store receiving said local signal from said audio input, said store making a first high fidelity record of said local signal, said store further having a connection to said communication channel interface, said store sending said first high fidelity record to each of said at least one remote audio processor, said store receiving a second high fidelity record from each of said at least one remote audio processor, said first high fidelity record and second high fidelity record combinable into a synchronized file.

16. The audio processor of claim 1, further comprising a timing control, wherein the inbound signal corresponding to one of said at least one remote audio processor is a late signal when the corresponding first latency exceeds said local delay value by a predetermined amount, and wherein said timing control selects an action from the group comprising warning said performer, silencing said late signal, and substituting a patch for said late signal.

17. A lobby for initiating a collaborative performance among a plurality of audio processors according to claim 9, said lobby comprising a server connected to said communication channel, each of said audio processors further having an address, said address being communicated to said server, said server providing said address to others of said plurality of audio processors, said address being used by the others to establish said collaborative performance.

18. The lobby of claim 17, wherein each of said plurality of audio processors further comprises a user interface able to accept at least one attribute of said performer selected from the group comprising interest and skill, said at least one attribute being provided through said communication channel interface to said server, said server providing said at least one attribute to others of said plurality of audio processors, the attribute being used by the others to find said performer by said attribute.

19. A method for real time, distributed, acoustic performance, comprising the steps of:

a) providing a first audio processor having access through a communication channel to at least one other audio processor at a corresponding remote location, said access having a first latency associated with each of said at least one other audio processor;

b) converting a local acoustic performance of a performer into a local signal in real time;

c) substantially immediately advancing said local signal through said communication channel to each of said at least one other audio processor;

d) receiving through said communication channel from each of said at least one other audio processor an inbound signal corresponding in real time to a remote acoustic performance at each corresponding remote location;

e) adding a non-zero second latency to said local signal;

f) adding a third latency to each of said at least one inbound signal, said third latency associated with the remote location which originated the remote performance; and

,

g) playing said local signal with said second latency and each inbound signal with each corresponding third latency into an acoustic playing said local acoustic performance as delayed.

20. The method of claim 19, further comprising the steps of:

h) measuring a round trip latency to a one of said at least one remote location; and

,

i) setting said second latency to substantially half of said round trip latency.

21. The method of claim 19, further comprising the step of:

h) setting each third latency to a corresponding value of at least zero by which said second latency exceeds the first latency corresponding to each third latency.

22. The method of claim 19, further comprising the steps of:

h) encoding said local signal before said sending step c); and

,

i) decoding each of said at least one inbound signal before said playing step g).

23. The method of claim 19 further comprising the steps of:

h) detecting a gap in one of the at least one inbound signal;

i) synthesizing a patch to fill said gap; and

,

j) substituting said patch in the place of said gap.

24. The method of claim 19, wherein said audio processor further comprises a store, said method further comprising the steps of:

h) recording said local signal in high fidelity in said store;

i) sending said local signal in high fidelity from said store to each of said at least one other audio processor;

j) receiving from each other audio processor a high fidelity record of said corresponding remote acoustic performance; and

,

k) combining said local signal in high fidelity from said store with the high fidelity record from each other audio processor to make a synchronized file.