Temporal drift correction

• US 7,600,032 B2
• Filed: 06/08/2007
• Issued: 10/06/2009
• Est. Priority Date: 10/13/2000
• Status: Expired due to Term

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: AOL LLC (Apollo Global Management, Inc.)
• Inventors: Mantegna, John, Wu, Shuwu
• Primary Examiner(s): LAZARO, DAVID R
• Application Number:
  US11/760,204
• Publication Number:
  US 20070230514A1
• Time in Patent Office: 851 Days
• Field of Search: 709/231, 709/248, 370/354, 370/516
• US Class Current: 709/231
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04M 2203/4536 Voicemail combined with text-based messaging
• CPC Class Codes: H04M 3/2236 Quality of speech transmission monitoring
• CPC Class Codes: H04M 3/305 testing of physical copper line parameters, e.g. capacitance or resistance locating faults in cables G01R31/08
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04M 7/0039 where the data service is provided by a stream of packets which are rendered in real time by the receiving terminal network streaming of media packets H04L65/60
• CPC Class Codes: H04M 7/006 Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP) , including next generation networks with a packet-switched transport layer H04L65/00 takes precedence; aspects not specific to the type of network H04M3/00; special services in those networks H04M3/42

Abstract

Temporal drift correction may be provided in a real-time audio communication system by measuring a size of a receiving data buffer and comparing that size to a predetermined nominal data buffer size. An amount of temporal drift is characterized as a number of samples per audio playback data block based on the measured data buffer size and the nominal data buffer size. A number of samples to be inserted or removed for each audio playback data block to correct the temporal drift may be determined, and the number of samples for each audio playback data block may be modified. For example, an instantaneous size of the receiving data buffer may be measured, and if measured multiple times, may be averaged over a time period. Heuristic resampling of the audio playback data block also may be performed. When heuristic resampling is performed, multiple consecutive samples of audio data in an audio buffer may be analyzed, consecutive samples with minimal variation in a parameter of their data may be identified, and the number of samples in the identified consecutive samples may be adjusted. A sample may be removed from or added to the identified consecutive samples.

Claims

1. A method for temporal drift correction in a real-time electronic communication comprising:

receiving an electronic communication including multiple samples;

computing, using at least one processor, an amount of temporal drift associated with the received electronic communication, the computing including performing a calculation using an average data buffer size related to the received electronic communication and a nominal data buffer size and multiplying a result, that is based on the calculation, by a convergent factor;

determining a number of samples to be inserted in or removed from the received electronic communication based on the computed amount of temporal drift associated with the received electronic communication;

analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples; and

modifying the received electronic communication by inserting or removing the determined number of samples at the selected one or more portions of the received electronic communication.

2. The method of claim 1 wherein analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples includes selecting one or more portions of the received electronic communication in which insertion or removal of the determined number of samples would result in minimal audible artifacts.

3. The method of claim 2 wherein selecting the one or more portions of the received electronic communication in which insertion or removal of the determined number of samples would result in minimal audible artifacts includes selecting the one or more portions of the received electronic communication in which insertion or removal of the determined number of samples would be inaudible.

4. The method of claim 1 wherein analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples includes selecting one or more portions of the received electronic communication based on waveform properties of the received electronic communication.

5. The method of claim 1 wherein analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples includes selecting one or more portions of the received electronic communication that have minimal variance.

6. The method of claim 1 wherein analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples includes selecting one or more portions of the received electronic communication based on a heuristic resampling technique.

7. The method of claim 1 wherein the received electronic communication includes multiple data blocks, and analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples includes selecting one or more data blocks of the received electronic communication at which to insert or remove samples.

8. The method of claim 7 wherein selecting one or more data blocks of the received electronic communication at which to insert or remove samples includes selecting multiple data blocks at which to insert or remove samples.

9. The method of claim 8 wherein selecting multiple data blocks at which to insert or remove samples includes selecting less than all data blocks at which to insert or remove samples.

10. The method of claim 7 wherein:

computing an amount of temporal drift associated with the received electronic communication includes computing an amount of temporal drift for each data block included in the electronic communication;

determining a number of samples to be inserted in or removed from the received electronic communication based on the computed amount of temporal drift associated with the received electronic communication includes determining a number of samples to be inserted in or removed from each data block included in the electronic communication based on the computed amount of temporal drift associated with each data block included in the electronic communication; and

modifying the received electronic communication by inserting or removing the determined number of samples at the selected one or more portions of the received electronic communication includes modifying each of the selected data blocks by inserting or removing the determined number of samples for the selected data blocks.

11. The method of claim 1 wherein multiplying a result, that is based on the calculation, by a convergent factor comprises multiplying a result, that is based on the calculation, by a convergent factor that is greater than one.

12. The method of claim 1 wherein computing, using at least one processor, an amount of temporal drift associated with the received electronic communication, the computing including performing a calculation using an average data buffer size related to the received electronic communication and a nominal data buffer size and multiplying a result, that is based on the calculation, by a convergent factor comprises computing an amount of temporal drift for a playback block of the received electronic communication, the computing including:

calculating a difference between an average data buffer size in samples for the playback block of the received electronic communication and a nominal number of samples in the data buffer;

dividing the calculated difference by a number of nominal playback blocks in the data buffer to determine a particular result; and

multiplying the particular result by the convergent factor.

13. The method of claim 1 wherein:

analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples comprises selecting a group of at least three samples included in the received electronic communication that have a minimal variance; and

modifying the received electronic communication by inserting or removing the determined number of samples at the selected one or more portions of the received electronic communication comprises;

maintaining end points of the selected group of at least three samples included in the received electronic communication; and

repeating or deleting at least one middle sample that is included in the selected group of at least three samples included in the received electronic communication and that is positioned between the end points in the selected group of at least three samples included in the received electronic communication.

14. At least one computer readable medium storing one or more computer programs, the one or more computer programs comprising instructions for:

receiving an electronic communication including multiple samples;

computing an amount of temporal drift associated with the received electronic communication, the computing including performing a calculation using an average data buffer size related to the received electronic communication and a nominal data buffer size and multiplying a result, that is based on the calculation, by a convergent factor;

determining a number of samples to be inserted in or removed from the received electronic communication based on the computed amount of temporal drift associated with the received electronic communication;

analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples; and

modifying the received electronic communication by inserting or removing the determined number of samples at the selected one or more portions of the received electronic communication.

15. The at least one medium of claim 14 wherein analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples includes selecting one or more portions of the received electronic communication in which insertion or removal of the determined number of samples would result in minimal audible artifacts.

16. The at least one medium of claim 15 wherein selecting the one or more portions of the received electronic communication in which insertion or removal of the determined number of samples would result in minimal audible artifacts includes selecting the one or more portions of the received electronic communication in which insertion or removal of the determined number of samples would be inaudible.

17. The at least one medium of claim 14 wherein analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples includes selecting one or more portions of the received electronic communication based on waveform properties of the received electronic communication.

18. The at least one medium of claim 14 wherein analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples includes selecting one or more portions of the received electronic communication that have minimal variance.

19. The at least one medium of claim 14 wherein analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples includes selecting one or more portions of the received electronic communication based on a heuristic resampling technique.

20. The at least one medium of claim 14 wherein the received electronic communication includes multiple data blocks, and analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples includes selecting one or more data blocks of the received electronic communication at which to insert or remove samples.

21. The at least one medium of claim 20 wherein selecting one or more data blocks of the received electronic communication at which to insert or remove samples includes selecting multiple data blocks at which to insert or remove samples.

22. The at least one medium of claim 20 wherein:

computing an amount of temporal drift associated with the received electronic communication includes computing an amount of temporal drift for each data block included in the electronic communication;

determining a number of samples to be inserted in or removed from the received electronic communication based on the computed amount of temporal drift associated with the received electronic communication includes determining a number of samples to be inserted in or removed from each data block included in the electronic communication based on the computed amount of temporal drift associated with each data block included in the electronic communication; and

modifying the received electronic communication by inserting or removing the determined number of samples at the selected one or more portions of the received electronic communication includes modifying each of the selected data blocks by inserting or removing the determined number of samples for the selected data blocks.

23. A system comprising:

at least one computer; and

at least one computer-readable medium coupled to the at least one computer having instructions stored thereon which, when executed by the at least one computer, causes the at least one computer to perform operations comprising;

receiving an electronic communication including multiple samples;

computing an amount of temporal drift associated with the received electronic communication, the computing including performing a calculation using an average data buffer size related to the received electronic communication and a nominal data buffer size and multiplying a result, that is based on the calculation, by a convergent factor;

determining a number of samples to be inserted in or removed from the received electronic communication based on the computed amount of temporal drift associated with the received electronic communication;

analyzing the multiple samples included in the received electronic communication to select one or more portions of the received electronic communication at which to insert or remove samples; and

modifying the received electronic communication by inserting or removing the determined number of samples at the selected one or more portions of the received electronic communication.