Temporal drift correction

• US 7,231,453 B2
• Filed: 04/30/2001
• Issued: 06/12/2007
• Est. Priority Date: 10/13/2000
• Status: Expired due to Term

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: AOL LLC (Apollo Global Management, Inc.)
• Inventors: Mantegna, John, Wu, Shuwu
• Primary Examiner(s): Najjar; Saleh
• Assistant Examiner(s): LAZARO, DAVID R
• Application Number:
  US09/845,084
• Publication Number:
  US 20020097822A1
• Time in Patent Office: 2,234 Days
• Field of Search: 709/200, 709/221, 709/225, 709230-235, 709/246, 709/247, 709/248, 713/503, 710/53, 710/56, 704/500, 704/502, 370/516, 375/354, 714/700
• US Class Current: 709/231
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04M 2203/4536 Voicemail combined with text-based messaging
• CPC Class Codes: H04M 3/2236 Quality of speech transmission monitoring
• CPC Class Codes: H04M 3/305 testing of physical copper line parameters, e.g. capacitance or resistance locating faults in cables G01R31/08
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04M 7/0039 where the data service is provided by a stream of packets which are rendered in real time by the receiving terminal network streaming of media packets H04L65/60
• CPC Class Codes: H04M 7/006 Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP) , including next generation networks with a packet-switched transport layer H04L65/00 takes precedence; aspects not specific to the type of network H04M3/00; special services in those networks H04M3/42

Abstract

Temporal drift correction may be provided in a real-time audio communication system by measuring a size of a receiving data buffer and comparing that size to a predetermined nominal data buffer size. An amount of temporal drift is characterized as a number of samples per audio playback data block based on the measured data buffer size and the nominal data buffer size. A number of samples to be inserted or removed for each audio playback data block to correct the temporal drift may be determined, and the number of samples for each audio playback data block may be modified. For example, an instantaneous size of the receiving data buffer may be measured, and if measured multiple times, may be averaged over a time period. Heuristic resampling of the audio playback data block also may be performed. When heuristic resampling is performed, multiple consecutive samples of audio data in an audio buffer may be analyzed, consecutive samples with minimal variation in a parameter of their data may be identified, and the number of samples in the identified consecutive samples may be adjusted. A sample may be removed from or added to the identified consecutive samples.

Claims

1. A method for temporal drift correction in a real-time electronic communication comprising:

measuring a size of a receiving data buffer;

comparing the measured size of the receiving data buffer to a predetermined nominal data buffer size to produce a comparison result;

weighting the comparison result with a parameter that relates to and amplifies a perceived value of temporal drift;

determining, based on the comparison result weighted with the parameter, a number of samples to be inserted in or removed from a playback data block; and

modifying the playback data block by inserting of removing a number of samples that is based on the determined number of samples.

2. The method of claim 1 wherein the number of samples is modified without introducing audible artifacts.

3. The method of claim 1 wherein measuring the size of the receiving data buffer comprises measuring an instantaneous size of the receiving data buffer.

4. The method of claim 3 wherein measuring the size of the receiving data buffer comprises:

measuring an instantaneous communication delay associated with the receiving data buffer two or more times; and

averaging the measurements.

5. The method of claim 1 wherein the real-time electronic communication includes an audio communication.

6. The method of claim 5 wherein modifying the playback data block by inserting or removing the number of samples comprises performing heuristic resampling of the playback data block.

7. The method of claim 6 wherein performing heuristic resampling comprises:

analyzing multiple consecutive samples of audio data in the playback data block;

identifying consecutive samples with minimal variation in a parameter of their data; and

adjusting the number of samples in the identified consecutive samples.

8. The method of claim 7 wherein adjusting the number of samples comprises removing a sample from the identified consecutive samples.

9. The method of claim 7 wherein adjusting the number of samples comprises adding a sample to the identified consecutive samples.

10. The method of claim 1 wherein the samples are not associated with a timestamp.

11. A computer program, residing on a computer-readable medium, for correcting temporal drift in a real-time electronic communication, comprising instructions for causing a computer to:

measure a size of a receiving data buffer;

compare the measured size of the receiving data buffer to a predetermined nominal data buffer size to produce a comparison result;

weight the comparison result with a parameter that relates to and amplifies a perceived value of temporal drift;

determine, based on the comparison result weighted with the parameter, a number of samples to be inserted in or removed from a playback data block; and

modify the playback data block by inserting or removing a number of samples that is based on the determined number of samples.

12. The computer program of claim 11 wherein the number of samples is modified without introducing audible artifacts.

13. The computer program of claim 11 wherein instructions for causing a computer to measure the size of the receiving data buffer comprise instructions for causing a computer to measure an instantaneous size of the receiving data buffer.

14. The computer program of claim 13 wherein instructions for causing a computer to measure the size of the receiving data buffer comprise instructions for causing a computer to:

measure the instantaneous size of the receiving data buffer two or more times; and

average the measurements.

15. The computer program of claim 11 wherein the real-time electronic communication includes an audio communication.

16. The computer program of claim 15 wherein instructions for causing a computer to modify the playback data block by inserting or removing the number of samples comprise instructions for causing a computer to perform heuristic resampling of the playback data block.

17. The computer program of claim 16 wherein instructions for causing a computer to perform heuristic resampling comprise instructions for causing a computer to:

analyze multiple consecutive samples of audio data in the playback data block;

identify consecutive samples with minimal variation in a parameter of their data;

and adjust the number of samples in the identified consecutive samples.

18. The computer program of claim 11 wherein the samples are not associated with a timestamp.

19. A computer system running programmed processes comprising a process for correcting temporal drift in a real-time electronic communication, the process causing the computer system to:

measure a size of a receiving data buffer;

compare the measured size the receiving data buffer to predetermined nominal data buffer size to produce a comparison result;

weight the comparison result with a parameter that relates to and amplifies a perceived value of temporal drift;

determine, based on the comparison result weighted with the parameter, a number of samples to be inserted in or removed from a playback data block; and

modify the playback data block by inserting or removing a number of samples that is based on the determined number of samples.

20. The computer system of claim 19 wherein the number of samples is modified without introducing audible artifacts.

21. The computer system of claim 19 wherein measuring the size of the receiving data buffer comprises measuring an instantaneous size of the receiving data buffer.

22. The computer system of claim 21 wherein measuring the size of the receiving data buffer comprises:

measuring the instantaneous communication delay associated with the receiving data buffer two or more times; and

averaging the measurements.

23. The computer system of claim 19 wherein the real-time electronic communication includes an audio communication.

24. The computer system of claim 23 wherein modifying the playback data block by inserting or removing the number of samples comprises performing heuristic resampling of the playback data block.

25. The computer system of claim 24 wherein performing heuristic resampling comprises:

analyzing multiple consecutive samples of audio data in the playback data block;

identifying consecutive samples with minimal variation in a parameter of their data; and

adjusting the number of samples in the identified consecutive samples.

26. The computer system claim 19 wherein the samples are not associated with a timestamp.

27. A method for temporal drift correction in a real-time electronic communication comprising:

comparing a measured size of a receiving data buffer to a predetermined nominal data buffer size to produce a comparison result;

weighting the comparison result with a parameter that relates to and amplifies a perceived value of temporal drift;

dividing the weighted comparison result by a number of nominal playback blocks in the data buffer;

determining, based on the comparison result weighted with the parameter and divided by the number of nominal play back blocks, a number of samples to be inserted in or removed from a playback data block; and

modifying the playback data block by inserting or removing a number of samples that is based on the determined number of samples,

wherein;

the comparing, weighting and determining are performed in accordance with the formula TD[i]=CF*(AS[i]−

Ns)/Nb, where

TD[i] is the perceived value of temporal drift for an i-th playback block,

CF is the parameter and is greater than one,

AS[i] is the measured size of the receiving data buffer for the i-th playback block,

Ns is the predetermined nominal data buffer size, and

Nb is a number of nominal playback blocks in the data buffer.

28. The method of claim 27 wherein the number of samples is modified without introducing audible artifacts.

29. The method of claim 27 wherein measuring the size of the receiving data buffer comprises measuring an instantaneous size of the receiving data buffer.

30. The method of claim 29 wherein measuring the size of the receiving data buffer comprises:

measuring an instantaneous communication delay associated with the receiving data buffer two or more times; and

averaging the measurements.

31. The method of claim 27 wherein the real-time electronic communication includes an audio communication.

32. The method of claim 31 wherein modifying the playback data block by inserting or removing the number of samples comprises performing heuristic resampling of the playback data block.

33. The method of claim 32 wherein performing heuristic resampling comprises:

analyzing multiple consecutive samples of audio data in the playback data block;

identifying consecutive samples with minimal variation in a parameter of their data; and

adjusting the number of samples in the identified consecutive samples.

34. The method of claim 33 wherein adjusting the number of samples comprises removing a sample from the identified consecutive samples.

35. The method of claim 33 wherein adjusting the number of samples comprises adding a sample from the identified consecutive samples.