Method and system for dynamic latency management and drift correction

• US 7,281,053 B2
• Filed: 04/30/2001
• Issued: 10/09/2007
• Est. Priority Date: 10/13/2000
• Status: Expired due to Term

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: AOL LLC (Apollo Global Management, Inc.)
• Inventors: Mantegna, John, Wu, Shuwu
• Primary Examiner(s): Najjar; Saleh
• Assistant Examiner(s): LAZARO, DAVID R
• Application Number:
  US09/845,083
• Publication Number:
  US 20020046288A1
• Time in Patent Office: 2,353 Days
• Field of Search: 709/200, 709/221, 709/225, 709/246, 709/231, 713/401, 713/503, 375/356, 375/371, 710/53, 710/56, 710/52, 710/57, 704/500, 704/502, 370/516
• US Class Current: 709/231
• CPC Class Codes: G10L 19/18 Vocoders using multiple modes
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04M 3/2236 Quality of speech transmission monitoring
• CPC Class Codes: H04M 3/305 testing of physical copper line parameters, e.g. capacitance or resistance locating faults in cables G01R31/08

Abstract

Latency in a real-time electronic communication is dynamically managed. A communication delay arising from a receiving data buffer is measured and a latency adjustment necessary to adjust the size of the communication delay to within a predetermined range and an optimal range for a size of the communication delay are determined. Using these parameters, the number of samples for an audio playback data block passing through the receiving data buffer is modified.

Claims

1. A method for dynamic latency management in a real-time electronic communication comprising:

measuring, at least twice, an instantaneous communication delay arising from a receiving data buffer;

determining a communication delay by averaging the instantaneous communication delay measurements;

selecting a time window defined based on a time necessary to capture a complete cycle of fluctuations in the receiving buffer;

determining a working range for a communication delay by measuring and analyzing delays experienced during the selected time window, the working range including an upper and lower bound;

comparing the determined communication delay with the working range for a communication delay;

if, based on the comparison, the determined communication delay is determined to be outside the upper or lower bound of the working range, determining a latency adjustment necessary to adjust the size of the determined communication delay to within the working range; and

modifying a number of samples of a playback data block passing through the receiving data buffer based on the latency adjustment determined to be necessary to adjust the size of the determined communication delay;

wherein modifying the number of samples further comprises performing heuristic resampling of a playback block.

2. The method of claim 1 wherein the number of samples is modified without introducing audible artifacts.

3. The method of claim 1 wherein the real-time electronic communication includes an audio communication.

4. The method of claim 1 wherein performing heuristic resampling comprises:

analyzing multiple consecutive samples of audio data in the playback block;

identifying consecutive samples with minimal variation in a parameter of their data; and

adjusting the number of samples in the identified consecutive samples.

5. The method of claim 4 wherein adjusting the number of samples comprises removing;

a sample from the identified consecutive samples.

6. The method of claim 4 wherein adjusting the number of samples comprises adding a sample to the identified consecutive samples.

7. The method of claim 1 wherein determining a working range for a communication delay by measuring and analyzing delays experienced during the selected time window comprises:

decreasing the upper and lower bounds of a prior working range conditioned on measuring no individual delay that exceeds a threshold amount during the selected time window; and

increasing the upper and lower bounds of the prior working range conditioned on measuring no individual delays during the selected time window.

8. The method of claim 1 wherein selecting the time window includes selecting a time window that is long enough to capture a complete cycle of high-to-low fluctuations in the receiving data buffer.

9. The method of claim 8 wherein a complete cycle of high-to-low fluctuations in the receiving data buffer comprises a cycle that takes into account fluctuations in the data buffer size, the fluctuations being caused by a relationship between the asynchronous processes of receiving data from a source and outputting data for playback.

10. A computer program, residing on a computer-readable medium, for dynamically managing latency in a real-time electronic communication, comprising instructions for causing a computer to:

measure, at least twice, an instantaneous communication delay arising from a receiving data buffer;

determine a communication delay by averaging the instantaneous communication delay measurements;

select a time window defined based on a time necessary to capture a complete cycle of fluctuations in the receiving buffer;

determine a working range for a communication delay by measuring and analyzing delays experienced during the selected time window, the working range including an upper and lower bound;

compare the determined communication delay with the working range for a communication delay;

if, based on the comparison, the determined communication delay is determined to be outside the upper or lower bound of the working range, determine a latency adjustment necessary to adjust the size of the determined communication delay to within the working range; and

modify the number of samples of a playback data block passing through the receiving data buffer based on the latency adjustment determined to be necessary to adjust the size of the determined communication delay;

wherein instructions for causing a computer to modify the number of samples further comprise instructions for causing a computer to perform heuristic resampling of a playback block.

11. The computer program of claim 10 further comprising instructions for causing a computer to modify the number of samples without introducing audible artifacts.

12. The computer program or claim 10 wherein the real-time electronic communication includes an audio communication.

13. The computer program of claim 10 wherein instructions for causing a computer to perform heuristic resampling comprise instructions for causing a computer to:

analyze multiple consecutive samples of audio data in the playback block;

identify consecutive samples with minimal variation in a parameter of their data; and

adjust the number of samples in the identified consecutive samples.

14. The computer program of claim 13 wherein adjusting the number of samples comprises removing a sample from the identified consecutive samples.

15. The computer program of claim 13 wherein adjusting the number of samples comprises adding a sample to the identified consecutive samples.

16. The computer program of claim 10 wherein the instructions for causing a computer to determine a working range for a communication delay by measuring and analyzing delays experienced during the selected time window comprise instructions for causing a computer to:

decrease the upper and lower bounds of a prior working range conditioned on measuring at least one individual delay that exceeds a threshold amount during the selected time window, and

increase the upper and lower bounds of the prior working range conditioned on measuring no individual delays during the selected time window.

17. The computer program of claim 10 wherein the instructions for causing a computer to select the time window comprise instructions for causing a computer to select a time window that is long enough to capture a complete cycle of high-to-low fluctuations in the receiving data buffer.

18. The computer program of claim 17 wherein a complete cycle of high-to-low fluctuations in the receiving data butter comprises a cycle that takes into account fluctuations in the data buffer size, the fluctuations being caused by a relationship between the asynchronous processes of receiving data from a source and outputting data for playback.

19. A computer system running programmed processes comprising a process for dynamically managing latency in a real-time electronic communication, which process causes the computer system to:

measure, at least twice, an instantaneous communication delay arising from a receiving data buffer;

determine a communication delay by averaging the instantaneous communication delay measurements;

select a time window defined based on a time necessary to capture a complete cycle of fluctuations in the receiving buffer;

determine a working range for a communication delay by measuring and analyzing delays experienced during the selected time window, the working range including an upper and lower bound;

compare the determined communication delay with the working range for a communication delay;

if, based on the comparison, the determined communication delay is determined to be outside the upper or lower bound of the working range, determine a latency adjustment necessary to adjust the size of the determined communication delay to within the working range; and

modify the number of samples of a playback data block passing through the receiving data buffer based on the latency adjustment determined to be necessary to adjust the size of the determined communication delay;

wherein the process that causes a computer system to modify the number of samples comprises a process that causes a computer system to perform heuristic resampling of a playback block.

20. The computer system of claim 19 wherein the number of samples is modified without introducing audible artifacts.

21. The computer system of claim 19 wherein the real-time electronic communication includes an audio communication.

22. The computer system of claim 19 wherein performing heuristic resampling comprises:

analyzing multiple consecutive samples of audio data in the playback block;

identifying consecutive samples with minimal variation in a parameter of their data; and

adjusting the number of samples in the identified consecutive samples.

23. The computer system of claim 22 wherein adjusting the number of samples comprises removing a sample from the identified consecutive samples.

24. The computer system of claim 22 wherein adjusting the number of samples comprises adding a sample to the identified consecutive samples.

25. The computer system of claim 19 wherein process for dynamically managing latency that causes the computer system to determine a working range for a communication delay by measuring and analyzing delays experienced during the selected time window comprises a process for dynamically managing latency that causes the computer system to:

decrease the upper and lower bounds of a prior working range conditioned on measuring at least one individual delay that exceeds a threshold amount during the selected time window, and

increase the upper and lower bounds of the prior working range conditioned on measuring no individual delays during the selected time window.

26. The computer system of claim 19 wherein the process for dynamically managing latency that causes the computer system to select the time window comprises a process for dynamically managing latency that causes the computer system to select a time window that is long enough to capture a complete cycle of high-to-low fluctuations in the receiving data buffer.

27. The computer system of claim 26 wherein a complete cycle of high-to-low fluctuations in the receiving data buffer comprises a cycle that takes into account fluctuations in the data buffer size, the fluctuations being caused by a relationship between the asynchronous processes of receiving data from a source and outputting data for playback.

28. A method for dynamic latency management in a real-time electronic communication comprising:

measuring, at least twice, an instantaneous communication delay arising from a receiving data buffer;

determining a communication delay by averaging the instantaneous communication delay measurements;

selecting a time window defined based on a tune necessary to capture a complete cycle of fluctuations in the receiving buffer;

determining a working range for a communication delay by measuring and analyzing delays experienced during the selected time window, the working range including an upper and lower bound;

comparing the determined communication delay with the working range;

making determinations related to latency adjustment based on the comparison; and

modifying data passing through the receiving buffer based on the determinations related to latency adjustment.

29. The method of claim 28 wherein selecting the time window includes selecting a time window that is long enough to capture a complete cycle of high-to-low fluctuations in the receiving data buffer.