Method and apparatus for inserting variable audio delay to minimize latency in video conferencing

• US 7,170,545 B2
• Filed: 04/27/2004
• Issued: 01/30/2007
• Est. Priority Date: 04/27/2004
• Status: Expired due to Fees

• Current Assignee: Polycom Incorporated (HP Inc.)
• Original Assignee: Polycom Incorporated (HP Inc.)
• Inventors: Rodman, Jeffrey C.
• Primary Examiner(s): Ramakrishnaiah; Melur
• Application Number:
  US10/832,547
• Publication Number:
  US 20050237378A1
• Time in Patent Office: 1,008 Days
• Field of Search: 348 1401- 1409, 348/14.1, 348 1411- 1413, 348/512, 348/518, 348/515, 348/423.1, 709/231, 704/270, 704/258
• US Class Current: 348/14.12
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A method and apparatus to insert variable audio delay during video conferencing to achieve conflicting goals of lip-sync and interactive conversation. The amount of audio delay is variable according to the condition of the videoconferencing: long audio delay is inserted to achieve lip-sync between audio and video playback during monologue speech, but minimum or no audio delay is inserted during interactive discussion or argument. Variable audio playback speeds may be used instead of inserting quantum delay to achieve the same result. Various methods and apparatuses to detect the non-interactive mode or interactive modes are also disclosed.

Claims

1. A method for adjusting video and audio synchronization in a video conference with a near end and at least one far end, the method comprising the steps of:

an interactivity detector indicating a mode of conference;

wherein an interactive mode of conference is detected when an amplitude of an audio signal from the far end exceeds a first threshold, and

wherein a non-interactive mode of conference is detected when the amplitude of the audio signal remains above the first threshold for a period of time;

replaying the audio with a minimum delay if the mode of conference is interactive;

buffering the audio in a buffer and replaying the audio with a sync delay if the mode of conference is not interactive;

measuring a background noise level at the far end; and

adjusting the first threshold based on the background noise level.

2. The method in claim 1, further comprising:

replaying the audio with a varied delay if the conference is in a transition mode, wherein the conference is switching between non-interactive mode and interactive mode, and within a transition time period.

3. The method in claim 2,

wherein when the transition mode is from non-interactive mode to interactive mode, replaying the audio upon reception with the minimum delay and discarding any audio in the buffer;

and

wherein the transition time period is substantially zero seconds.

4. The method in claim 2,

wherein when the transition mode is from interactive mode to non-interactive mode, buffering without replaying audio received until the audio delay is equal to the sync delay;

and

wherein the transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the sync delay.

5. The method in claim 2,

wherein when the transition mode is from interactive mode to non-interactive mode, replaying audio at a slower speed until the audio delay is equal to the sync delay;

wherein the transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the sync delay; and

wherein the replaying speed is slowed digitally without changing the audio pitch.

6. The method in claim 5,

wherein the replaying audio speed is 80% of normal speed.

7. The method in claim 2,

wherein when the transition mode is from non-interactive mode to interactive mode, replaying audio at a faster speed until the audio delay is equal to the minimum delay;

wherein the transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the minimum delay; and

wherein the replaying speed is quickened digitally without changing the audio pitch.

8. The method in claim 7,

wherein the replaying audio speed is 120% of normal speed.

9. The method in claim 7,

wherein the conference is within interactive mode and after the transition mode, the audio replay speed is switched to the normal speed.

10. The method in claim 7,

wherein the conference is within interactive mode and after the transition mode, the audio replay speed is switched to a speed slower than the normal speed.

11. The method in claim 1,

wherein when the amplitude of the audio signal from the far end drops from above the first threshold to below a second threshold, the interactivity detector indicates interactive mode;

and when the amplitude remains below the second threshold for a second period of time, the interactivity detector indicates non-interactive mode.

12. The method in claim 11,

wherein when the interactivity detector delays the indication of interactive mode by a third period of time.

13. The method in claim 12 wherein the third period of tune is 0.1 second.

14. The method in claim 11,

wherein during a fourth period of time at the beginning of each interactive mode, the interactivity detector indicates the transition mode.

15. The method in claim 1,

wherein when the amplitude of the audio signal within a first frequency band from the far end increases and exceeds a third threshold, the interactivity detector indicates interactive mode;

and when the amplitude remains above the third threshold for a first period of time, the interactivity detector indicates non-interactive mode; and

wherein when the amplitude of the audio signal within a first frequency band from the far end drops from above the third threshold to below a fourth threshold, the interactivity detector indicates interactive mode; and

when the amplitude remains below the fourth threshold for a second period of time, the interactivity detector indicates non-interactive mode.

16. The method in claim 1,

wherein when the amplitude of the audio signal from the near end increases and exceeds a first threshold, the interactivity detector indicates interactive mode;

and when the amplitude remains above the first threshold for a first period of time, the interactivity detector indicates non-interactive mode; and

wherein when the amplitude of the audio signal from the near end drops from above the first threshold to below a second threshold, the interactivity detector indicates interactive mode; and

when the amplitude remains below the second threshold for a second period of time, the interactivity detector indicates non-interactive mode.

17. The method in claim 1,

wherein when the interactivity detector receives a control signal indicating change of interactivity from the far end, the interactivity detector indicates interactive mode;

and after the control signal remains the same for a first period of time, the interactivity detector indicates non-interactive mode.

18. The method in claim 1,

wherein when the interactivity detector receives control signals from microphones at near end and far end and the control signals change within a fifth period of time, interactivity detector indicates interactive mode;

and when the control signals remain the same for a first period of time, the interactivity detector indicates non-interactive mode.

19. The method in claim 18,

wherein when the interactivity detector delays the indication of interactive mode by a third period of time.

20. A method for adjusting video and audio synchronization in a video conference with a near end and at least one far end, the method comprising the acts of:

an interactivity detector indicating a mode of conference;

wherein when an amplitude of an audio signal from the far end increases and exceeds a first threshold, the interactivity detector indicates interactive mode; and

when the amplitude remains above the first threshold for a first period of time, the interactivity detector indicates non-interactive mode; and

wherein when the amplitude of the audio signal from the far end drops from above the first threshold to below a second threshold, the interactivity detector indicates interactive mode; and

when the amplitude remains below the second threshold for a second period of time, the interactivity detector indicates non-interactive mode.

replaying the audio with a minimum delay if the mode of the conference is interactive;

buffering the audio in a buffer and replaying the audio with a sync delay if the mode of conference is not interactive;

measuring a background noise level at the far end; and

adjusting the first threshold and the second threshold based on the background noise level.

21. A device at a near end for video conferencing with a video conference device at a far end, the device at the near end comprising:

a network interface for coupling with the conference device at the far end;

a controller unit coupled to the network, the audio speaker and the video display for controlling audio and video replay through the speaker and the video display;

a buffer for storing audio, video and control data;

wherein the controller unit is operative to replay the video with a sync delay;

wherein the controller unit is operative to replay the audio with a minimum delay if the mode of the conference is interactive; and

wherein the controller unit is operative to receive the audio from far end, store the audio in the buffer and replay the audio with the sync delay if the mode of conference is non-interactive.

22. The device in claim 21,

wherein the controller is operative to replay the audio with a varied delay if the conference is in a transition mode, wherein the conference is switching between non-interactive mode and interactive mode, and within a transition time period.

23. The device in claim 22,

wherein when the transition mode is from non-interactive mode to interactive mode, the controller is operative to replay the audio upon reception with the minimum delay and discarding any audio in the buffer that has not been replayed;

and

wherein the transition time period is substantially zero second.

24. The device in claim 22,

wherein when the transition mode is from interactive mode to non-interactive mode, the controller is operative not to replay audio received until the audio delay is equal to the sync delay;

and

wherein the transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the sync delay.

25. The device in claim 22,

wherein when the transition mode is from interactive mode to non-interactive mode, the controller is operative to replay audio at a slower speed until the audio delay is equal to the sync delay;

wherein the transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the sync delay; and

wherein the replaying speed is slowed digitally without changing the audio pitch.

26. The device in claim 25,

wherein the replaying audio speed is 90% of normal speed.

27. The device in claim 22,

wherein when the transition mode is from non-interactive mode to interactive mode, the controller is operative to replay audio at a faster speed until the audio delay is equal to the minimum delay;

wherein the transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the minimum delay; and

wherein the replaying speed is quickened digitally without changing the audio pitch.

28. The device in claim 27,

wherein the replaying audio speed is 120% of normal speed.

29. The device in claim 27,

wherein the conference is within interactive mode and after the transition mode, the audio replay speed is switched to the normal speed.

30. The device in claim 27,

wherein the conference is within interactive mode and after the transition mode, the audio replay speed is switched to a speed slower than the normal speed.

31. The device in claim 21,

wherein the controller comprises an interactivity detector;

wherein the interactivity detector is operative to indicate interactive mode when the amplitude of the audio signal from the far end increases and exceeds a first threshold, and to indicate non-interactive mode when the amplitude remains above the first threshold for a first period of time; and

wherein the interactivity detector is operative to indicate interactive mode when the amplitude of the audio signal from the far end drops from above the first threshold to below a second threshold, and to indicate non-interactive mode when the amplitude remains below the second threshold for a second period of time.

32. The device in claim 31,

wherein when the interactivity detector is operative to delay the indication of interactive mode by a third period of time.

33. The device in claim 32 wherein the third period of time is 0.1 second.

34. The device in claim 31,

wherein for a fourth period of time at the beginning of each interactive mode, the interactivity detector is operative to indicate the transition mode.

35. The device in claim 21,

wherein the controller comprising an interactivity detector;

wherein the interactivity detector is operative to indicate interactive mode when the amplitude of the audio signal within a first frequency band from the far end increases and exceeds a third threshold, and to indicate non-interactive mode when the amplitude remains above the first threshold for a third period of time; and

wherein the interactivity detector is operative to indicate interactive mode when the amplitude of the audio signal within a first frequency band from the far end drops from above the first threshold to below a fourth threshold, and to indicate non-interactive mode when the amplitude remains below the fourth threshold for a second period of time.

36. The device in claim 21,

wherein the interactivity detector is operative to indicate interactive mode when the amplitude of the audio signal from the near end increases and exceeds a first threshold, and to indicate non-interactive mode when the amplitude remains above the first threshold for a first period of time;

and

wherein the interactivity detector is operative to indicate interactive mode when the amplitude of the audio signal from the near end drops from above the first threshold to below a second threshold, and to indicate non-interactive mode when the amplitude remains below the second threshold for a second period of time.

37. The device in claim 21,

wherein the interactivity detector is operative to receive a control signal indicating change of interactivity from the far end and the interactivity detector to indicate the interactive mode;

and after the control signal remains the same for a first period of time, the interactivity detector is operative to indicate non-interactive mode.

38. The device in claim 21,

wherein when the interactivity detector is operative to receive control signals from microphones at near end and far end and the control signals change within a fifth period of time, and interactivity detector is operative to indicate interactive mode;

and when the control signals remain the same for a first period of time, the interactivity detector is operative to indicate non-interactive mode.

39. The device in claim 38,

wherein when the interactivity detector delays the indication of interactive mode by a third period of time.

40. A device at a near end for video conferencing with a video conference device at a far end, the device at the near end comprising:

a network interface for coupling with the conference device at the far end;

a controller unit coupled to the network, the audio speaker and the video display for controlling audio and video replay through the speaker and the video display;

a buffer for storing audio, video and control data;

wherein the controller unit is operative to replay the video with a sync delay;

wherein the controller unit is operative to replay the audio with a minimum delay if the mode of the conference is interactive;

wherein the controller unit is operative to receive the audio from far end, store the audio in the buffer and replay the audio with the sync delay if the mode of conference is non-interactive;

wherein the controller comprises an interactivity detector;

wherein the interactivity detector is operative to indicate interactive mode when the amplitude of the audio signal from the far end increases and exceeds a first threshold, and to indicate non-interactive mode when the amplitude remains above the first threshold for a first period of time;

wherein the interactivity detector is operative to indicate interactive mode when the amplitude of the audio signal from the far end drops from above the first threshold to below a second threshold, and to indicate non-interactive mode when the amplitude remains below the second threshold for a second period of time; and

wherein the controller is operative to derive a background noise level at the near end and transmit a control signal indicating the background noise level.

41. A video conference system having a near end and at least one far end, the system comprising:

a microphone, a video camera, a speaker and a video display at the near end;

a network interface at the near end for coupling with the conference system at the far end;

a controller unit at the near end coupled to the network, the microphone, the video camera, the audio speaker and the video display for controlling audio and video generation and replay;

a buffer at the near end for storing audio, video and control data; and

a far end system;

wherein the controller unit is operative to replay the video with a sync delay;

wherein the controller unit is operative to replay the audio with a minimum delay if the mode of the conference at near end is interactive; and

wherein the controller unit is operative to receive the audio from far end, store the audio in the buffer and replay the audio with the sync delay if the mode of conference at near end is non-interactive.

42. The system in claim 41,

wherein the controller is operative to replay the audio with a varied delay if the conference at near end is in a transition mode, wherein conference at near end is switching between non-interactive mode and interactive mode, and within a transition time period.

43. The system in claim 42,

wherein when the transition mode is from non-interactive mode to interactive mode, the controller is operative to replay the audio upon reception with the minimum delay and discarding any audio in the buffer that has not been replayed;

and

wherein the transition time period is substantially zero second.

44. The system in claim 42,

wherein when the transition mode is from interactive mode to non-interactive mode, the controller is operative not to replay audio received until the audio delay is equal to the sync delay;

and

wherein the transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the sync delay.

45. The system in claim 42,

wherein when the transition mode is from interactive mode to non-interactive mode, the controller is operative to replay audio at a slower speed until the audio delay is equal to the sync delay;

wherein the transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the sync delay; and

wherein the replaying speed is slowed digitally without changing the audio pitch.

46. The system in claim 42,

wherein when the transition mode is from non-interactive mode to interactive mode, the controller is operative to replay audio at a faster speed until the audio delay is equal to the minimum delay;

wherein the transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the minimum delay; and

wherein the replaying speed is quickened digitally without changing the audio pitch.

47. The system in claim 46,

wherein the conference at the near end is within interactive mode and after the transition mode, the audio replay speed is switched to the normal speed.

48. The system in claim 46,

wherein the conference at the near end is within interactive mode and after the transition mode, the audio replay speed is switched to a speed slower than the normal speed.

49. The system in claim 41,

wherein the controller at the near end comprising an interactivity detector;

wherein the interactivity detector is operative to indicate interactive mode when the amplitude of the audio signal from the far end increases and exceeds a first threshold, and to indicate non-interactive mode when the amplitude remains above the first threshold for a first period of time; and

wherein the interactivity detector is operative to indicate interactive mode when the amplitude of the audio signal from the far end drops from above the first threshold to below a second threshold, and to indicate non-interactive mode when the amplitude remains below the second threshold for a second period of time.

50. The system in claim 49,

wherein when the interactivity detector delays the indication of interactive mode by a third period of time.

51. The system in claim 41, wherein the far end system comprises:

a microphone, a video camera, a speaker and a video display at a far end;

a network interface at the far end coupled to the network interface at the near end;

a controller at the far end coupled to the above far end devices including the network interface, the microphone, the video camera, the audio speaker and the video display, for controlling audio and video generation and replay at the far end;

a buffer at the far end for storing audio, video and control data;

wherein the controller at the far end is operative to replay the video with a sync delay;

wherein the controller at the far end is operative to replay the audio with a minimum delay if the mode of the conference at the far end is interactive; and

wherein the controller at the far end is operative to receive the audio from the near end, store the audio in the buffer and replay the audio with the sync delay if the mode of conference at the far end is non-interactive.

52. The system in claim 51,

wherein the controller at the far end is operative to replay the audio with a varied delay if the conference at the far end is in a transition mode, wherein the conference at the far end is switching between a non-interactive mode and an interactive mode, and within a far end transition time period.

53. The system in claim 52,

wherein when the transition mode is from non-interactive mode to interactive mode, the controller at far end is operative to replay the audio upon reception with the minimum delay and discarding any audio in the buffer that has not been replayed;

and

wherein the far end transition time period is substantially zero second.

54. The system in claim 52,

wherein when the transition mode is from interactive mode to non-interactive mode, the controller at the far end is operative not to replay audio received until the audio delay is equal to the sync delay;

and

wherein the far end transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the sync delay.

55. The system in claim 52,

wherein when the transition mode is from interactive mode to non-interactive mode, the controller at the far end is operative to replay audio at a slower speed until the audio delay is equal to the sync delay;

wherein the far end transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the sync delay; and

wherein the replaying speed is slowed digitally without changing the audio pitch.

56. The system in claim 52,

wherein when the transition mode is from non-interactive mode to interactive mode, the controller at the far end is operative to replay audio at a faster speed until the audio delay is equal to the minimum delay;

wherein the far end transition time period is the time period from the beginning of the transition mode until the audio delay is equal to the minimum delay; and

wherein the replaying speed is quickened digitally without changing the audio pitch.

57. The system in claim 56,

wherein the conference at the far end is within interactive mode and after the transition mode, the audio replay speed is switched to the normal speed.

58. The system in claim 56,

wherein the conference at the far end is within interactive mode and after the transition mode, the audio replay speed is switched to a speed slower than the normal speed.

59. The system in claim 41,

wherein the controller unit is operative to derive a background noise level at the near end and transmit a control signal indicating the background noise level.

60. A method for controlling synchronization of video and audio in a video conference between a near end and at least one far end, the method comprising the acts of:

detecting a conference mode,

wherein an interactive mode is detected when an amplitude of an audio signal from the far end exceeds a threshold, and

wherein a non-interactive mode is detected when the amplitude of the audio signal remains above the threshold for a period of time;

replaying the audio with a minimum delay if the mode is interactive;

buffering the audio in a buffer and replaying the audio with a sync delay if the mode is not interactive;

measuring a speech level at the far end; and

adjusting the threshold based on the speech level.