Lip sync error detection and correction

• US 9,565,426 B2
• Filed: 11/12/2010
• Issued: 02/07/2017
• Est. Priority Date: 11/12/2010
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Yang, Chen-Yui, Ching, Chee, Kahn, Paul, Lu, David H., Srdar, Anthony Mark
• Primary Examiner(s): CHOUDHRY, SAMINA F
• Application Number:
  US12/945,250
• Publication Number:
  US 20120120314A1
• Time in Patent Office: 2,279 Days
• Field of Search: 370/503
• US Class Current: 1/1
• CPC Class Codes: H04N 17/004 for digital television systems
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device

Abstract

A method of managing lip synchronization error in a multimedia content delivery network includes identifying a video packet and an audio packet associated with the video packet and determining a synchronization offset between the audio and video packets at a first monitoring point in the network. The audio and video packets are then detected at a second monitoring point in the network and a second synchronization offset is determined. When a delta between the first synchronization offset and the second synchronization offset exceeds a threshold, lip synchronization error information may be automatically recorded and/or reported to a service provider and corrective action may be taken if potential sources of the lip synchronization error are within the domain of the service provider. The video packet may be identified by a timestamp within the packet and the audio packet may be identified by audio data within the audio packet.

Claims

1. A multimedia synchronization method, comprising:

identifying, by a server, a packet pair, the packet comprising a video stream packet comprising a video packet in a video stream associated with a multimedia program and an audio stream packet comprising an audio packet in an audio stream associated with the multimedia program;

receiving the video stream packet at a first monitoring point in a multimedia network;

decoding a timestamp associated with the video stream packet;

determining a first inter-stream synchronization offset, the first inter-stream synchronization offset indicative of a time difference between arrival of the video stream packet at the first monitoring point and arrival of the audio stream packet at the first monitoring point;

responsive to detecting the video stream packet and the audio stream packet at a second monitoring point of the multimedia network, determining a second inter-stream synchronization offset, the second inter-stream synchronization offset indicative of a time difference between arrival of the video stream packet at the second monitoring point and arrival of the audio stream packet at the second monitoring point, wherein the video stream packet traverses a first network path, comprising a first set of network switches, between the first monitoring point and the second monitoring point and the audio stream packet traverses a second network path, comprising a second set of network switches, different than the first set of network switches, between the first monitoring point and the second monitoring point;

determining, by the server, a synchronization delta indicative of a difference between the first inter-stream synchronization offset and the second inter-stream synchronization offset, wherein the synchronization delta indicates a variation in inter-stream synchronization offset associated with the packet pair between two points in the multimedia network; and

responsive to determining that the synchronization delta exceeds a predetermined threshold;

logging, by the server, synchronization error information; and

initiating, at a monitoring point, a corrective action procedure.

2. The method of claim 1, wherein identifying the video stream packet comprises identifying the timestamp associated with the video stream packet.

3. The method of claim 2, wherein the video stream packet is encoded according to a motion picture expert group compliant video encoding and wherein the timestamp comprises a presentation timestamp.

4. The method of claim 3, wherein the audio stream packet is encoded according to a motion picture expert group compliant audio encoding and wherein identifying the audio stream packet includes identifying the audio stream packet based on pulse code modulation data included in the audio stream packet.

5. The method of claim 4, wherein the audio stream packet and the video stream packet occur contemporaneously in a multimedia content program.

6. The method of claim 1, wherein determining the inter-stream synchronization offset comprises:

determining a video network timestamp for the video stream packet and an audio network timestamp for the audio stream packet; and

determining a difference between the video network timestamp and the audio network timestamp.

7. The method of claim 6, wherein determining the video network timestamp comprises assigning a network time protocol timestamp to the video stream packet.

8. The method of claim 7, wherein assigning the network time protocol timestamp to the video stream packet comprises assigning the network time protocol timestamp to the video stream packet while the video stream packet is being processed at a first monitoring point.

9. The method of claim 6, wherein determining the synchronization offset further comprises adjusting the determined difference between the video network timestamp and the audio network timestamp, wherein the adjusting is, at least in part, in response to a network time protocol message indicating a network time protocol clock shift.

10. An audio/video synchronization server comprising:

a processor having access to a non-transitory computer readable storage medium including processor-executable instructions that, when executed by a processor, cause the processor to perform operations comprising;

identifying a packet pair, the packet comprising a video stream packet comprising a video packet in a video stream associated with a multimedia program and an audio stream packet comprising an audio packet in an audio stream associated with the multimedia program;

responsive to detecting the video stream packet and the audio stream packet at a first monitoring point in a multimedia content network, determining a first inter-stream synchronization offset, the first inter-stream synchronization offset indicative of a time difference between arrival of the video stream packet at a first monitoring point of the multimedia content network and arrival of the audio stream packet at the first monitoring point;

responsive to detecting the video stream packet and the audio stream packet at a second monitoring point in the multimedia content network, determining a second inter-stream synchronization offset, the second inter-stream synchronization offset indicative of a time difference between arrival of the video stream packet at the second monitoring point and arrival of the audio stream packet at the second monitoring point in the multimedia content network, wherein the video stream packet traverses a first network path, comprising a first set of network switches, between the first monitoring point and the second monitoring point and the audio stream packet traverses a second network path, comprising a second set of network switches, different than the first set of network switches, between the first monitoring point and the second monitoring point;

determining a synchronization delta indicative of a difference between the first inter-stream synchronization offset and the second inter-stream synchronization offset, wherein the synchronization delta indicates a variation in inter-stream synchronization offset associated with the packet pair between two points in the multimedia content network; and

responsive to determining the synchronization delta exceeding a predetermined threshold;

logging synchronization error information, indicative of the audio stream packet, the video stream packet, the first inter-stream synchronization offset, and the second inter-stream synchronization offset; and

initiating, at a monitoring point, a corrective action procedure by adding packets to at least one of;

a video stream carrying the video stream packet and an audio stream carrying the audio stream packet.

11. The server of claim 10, wherein the first monitoring point comprises an encoder of the multimedia content delivery network and the second monitoring point comprises a central office switch of the multimedia content network.

12. The server of claim 11, further comprising determining inter-stream synchronization offset between the second monitoring point and a third monitoring point, wherein the third monitoring point comprises customer premises equipment at a client site of the multimedia content network.

13. The server of claim 10, wherein identifying the video stream packet and the audio stream packet includes identifying a timestamp in the video stream packet and pulse code modulation information in the audio stream packet.

14. The server of claim 10, wherein determining synchronization offset comprises determining a difference between network time protocol timestamps assigned to the video stream packet and the audio stream packet.

15. A non-transitory computer readable medium including processor executable instructions that, when executed by a processor, cause the processor to perform operations comprising:

identifying a video stream packet, comprising a video packet in a video stream associated with a multimedia program, of a packet pair based on timestamp information in the video packet;

identifying an audio stream packet, comprising an packet in an audio stream associated with the multimedia program, of the packet pair based on pulse code modulation data in the audio stream packet;

responsive to detecting the video stream packet and the audio stream packet at an upstream monitoring point in a multimedia network, determining a first inter-stream synchronization offset, the first inter-stream synchronization offset indicative of a time difference between arrival of the video stream packet at the upstream monitoring point and arrival of the audio stream packet at the upstream monitoring point; and

responsive to detecting the video stream packet and the audio stream packet at a midstream monitoring point in the multimedia network, determining a second inter-stream synchronization offset, the second inter-stream synchronization offset indicative of a time difference between arrival of the video stream packet at the midstream monitoring point and arrival of the audio stream packet at the midstream monitoring point in the multimedia network, wherein the video stream packet traverses a first network path, comprising a first set of network switches, between the first monitoring point and the second monitoring point and the audio stream packet traverses a second network path, comprising a second set of network switches, different than the first set of network switches, between the first monitoring point and the second monitoring point;

determining a second inter-stream synchronization offset between the video stream packet and the audio stream packet at a midstream monitoring point in the multimedia network;

determining a synchronization delta indicative a difference between the first inter-stream synchronization offset and the second inter-stream synchronization offset, wherein the synchronization delta indicates a variation in inter-stream synchronization offset associated with the packet pair between two points in the multimedia content delivery network; and

responsive to determining the synchronization delta exceeding a predetermined threshold;

logging synchronization error information, indicative of the audio stream packet, the video stream packet, the first inter-stream synchronization offset, and the second inter-stream synchronization offset; and

initiating, at a monitoring point, a corrective action procedure by adding packets to at least one of;

a video stream carrying the video stream packet and an audio stream carrying the audio stream packet.

16. The system of claim 15, wherein the timestamp information comprises a presentation timestamp in the video stream packet and the pulse code modulation data comprises audio compression modulation data.

17. The system of claim 15, wherein the upstream monitoring point comprises an encoder of a multimedia content delivery network and the midstream monitoring point comprises a central office switch of the multimedia network.

18. The system of claim 17, further comprising determining inter-stream synchronization offset between the midstream monitoring point and a downstream monitoring point, wherein the downstream monitoring point comprises customer premises equipment at a client site of the multimedia network.

19. The system of claim 15, wherein identifying the video stream packet comprise identifying the presentation timestamp in the video stream packet.

20. The system of claim 15, wherein determining inter-stream synchronization offset comprises determining a difference between network time protocol timestamps assigned to the video stream packet and the audio stream packet.