Lip sync error detection and correction

• US 10,045,016 B2
• Filed: 02/07/2017
• Issued: 08/07/2018
• Est. Priority Date: 11/12/2010
• Status: Expired due to Fees

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Yang, Chen-Yui, Ching, Chee, Kahn, Paul, Lu, David H., Srdar, Anthony Mark
• Primary Examiner(s): Choudhry, Samina
• Application Number:
  US15/426,711
• Publication Number:
  US 20170150141A1
• Time in Patent Office: 546 Days
• CPC Class Codes: H04N 17/004 for digital television systems
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device

Abstract

A method of managing lip synchronization error in a multimedia content delivery network includes identifying a video packet and an audio packet associated with the video packet and determining a synchronization offset between the audio and video packets at a first monitoring point in the network. The audio and video packets are then detected at a second monitoring point in the network and a second synchronization offset is determined. When a delta between the first synchronization offset and the second synchronization offset exceeds a threshold, lip synchronization error information may be automatically recorded and/or reported to a service provider and corrective action may be taken if potential sources of the lip synchronization error are within the domain of the service provider. The video packet may be identified by a timestamp within the packet and the audio packet may be identified by audio data within the audio packet.

Claims

1. A multimedia synchronization method, comprising:

identifying, by a server, a packet pair comprising a video stream packet including a video packet in a video stream associated with a multimedia program and an audio stream packet comprising an audio packet in an audio stream associated with the multimedia program;

detecting a first offset between the video stream packet and the audio stream packet at a first monitoring point in a multimedia network;

detecting a second offset between the video stream packet and the audio stream packet at a second monitoring point of the multimedia network, wherein the video stream packet and the audio stream packet traverse different network paths between the first monitoring point and the second monitoring point;

determining, by the server, a synchronization delta offset wherein the synchronization delta offset indicates a variation in inter-stream synchronization offset associated with the packet pair between two points in the multimedia network; and

responsive to determining that the synchronization delta offset exceeds a predetermined threshold, initiating, at a monitoring point, a corrective action procedure, wherein the corrective active procedure includes injecting empty or null packets into the component of content that is leading or lagging as appropriate;

wherein the audio stream packet and the video stream packet occur contemporaneously in the multimedia program and further wherein determining the synchronization delta offset includes decoding a presentation timestamp associated with the video stream packet; and

wherein identifying the audio stream packet includes identifying the audio stream packet based on pulse code modulation data included in the audio stream packet.

2. The method of claim 1, wherein the video stream packet is encoded according to a motion picture expert group compliant video encoding.

3. The method of claim 2, wherein the audio stream packet is encoded according to a motion picture expert group compliant audio encoding.

4. The method of claim 1, wherein detecting the first offset includes:

detecting a first network timestamp for the video stream packet and a second network timestamp for the audio stream packet; and

calculating a timestamp difference comprising a difference between the first network timestamp and the second network timestamp.

5. The method of claim 4, wherein detecting the first network timestamp comprises assigning a network time protocol timestamp to the video stream packet.

6. The method of claim 5, wherein assigning the network time protocol timestamp to the video stream packet comprises assigning the network time protocol timestamp to the video stream packet while the video stream packet is being processed at a first monitoring point.

7. The method of claim 4, wherein determining the synchronization delta offset further comprises adjusting the timestamp difference in response to a network time protocol message indicating a network time protocol clock shift.

8. A system comprising:

a processor; and

a non-transitory computer readable storage medium including processor-executable instructions that, when executed by a processor, cause the processor to perform operations comprising;

identifying, by a server, a packet pair comprising a video stream packet including a video packet in a video stream associated with a multimedia program and an audio stream packet comprising an audio packet in an audio stream associated with the multimedia program;

detecting a first offset between the video stream packet and the audio stream packet at a first monitoring point in a multimedia network;

detecting a second offset between the video stream packet and the audio stream packet at a second monitoring point of the multimedia network, wherein the video stream packet and the audio stream packet traverse different network paths between the first monitoring point and the second monitoring point;

determining, by the server, a synchronization delta offset wherein the synchronization delta offset indicates a variation in inter-stream synchronization offset associated with the packet pair between two points in the multimedia network; and

responsive to determining that the synchronization delta offset exceeds a predetermined threshold, initiating, at a monitoring point, a corrective action procedure, wherein the corrective active procedure includes injecting empty or null packets into the component of content that is leading or lagging as appropriate;

wherein the audio stream packet and the video stream packet occur contemporaneously in the multimedia program and further wherein determining the synchronization delta offset includes decoding a presentation timestamp associated with the video stream packet; and

wherein identifying the audio stream packet includes identifying the audio stream packet based on pulse code modulation data included in the audio stream packet.

9. The system of claim 8, wherein the video stream packet is encoded according to a motion picture expert group compliant video encoding.

10. The system of claim 9, wherein the audio stream packet is encoded according to a motion picture expert group compliant audio encoding.

11. The system of claim 8, wherein determining the first offset includes:

determining a first network timestamp for the video stream packet and a second network timestamp for the audio stream packet; and

calculating a timestamp difference comprising a difference between the first network timestamp and the second network timestamp.

12. The system of claim 11, wherein determining the first network timestamp comprises assigning a network time protocol timestamp to the video stream packet.

13. The system of claim 12, wherein assigning the network time protocol timestamp to the video stream packet comprises assigning the network time protocol timestamp to the video stream packet while the video stream packet is being processed at a first monitoring point.

14. The system of claim 13, wherein determining the synchronization delta offset further comprises adjusting the timestamp difference in response to a network time protocol message indicating a network time protocol clock shift.

15. A non-transitory computer readable medium including processor-executable instructions that, when executed by the processor, cause the processor to perform operations, comprising:

identifying, by a server, a packet pair comprising a video stream packet including a video packet in a video stream associated with a multimedia program and an audio stream packet comprising an audio packet in an audio stream associated with the multimedia program;

detecting a first offset between the video stream packet and the audio stream packet at a first monitoring point in a multimedia network;

detecting a second offset between the video stream packet and the audio stream packet at a second monitoring point of the multimedia network, wherein the video stream packet and the audio stream packet traverse different network paths between the first monitoring point and the second monitoring point;

determining, by the server, a synchronization delta offset wherein the synchronization delta offset indicates a variation in inter-stream synchronization offset associated with the packet pair between two points in the multimedia network; and

responsive to determining that the synchronization delta offset exceeds a predetermined threshold, initiating, at a monitoring point, a corrective action procedure, wherein the corrective active procedure includes injecting empty or null packets into the component of content that is leading or lagging as appropriate;

wherein the audio stream packet and the video stream packet occur contemporaneously in the multimedia program and further wherein determining the synchronization delta offset includes decoding a presentation timestamp associated with the video stream packet; and

wherein identifying the audio stream packet includes identifying the audio stream packet based on pulse code modulation data included in the audio stream packet.

16. The non-transitory computer readable medium of claim 15, wherein determining the synchronization delta offset includes:

determining a first network timestamp for the video stream packet and a second network timestamp for the audio stream packet; and

calculating a timestamp difference comprising a difference between the first network timestamp and the second network timestamp.