Lip synchronization in a video conference

• US 9,055,332 B2
• Filed: 10/25/2011
• Issued: 06/09/2015
• Est. Priority Date: 10/26/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Westin, Patrik
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US13/281,092
• Publication Number:
  US 20120098923A1
• Time in Patent Office: 1,323 Days
• Field of Search: 348/14.01, 348/14.12, 348/14.09, 348/14.08
• US Class Current: 1/1
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/23608 Remultiplexing multiplex streams, e.g. involving modifying time stamps or remapping the packet identifiers
• CPC Class Codes: H04N 21/4344 Remultiplexing of multiplex streams, e.g. by modifying time stamps or remapping the packet identifiers
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

Disclosed herein is a method of synchronizing of a plurality of data streams having time-stamped packets, wherein the plurality of data streams are of a first type and at least a second type different than the first type, of which at least one stream of the first type is to be processed at a server. The method includes re-dating a timestamp of a packet, using a processor, based on an estimated relative delay for transport up to the server between first-type and second-type streams and a predicted relative delay representative of a processing time duration between first-type and second-type streams, thereby allowing the relative delays to cancel at least partially.

Claims

1. A method of synchronizing a plurality of data streams having time-stamped packets, wherein the plurality of data streams are of a first type and at least a second type different than the first type, of which at least one stream of the first type is to be processed at a server, the method comprising:

re-dating a timestamp of a packet, using a processor, based on an estimated relative delay for transport up to the server between first-type and second-type streams and a predicted relative delay representative of a processing duration difference between first-type and second-type streams, thereby allowing the relative delays to cancel at least partially.

2. The method of claim 1, further comprising:

processing the at least one stream of the first type to obtain at least one processed stream of time-stamped data packets,

wherein at least one of a data packet in the at least one processed stream or a data packet in a second-type stream is re-dated.

3. The method of claim 2, wherein the at least one stream of the first type includes two first-type streams and wherein the two first-type streams are processed to obtain the at least one processed stream of time-stamped data packets.

4. The method of claim 1, further comprising:

relaying packets of the first-type and second-type streams independently of one another.

5. The method of claim 1, wherein the relative transport delay is estimated as a difference between a first period, elapsed from a timestamp of a packet in a first-type stream to a receipt time of this packet, and a second period, elapsed from a timestamp of a packet in a second-type stream to a receipt time of this packet.

6. The method of claim 1, wherein:

the streams are ordered in synchronous pairs, each pair includes a first-type stream and a second-type stream;

first-type streams are to be processed to obtain a processed stream; and

timestamps in packets of each second-type stream are re-dated, to preserve synchronicity of the stream and a contribution to the processed stream made by the first-type stream of the pair, and these packets are relayed.

7. The method of claim 1, wherein the first type is audio and the second type is video, and wherein streams of the first type are to be processed by combination.

8. The method of claim 1, wherein each stream includes a sub-stream of data packets and an accompanying sub-stream of control packets including timestamps associating a point in the sub-stream of data packets with a point in time, wherein re-dating the timestamp of a packet comprises:

re-dating a timestamp of a control packet.

9. The method of claim 8, wherein:

the first-type streams are processed to generate a processed stream including a sub-stream of data packets and a sub-stream of control packets, each control packet carrying a timestamp associating a point in the sub-stream of data packets with a point in reference time at which this point in the sub-stream of data packets was obtained by processing; and

the re-dating includes setting a timestamp of a control packet accompanying a second-type stream to the sum of a current reference time, the relative processing delay and the relative transport delay between first-type and second-type streams.

10. A apparatus for handling a plurality of data streams of time-stamped packets, comprising:

a first receiving means for receiving packets in streams of a first type to be processed;

a second receiving means for receiving packets in streams of a second type different than the first type; and

a relaying means for re-dating timestamps of packets in streams of at least one of the first and second types, on the basis of a predicted relative delay representative of processing time duration difference between first-type and second-type streams and an estimated relative delay for transport up to the apparatus between first-type and second-type streams, thereby allowing the relative delays to cancel at least partially.

11. The apparatus of claim 10, further comprising:

a processing means for inputting at least one stream of the first type and for outputting a processed stream,

wherein the relaying means is configured to relay packets in streams of the one type after completion of re-dating.

12. The apparatus of claim 10, wherein:

the first and second receiving means record receipt times of control packets; and

the relaying means estimates the relative transport delay as a difference between a first period, elapsed from a timestamp to a receipt time of a packet in a first-type stream, and a second period, elapsed from a timestamp to a receipt time of a packet in a second-type stream.

13. The apparatus of claim 10, wherein the apparatus is configured to handle synchronous pairs of streams, each synchronous pair of streams including a first-type stream and a second-type stream, wherein:

the processing means processes first-type streams to obtain a processed stream; and

the relaying means re-dates timestamps in packets of each second-type stream, to preserve synchronicity of the stream and a contribution to the processed stream made by the first-type stream of the pair, and to relay packets with re-dated timestamps.

14. The apparatus of claim 13, wherein the apparatus is configured to handle streams each including a sub-stream of data packets and an accompanying sub-stream of control packets including timestamps each associating a point in the sub-stream of data packets with a point in time,

wherein the relaying means re-dates timestamps of control packets.

15. An apparatus for enabling synchronization of a plurality of data streams having time-stamped packets, wherein the plurality of data streams are of a first type and at least a second type different than the first type, of which at least one stream of the first type is due to be processed at a server, the apparatus comprising:

a memory; and

a processor configured to execute instructions stored in the memory to;

re-date a timestamp of a packet based on an estimated relative delay for transport up to the server between first-type and second-type streams and a predicted relative delay representative of a processing time difference between first-type and second-type streams.

16. The apparatus of claim 15, wherein the processor is further configured to:

process the at least one stream of the first type to obtain at least one processed stream of time-stamped data packets,

wherein at least one of a data packet in the at least one processed stream or a data packet in a second-type stream is re-dated.

17. The apparatus of claim 16, wherein the at least one stream of the first type includes two first-type streams and wherein the two first-type streams are processed to obtain the at least one processed stream of time-stamped data packets.

18. The apparatus of claim 15, wherein the processor is further configured to:

relay packets of the first-type and second-type streams independently of one another.

19. The apparatus of claim 15, wherein the relative transport delay is estimated as a difference between a first period, elapsed from a timestamp of a packet in a first-type stream to a receipt time of this packet, and a second period, elapsed from a timestamp of a packet in a second-type stream to a receipt time of this packet.

20. The apparatus of claim 15, wherein:

the streams are ordered in synchronous pairs, each pair includes a first-type stream and a second-type stream;

first-type streams are due to be processed to obtain a processed stream; and

timestamps in packets of each second-type stream are re-dated, to preserve synchronicity of the stream and a contribution to the processed stream made by the first-type stream of the pair, and these packets are relayed.