Synchronizing method and system

• US 7,913,155 B2
• Filed: 02/15/2006
• Issued: 03/22/2011
• Est. Priority Date: 02/15/2006
• Status: Expired due to Fees

• Current Assignee: Cape Breton University, International Business Machines Corporation
• Original Assignee: Cape Brown University, International Business Machines Corporation
• Inventors: MacNeil, Ronald Francis, Conrod, Sarah Louise, Basson, Sara H., Faisman, Alexander, Kanevsky, Dimitri, Ferre, Wilfredo, Ghez, Julien
• Primary Examiner(s): Ries; Laurie
• Assistant Examiner(s): NGUYEN, CHAU T
• Application Number:
  US11/355,124
• Publication Number:
  US 20070188657A1
• Time in Patent Office: 1,861 Days
• Field of Search: 715/200, 715/203
• US Class Current: 715/203
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23614 Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/43076 of the same content streams on multiple devices, e.g. when family members are watching the same movie on different devices
• CPC Class Codes: H04N 21/4348 Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program

Abstract

A synchronization system and method. Text data is received by a computing device. The text data is associated with audio/video data. The audio/video data is generated during a related performance. The audio/video data and the text data are discrete data. The text data is synchronized to correspond with the audio/video data during the performance. The synchronized text data is displayed by the computing device during the performance.

Claims

1. A method, comprising:

receiving, by a computing device, text data, said text data associated with audio/video data, wherein said audio/video data is generated during a related performance, wherein said related performance is a live performance attended by a user of said computing device, and wherein said audio/video data and said text data are discrete data;

receiving, by a translation server of said computing device, a portion of video from said audio/video data during said performance;

comparing, by said translation server of said computing device, said portion of video to a plurality of pre-stored reference video images on a synchronization server of said computing device;

determining, by said translation server of said computing device based on results of said comparing, a match between a first reference video image of said plurality of pre-stored reference video images and a first image of said portion of video;

synchronizing, said text data to correspond with said audio/video data during said performance, wherein said synchronizing comprises associating said first reference video image to a corresponding portion of said text data and aligning said corresponding portion of said text data with said first image; and

displaying for said user, by a first discrete device and a second discrete device of said computing device, said synchronized text data during said performance, wherein said first discrete device consists of video glasses, wherein said second discrete device consists of a personal digital assistant, and wherein said audio/video data generated during said performance is not displayed by said computing device.

2. The method of claim 1, wherein said text data comprises time stamp data that identifies portions of said text data corresponding to potions of said performance, and wherein said method further comprises:

counting time units beginning with a starting time of said performance, wherein said synchronizing further comprises comparing each of said time units to said time stamp data to determine a match.

3. The method of claim 1, further comprising:

receiving, by the translation server of said computing device, a first portion of audio from said audio/video data during said performance; and

generating a first portion of text from said first portion of said audio/video data, wherein said synchronizing further comprises matching said first portion of text to a corresponding portion of said text data and aligning said corresponding portion of text data with said first portion of said audio/video data.

4. The method of claim 1, wherein said text data is in a first language, wherein said audio/video data comprises a second language, and wherein said first language and said second language are different languages.

5. The method of claim 4, further comprising:

translating said text data into said second language.

6. The method of claim 1, wherein said text data is received in realtime during said performance.

7. The method of claim 1, wherein said text data is received prior to said performance.

8. The method of claim 1, wherein said performance is selected from the group consisting of a television show, a movie, and a live performance.

9. The method of claim 1, wherein said text data describes actions performed in said audio/video data, and wherein said actions are related to images in said audio/video data viewed by a user.

10. The method of claim 1, wherein said text data comprises reference data identifying a starting time for said performance, and wherein said displaying is initiated at said starting time.

11. The method of claim 1, wherein said portion of video is captured by a camera within said first discrete device.

12. A computing device comprising a processor coupled to a computer-readable memory unit, said computer-readable memory unit comprising instructions that when executed by the processor implement an association method, said association method comprising;

receiving, by said computing device, text data, said text data associated with audio/video data, wherein said audio/video data is generated during a related performance, wherein said related performance is a live performance attended by a user of said computing device, and wherein said audio/video data and said text data are discrete data;

receiving, by a translation server of said computing device, a portion of video from said audio/video data during said performance;

comparing, by said translation server of said computing device, said portion of video to a plurality of pre-stored reference video images on a synchronization server of said computing device;

determining, by said translation server of said computing device based on results of said comparing, a match between a first reference video image of said plurality of pre-stored reference video images and a first image of said portion of video;

synchronizing, said text data to correspond with said audio/video data during said performance, wherein said synchronizing comprises associating said first reference video image to a corresponding portion of said text data and aligning said corresponding portion of said text data with said first image; and

displaying for said user, by a first discrete device and a second discrete device of said computing device, said synchronized text data during said performance, wherein said first discrete device consists of video glasses, wherein said second discrete device consists of a personal digital assistant, and wherein said audio/video data generated during said performance is not displayed by said computing device.

13. The computing device of claim 12, wherein said text data comprises time stamp data that identifies portions of said text data corresponding to potions of said performance, and wherein said method further comprises:

counting time units beginning with a starting time of said performance, wherein said synchronizing further comprises comparing each of said time units to said time stamp data to determine a match.

14. The computing device of claim 12, wherein said method further comprises:

receiving, by the translation server of said computing device, a first portion of audio from said audio/video data during said performance; and

generating a first portion of text from said first portion of said audio/video data, wherein said synchronizing further comprises matching said first portion of text to a corresponding portion of said text data and aligning said corresponding portion of text data with said first portion of said audio/video data.

15. The computing device of claim 12, wherein said text data is in a first language, wherein said audio/video data comprises a second language, and wherein said first language and said second language are different languages.

16. The computing device of claim 15, wherein said method further comprises:

translating said text data into said second language.

17. The computing device of claim 12, wherein said text data is received in realtime during said performance.

18. The computing device of claim 12, wherein said text data is received prior to said performance.

19. The computing device of claim 12, wherein said performance is selected from the group consisting of a television show, a movie, and a live performance.

20. The computing device of claim 12, wherein said text data describes actions performed in said audio/video data, and wherein said actions are related to images in said audio/video data viewed by a user.

21. The computing device of claim 12, wherein said text data comprises reference data identifying a starting time for said performance, and wherein said displaying is initiated at said starting time.

22. A computer program product, comprising a computer usable medium comprising a computer readable program code embodied therein, said computer readable program code comprising an algorithm adapted to implement an synchronization method within a computing device, said method comprising:

receiving, by said computing device, text data, said text data associated with audio/video data, wherein said audio/video data is generated during a related performance, wherein said related performance is a live performance attended by a user of said computing device, and wherein said audio/video data and said text data are discrete data;

receiving, by a translation server of said computing device, a portion of video from said audio/video data during said performance;

comparing, by said translation server of said computing device, said portion of video to a plurality of pre-stored reference video images on a synchronization server of said computing device;

determining, by said translation server of said computing device based on results of said comparing, a match between a first reference video image of said plurality of pre-stored reference video images and a first image of said portion of video;

synchronizing, said text data to correspond with said audio/video data during said performance, wherein said synchronizing comprises associating said first reference video image to a corresponding portion of said text data and aligning said corresponding portion of said text data with said first image; and

displaying for said user, by a first discrete device and a second discrete device of said computing device, said synchronized text data during said performance, wherein said first discrete device consists of video glasses, wherein said second discrete device consists of a personal digital assistant, and wherein said audio/video data generated during said performance is not displayed by said computing device.

23. The computer program product of claim 22, wherein said text data comprises time stamp data that identifies portions of said text data corresponding to potions of said performance, and wherein said method further comprises:

counting time units beginning with a starting time of said performance, wherein said synchronizing further comprises comparing each of said time units to said time stamp data to determine a match.

24. The computer program product of claim 22, wherein said method further comprises:

receiving, by the translation server of said computing device, a first portion of audio from said audio/video data during said performance; and

generating a first portion of text from said first portion of said audio/video data, wherein said synchronizing further comprises matching said first portion of text to a corresponding portion of said text data and aligning said corresponding portion of text data with said first portion of said audio/video data.

25. The computer program product of claim 22, wherein said text data is in a first language, wherein said audio/video data comprises a second language, and wherein said first language and said second language are different languages.

26. The computer program product of claim 25, wherein said method further comprises:

translating said text data into said second language.

27. The computer program product of claim 22, wherein said text data is received in realtime during said performance.

28. The computer program product of claim 22, wherein said text data is received prior to said performance.

29. The computer program product of claim 22, wherein said performance is selected from the group consisting of a television show, a movie, and a live performance.

30. The computer program product of claim 22, wherein said text data describes actions performed in said audio/video data, and wherein said actions are related to images in said audio/video data viewed by a user.

31. The computer program product of claim 22, wherein said text data comprises reference data identifying a starting time for said performance, and wherein said displaying is initiated at said starting time.

32. A process for integrating computing infrastructure, comprising integrating computer-readable code into a computer device, wherein the code in combination with the computer system is capable of performing a method comprising:

receiving, by said computing device, text data, said text data associated with audio/video data, wherein said audio/video data is generated during a related performance, wherein said related performance is a live performance attended by a user of said computing device, and wherein said audio/video data and said text data are discrete data;

receiving, by a translation server of said computing device, a portion of video from said audio/video data during said performance;

comparing, by said translation server of said computing device, said portion of video to a plurality of pre-stored reference video images on a synchronization server of said computing device;

determining, by said translation server of said computing device based on results of said comparing, a match between a first reference video image of said plurality of pre-stored reference video images and a first image of said portion of video;

synchronizing, said text data to correspond with said audio/video data during said performance, wherein said synchronizing comprises associating said first reference video image to a corresponding portion of said text data and aligning said corresponding portion of said text data with said first image; and

displaying for said user, by a first discrete device and a second discrete device of said computing device, said synchronized text data during said performance, wherein said first discrete device consists of video glasses, wherein said second discrete device consists of a personal digital assistant, and wherein said audio/video data generated during said performance is not displayed by said computing device.

33. The process of claim 32 wherein said text data comprises time stamp data that identifies portions of said text data corresponding to potions of said performance, and wherein said method further comprises:

counting time units beginning with a starting time of said performance, wherein said synchronizing further comprises comparing each of said time units to said time stamp data to determine a match.

34. The process of claim 32, wherein said method further comprises:

receiving, by the translation server of said computing device, a first portion of audio from said audio/video data during said performance; and

generating a first portion of text from said first portion of said audio/video data, wherein said synchronizing further comprises matching said first portion of text to a corresponding portion of said text data and aligning said corresponding portion of text data with said first portion of said audio/video data.

35. The process of claim 32, wherein said text data is in a first language, wherein said audio/video data comprises a second language, and wherein said first language and said second language are different languages.

36. The process of claim 35, wherein said method further comprises:

translating said text data into said second language.

37. The process of claim 32, wherein said text data is received in realtime during said performance.

38. The process of claim 32, wherein said text data is received prior to said performance.

39. The process of claim 32, wherein said text data describes actions performed in said audio/video data, and wherein said actions are related to images in said audio/video data viewed by a user.