Device and method for correcting lip sync problems on display devices

• US 9,723,180 B2
• Filed: 01/08/2014
• Issued: 08/01/2017
• Est. Priority Date: 01/08/2014
• Status: Active Grant

• Current Assignee: VIZIO, Inc.
• Original Assignee: VIZIO, Inc.
• Inventors: McRae, Matthew Blake
• Primary Examiner(s): Lee, Michael
• Application Number:
  US14/150,142
• Publication Number:
  US 20150195425A1
• Time in Patent Office: 1,301 Days
• Field of Search: 348515
• CPC Class Codes: H04N 17/04 for receivers
• CPC Class Codes: H04N 2017/006 for television sound
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4302 Content synchronisation processes, e.g. decoder synchronisation
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

A system which synchronizes audio and video, that uses a sensor sensing the display of video and audio from an external device. The sensor creates timestamps for each of the video and audio, and then calculates a difference between those timestamps. The differences sent to the external device which then compensates for the difference.

Claims

1. A system which synchronizes audio and video, comprising:

a sensing device, having an image sensing device sensing a display of a recognizable object in video that is playing on an external display by sensing images on the external display, where the sensing device senses the recognizable object as changing shape or color,

said sensing device also having a sound sensing device sensing a sound associated with the recognizable object as corresponding sounds that are associated with said recognizable object,

said sensing device having a processing device, and creating first information about a time that said recognizable object is detected to change shape or color, and creating second information about times that said corresponding sounds are detected,

wherein said sensing device senses information from multiple different objects in the video that is playing, and wherein the sensing device senses which of multiple audio systems are being used, where all of said multiple audio systems are used with the same external display, and sends information to sync audio and video to the external display, where said information is specific to a specific one of said multiple audio systems that are being used and is different for a first audio system being used with said external display, than for a second audio system being used with said external display,

and said processing device computing a difference value between times of the first information and times of the second information, and sending said difference value to a remote player of said audio and video.

2. The system as in claim 1, wherein said sensing device automatically sends the difference value that causes the audio to be played sooner in relation to the video being displayed, but less than a total difference, in order to require successive iterations of difference values until the video and audio are in synchronization, and continues to sense the display of said recognizable object until the video and audio are in synchronization.

3. The system as in claim 1, wherein the difference is, at a first time, a negative difference where the audio is playing before the video, and at a second time a positive difference where the video is playing before the audio.

4. The system as in claim 1, wherein said sensing device stores multiple time difference samples accumulated at multiple different times, and the processing device in said sensing device averages a computed delay across the multiple time difference samples before sending data indicating the delay over a communications link from the sensing device to the remote player.

5. The system as in claim 1, wherein an output is a wireless stream that is communicated with an external video producing device that produces video that is sensed by said video sensing device, wherein said sensing device includes a camera.

6. The system as in claim 1, wherein said sensing device is a cellular phone, said sensing device including a camera on the cellular phone, and also including a microphone on the cellular phone.

7. The system as in claim 1, wherein said sensing device is a PDA that runs downloaded applications, said sensing device including a camera and a PDA and also including a microphone on the PDA.

8. A video producing system, comprising:

a video output screen, producing video, and producing sounds associated with the video,

a communication device,

a processing device, that receives a communication from an external device that represents a computed difference value, between a time of that a recognizable object is detected to change shape or color, where said recognizable object is playing in a video that is being played to a user, and a time of a sound associated with the recognizable object in the video, and responsive to said communication automatically adjusting a time between the producing of the video and the producing of the sounds, to adjust the time between audio and video in a first way when the audio plays before the video, and to adjust the audio and video in a second way when the video plays before the audio, wherein the video output screen produces its sounds via one of multiple different audio systems, all of said multiple different audio systems being used with the same video output screen, said communication represents a time difference for a specific audio system which is different for a first audio system being used with said video output screen, than it is for a second audio system being used with said video output screen.

9. The system as in claim 8, wherein said processing device runs an application that communicates with an external sensor that senses the video and the audio.

10. The system as in claim 9, wherein said processing device receives a command from the sensor to produce specified audio and video, and responsively receives said communication responsive to producing said specified audio and video.

11. A system which synchronizes audio and video, comprising:

a sensing device, sensing a display of first video information on a remote player, and said sensing device also sensing a corresponding sound from the remote player that is associated with said first video information,

said sensing device calculating a first time difference between said display and said corresponding sound at a first time;

said sensing device storing multiple other time difference samples accumulated at multiple different times, than said first time,

and the sensing device computing an average of a computed delay across the multiple other time difference samples and the first time difference, and sending averaged delay information indicative of the average of the multiple time difference samples to the remote player;

where the averaged delay information is sent to the remote player of said audio and video,

wherein the sensing device senses which of multiple audio systems are being used, where all of said multiple audio systems are used with a same external display, and sends information to sync audio and video to the external display, where said information is specific to a specific one of said multiple audio systems that are being used and is different for a first audio system being used with said external display, than for a second audio system being used with said external display.

12. The system as in claim 11, wherein said sensing device senses information from multiple different objects in the video that is playing.

13. The system as in claim 11, wherein the sensing device not sending an individual computed delay before computing an average of a computed delay across the multiple other time difference samples and the first time difference.

14. A system which synchronizes audio and video, comprising:

a sensing device, having an image sensing device sensing a display of a recognizable object in video that is playing on an external display by sensing images on the external display;

said sensing device also having a sound sensing device, senses which of multiple different audio systems are being used with the same external display,

and sensing a sound associated with the recognizable object as corresponding sounds that are associated with said recognizable object,

said sensing device having a processing device, and creating first information about a time that said recognizable object is detected, and creating second information about times that said corresponding sounds are detected, and said processing device computing a difference value between times of the first information and times of the second information, and sending said difference value to a remote player of said audio and video and sends information to sync audio and video to the external display,

and where said information is specific to a specific one of said multiple audio systems that are being used with the external display,

wherein the sensing device senses which of multiple audio systems are being used, and sends information to sync audio and video to the external display, where said information is specific to a specific one of said multiple audio systems that are being used and is different for a first audio system being used with said external display, than for a second audio system being used with said external display.