Television program transmission, storage and recovery with audio and video synchronization

• US 20060015348A1
• Filed: 09/14/2005
• Published: 01/19/2006
• Est. Priority Date: 10/12/1994
• Status: Active Grant

• Current Assignee: Pixel Instruments Corporation
• Original Assignee: Pixel Instruments Corporation
• Inventors: Cooper, J. Carl, Anderson, Steve
• Granted Patent: US 8,428,427 B2
• Time in Patent Office: Days
• US Class Current: 704/500
• CPC Class Codes: G10L 21/04 Time compression or expansion
• CPC Class Codes: G11B 20/00007 Time or data compression or expansion audio compression based on psychoacoustics G10L19/00; data processing for reproducing audio data at different playback speeds G10L21/04; video compression H04N19/00; data compression per se H03M7/30
• CPC Class Codes: H04B 1/662 using a time/frequency relationship, e.g. time compression or expansion
• CPC Class Codes: H04N 21/4126 The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/4147 PVR [Personal Video Recorder] H04N5/76 takes precedence ; arrangements for broadcast with accumulation-type receivers H04H20/40
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42209 for displaying non-command information, e.g. electronic program guide [EPG], e-mail, messages or a second television channel
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/42227 the keys being reprogrammable, e.g. soft keys
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4331 Caching operations, e.g. of an advertisement for later insertion during playback
• CPC Class Codes: H04N 21/4333 Processing operations in response to a pause request
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/47214 for content reservation or setting reminders; for requesting event notification, e.g. of sport results or stock market stock exchange data over packet-switching network H04L12/1804; push services over packet-switching network H04L12/1859; notification of incoming messages in packet switching networks H04L51/224
• CPC Class Codes: H04N 21/482 End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/485 End-user interface for client configuration
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/775 between a recording apparatus and a television receiver
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate
• CPC Class Codes: H04N 9/8042 : involving data reduction

Abstract

An apparatus and methods are described for receiving, storing, and recovering from storage one or more television program while maintaining proper audio to video synchronization or lip sync when the program is displayed. Preferably the audio and video portions are in digital form and may be compressed or uncompressed. The audio and video may be recorded on a physical storage medium such as tape, an optical or magnetic computer disk, or solid state memory or other such device providing user control functions including start, stop, record, play and search, with increased reproduction capability of such stored programs being facilitated by the present invention. Timing information is recovered from the audio and video to be displayed utilized to alter the time duration of a contiguous signal block (the alpha length) of the audio and/or video to correct for any timing errors therebetween. Correction of frequency artifacts such as pitch errors which may be created with the alteration of alpha length is also taught.

Claims

1. In a system providing an entertainment program comprised of an audio and a video portion which portions are stored in compressed digital form, the content of said audio and video portions being related to each other, said video and audio portions each including time flags corresponding to said respective portion, which time flags are also stored by said system, a method of retrieving from storage and presenting to the viewer sound and images that are perceived by said viewer as properly timed with each other, said method including the following steps:

a) retrieving from storage said audio and video portions in compressed form and said time flags, decompressing said audio and video portions, said decompressed audio portion comprising digital samples of a digital audio signal and said decompressed video portion comprising digital samples of a digital video signal, each said digital audio and digital video signal having a respective alpha length, which alpha length is the time it takes for a contiguous block of digital samples to play;

b) in response to comparing at least one time flag associated with said digital video signal and one time flag associated with said digital audio signal of step a), changing the alpha length of at least one of said digital audio signal or said digital video signal to reduce any timing errors between them;

c) wherein, when the changing step b) operates by shortening the alpha length of a contiguous block of digital samples in said digital audio signal, one or more of the digital samples included therein are deleted and when said changing step b) operates by increasing said alpha length, digital samples are added thereto.

2. The method as claimed in claim 1 wherein in step c), when digital samples are added to said contiguous block, they are comprised of repeated ones of the samples originally making up said contiguous block.

3. The method as claimed in claim 1 wherein in step c), when digital samples are added to said contiguous block, they are formed by interpolating ones of the samples originally making up said contiguous block.

4. The method as claimed in claim 1 wherein in step c), when samples are added to or deleted from said contiguous block it is done in a manner such that audible frequency components of the audible version of the resulting digital audio signal are maintained at the same pitch they originally had in respect to the unaltered version of said digital audio signal from step a).

5. The method as claimed in claim 1 wherein in step b), said contiguous block of samples of said digital audio signal are temporarily stored at a first rate, with said stored samples recovered from storage at the same rate when said alpha length is not changed and recovered from storage at a different rate when said alpha length is changed.

6. The method as claimed in claim 1, 2, 3, 4 or 5 wherein said compressed digital form conforms to an MPEG standard.

7. In a system for providing an entertainment program of a determinable length of time which is stored, said entertainment program comprising audio and video portions each of which is of substantially the same length of time as said program, said audio and video portions being synchronized with each other and then stored in compressed digital form with corresponding time flags, a method for providing properly timed audio and video portions of said program for presentation to a viewer, said method including the following steps:

a) receiving a version of said program via transmission or physical storage medium and providing in decompressed digital sample form a digital audio signal and a digital video signal, each said digital audio and digital video signal being substantially the same length as the other;

b) recovering said time flags and changing the length of one of said digital audio signal or said digital video signal to reduce any synchronization error between them;

c) wherein, when said changing in step b) operates by shortening the length of said digital audio signal, one or more of the digital samples included therein are deleted and when said changing operates by increasing said length, digital samples are added thereto.

8. The method as claimed in claim 7 wherein in step c), when samples are added to said digital audio signal they are comprised of repeated ones of the samples making up said digital audio signal.

9. The method as claimed in claim 7 wherein in step c), when samples are added to said digital audio signal they are formed by interpolating ones of the samples making up said digital audio signal.

10. The method as claimed in claim 7 wherein in step c), when samples are added to or deleted from said digital audio signal they are chosen in a manner such that audible frequency components of the audible version of the resulting digital audio signal are maintained at the same pitch they originally had.

11. The method as claimed in claim 7 wherein in step b) said samples of said digital audio signal are stored at a first rate, and recovered from storage at the same rate when said length is not being changed and recovered from storage at a different rate when said length is being changed.

12. The method as claimed in claim 7, 8, 9, 10 or 11 wherein said compressed digital form conforms to an MPEG standard.

13. A method of providing a television like entertainment program comprising audio and video, wherein audio and video digital data is received with corresponding time flags, and wherein said method provides for the storage and recall of said audio and video portions in compressed form with properly time aligned presentation of uncompressed audio and video to a viewer, said system including the steps of:

a) storing when received, said digital data making up portions of said compressed audio and said compressed video,

b) retrieving stored, compressed audio data from said storage a), recovering audio time flags, converting said compressed audio data to an uncompressed audio signal and outputting said uncompressed audio signal,

c) retrieving stored, compressed video data from said storage a), recovering video time flags, converting said compressed video data to an uncompressed video signal and outputting said uncompressed video signal,

d) comparing the relative timing of said recovered audio and video time flags of steps b) and c) and outputting a timing signal indicating the relative timing of said uncompressed audio signal and said uncompressed video signal,

e) modifying said uncompressed audio signal from circuit b) in response to said timing signal of d) by changing the alpha length thereof to reduce any relative timing error between said uncompressed audio signal and said uncompressed video signal,

f) wherein when said changing in step e) operates to shorten the alpha length of said uncompressed audio signal one or more of the digital samples included therein are deleted and when said changing increases the alpha length of said uncompressed audio signal, digital samples are added thereto.

14. The method as claimed in claim 13 wherein said storing of a) is performed with one of:

a) a magnetic disk,

b) a computer disk drive

c) an optical disk, or

d) solid state random access memory.

15. The method as claimed in claim 13 wherein said comparing step d) operates by comparing said video time flag and a delayed version of said recovered audio time flag.

16. The method as claimed in claim 13 wherein said step e) operates to delay said audio signal and said comparing step d) operates by comparing said video time flag and a delayed version of said recovered audio time flag which delayed version audio time flag is delayed by the same amount as the delay of said audio signal of step e).

17. The method as claimed in claim 13 wherein said modifying step e) includes pitch correction operative to maintain proper pitch of said uncompressed audio signal whenever the alpha length thereof is changed.

18. The method as claimed in claim 13, 14, 15, 16 or 17 wherein in step f) only individual samples are deleted in each alpha length.

19. The method of providing a stored television program for presentation to a viewer with proper lip sync, said method including the steps:

a) receiving the video portion of said television program and storing said video portion in compressed form in a memory comprising one or more of a magnetic disk, computer disk drive, optical disk or semiconductor memory,

b) generating a video time flag in response to said received video portion of a) and storing said video time flag with said compressed form video portion in said memory as part of step a),

c) receiving an audio portion of said television program and storing said audio portion in compressed form in said memory,

d) generating an audio time flag in response to said received audio portion of b) and storing said audio time flag with said compressed form audio portion in said memory as part of step c),

e) recovering said stored video portion from said memory and decompressing it,

f) recovering said stored audio portion from said memory and decompressing it,

g) recovering said video time flag from said stored video portion,

h) recovering said audio time flag from said stored audio portion,

i) comparing the relative timing of said recovered audio and video time flags of g) and h) to indicate when decompressed audio of step f) leads or lags decompressed video of step e),

j) when said decompressed audio lags said decompressed video, decreasing the alpha length of said decompressed audio and when said decompressed audio leads said decompressed video, increasing the alpha length of said decompressed audio.

20. The method as claimed in claim 19 wherein said comparing step i) operates by comparing said video time flag and a delayed version of said recovered audio time flag.

21. The method as claimed in claim 19 wherein said step j) operates to delay said audio signal and said comparing step i) operates by comparing said video time flag and a delayed version of said recovered audio time flag which delayed version is delayed by the same amount as the delay of said audio signal of step j).

22. The method as claimed in claim 19 wherein said step j) includes pitch correction operative to maintain proper pitch of said uncompressed audio signal whenever the alpha length thereof is changed.

23. The method of providing to a viewer with proper lip sync, a television program which is stored with compressed audio and video portions, each including a time flag, said method including the steps of:

a) recovering said stored video portion from storage and decompressing it,

b) recovering said stored audio portion from storage and decompressing it,

c) recovering a video time flag from said stored video portion,

d) recovering a audio time flag from said stored audio portion,

e) delaying said audio to provide delayed audio,

f) in response to said recovered time flags of c) and d) indicating whether said delayed audio of step e) leads or lags decompressed video of step a),

g) decreasing the delay of e) when said delayed audio lags said decompressed video and increasing the delay of said delayed audio and when said delayed audio leads said decompressed video.

24. The method as in claim 23 wherein said audio time flag of d) is delayed by an amount substantially the same as the audio delay of e) before performing step f).

25. The method as in claim 23 wherein said audio is delayed in compressed form in step e).

26. The method as in claim 23 wherein said audio is delayed in uncompressed form in step e).

27. The method as in claim 23 wherein said video is delayed by recovery step a) in compressed or uncompressed form relative to recovered audio of b).

28. The method as in claim 23 wherein a plurality of audio channels are stored and in step e) each of said plurality of audio channels is delayed, with any increase or decrease in said delay taking place such that the relative timing therebetween is maintained.