TUNING VIDEO COMPRESSION FOR HIGH FRAME RATE AND VARIABLE FRAME RATE CAPTURE

• US 20140362918A1
• Filed: 08/21/2013
• Published: 12/11/2014
• Est. Priority Date: 06/07/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bushell, John Samuel, Concion, Davide, Saracino, David, Turnquist, Erik
• Granted Patent: US 10,009,628 B2
• Time in Patent Office: Days
• US Class Current: 375/240.15
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/16 for a given display mode, e.g. for interlaced or progressive display mode
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

Video data with high frame rates may be displayed on devices with limited resources (e.g., decoder and/or display resources). These devices may have their resources devoted to other tasks or may not be capable to display the video data at the high frame rates. The coding method may include coding the frames such that additional droppable frames are included in the encoded video data. The decoding method may include dropping droppable frames before the encoded video data is decoded to reduce the number of frames that will be decoded and displayed. These methods may be applied to video data that has a variable frame rate and may be combined with processing the image sequence for slow motion playback.

Claims

1. A coding method, comprising:

coding an image sequence according to predictive coding techniques in which select coded frames serve as prediction references for other coded frames, wherein the coding comprises;

distinguishing frames from the image sequence that are to be displayed during a normal playback mode from other frames of the image sequence that are to be displayed during an enhanced playback mode,

for the frames associated with the normal playback mode, coding the frames according to the predictive coding techniques in which the respective frames are candidates to serve as prediction references for other frames of the image sequence, and

for the frames associated with the enhanced playback mode, coding the frames according to the predictive coding techniques in which the respective frames are prevented from serving as prediction references for other frames of the image sequence.

2. The method of claim 1, wherein the image sequence includes a plurality of frames having a frame rate that exceeds a frame rate of the normal playback mode.

3. The method of claim 1, wherein the image sequence includes a plurality of frames having a frame rate that at least matches a frame rate of the enhanced playback mode.

4. The method of claim 1 further comprising transmitting the coded image sequence over a communication channel.

5. The method of claim 1, wherein the image sequence has a variable frame rate.

6. The method of claim 1, wherein a frame rate of the normal playback mode is a frame rate used to display the image sequence on a display device.

7. The method of claim 1, wherein the frames associated with the enhanced mode include an intra-frame (I-Frame), a predictive frame (P-Frame) and a bidirectional frame (B-Frame).

8. A decoding method, comprising:

determining a playback mode for decoded video from a normal playback mode and an enhanced playback mode,

decoding a coded image sequence according to predictive decoding techniques in which select coded frames are coded using other coded frames as prediction references, wherein the decoding comprises;

when the normal playback mode is selected,

dropping coded frames from the coded image sequence associated with the enhanced playback mode, and

decoding remaining frames according to the predictive decoding techniques, and when the enhanced playback mode is selected,

decoding frames of the coded image sequence that are associated with the enhanced playback mode, which include the frames of the normal playback mode and additional frames from the coded image sequence, according to the predictive decoding techniques.

9. The method of claim 8, wherein prediction references of the coded image sequence are constrained to prevent coded frames from serving as a prediction reference for other portions of the image sequence when the coded frames belong to the enhanced playback mode but not the normal playback mode.

10. The method of claim 8, wherein the coded image sequence includes a plurality of frames having a frame rate that exceeds a frame rate of the normal playback mode.

11. The method of claim 8, wherein the coded image sequence has a variable frame rate.

12. The method of claim 8, wherein the frames associated with the enhanced mode include an intra-frame (I-Frame), a predictive frame (P-Frame) and a bidirectional frame (B-Frame).

13. The method of claim 8, wherein the enhanced mode is selected based on resources available for decoding the coded image sequence.

14. The method of claim 8, wherein the enhanced mode is selected for a portion of the coded image sequence to be displayed with slow motion.

15. A non-transitory storage device that stores a predictively-coded image sequence comprising a plurality of coded frames, wherein select coded frames serve as prediction references for other coded frames, and the image sequence includes a number of coded frames to support both a normal playback mode and an enhanced playback mode having a higher number of frames than normal playback mode, wherein select coded frames associated with the normal playback mode serve as prediction references for other frames of the image sequence, but coded frames associated with the enhanced playback mode do not serve as prediction references for other frames of the image sequence.

16. A method for processing a coded image sequence for slow motion playback, the method comprising:

receiving a coded image sequence with a frame rate that is higher than a display frame rate;

receiving a command selecting a portion of the encoded image sequence for slow motion playback;

retiming the frames in the selected portion to provide frames at the display frame rate; and

dropping coded frames from portions of the coded image sequence outside of the selected portion, the dropped coded frames including frames that are coded according to the predictive coding techniques in which the respective frames are prevented from serving as prediction references for other frames of the image sequence.

17. The method of claim 16 further comprising, after dropping the coded frames, retiming the remaining frames in the portions outside of the selected portion.

18. The method of claim 16 further comprising storing the encoded image sequence without the dropped coded frames in memory.

19. The method of claim 16 further comprising decoding the coded image sequence without the decoding the dropped coded frames.

20. The method of claim 16 further comprising generating an intermediate slow motion portion in a portion of the image sequence adjacent to the selected portion, the intermediate slow motion portion including a playback speed that transitions between a defaults playback speed and a playback speed of the selected portion.