Tuning video compression for high frame rate and variable frame rate capture

• US 10,009,628 B2
• Filed: 08/21/2013
• Issued: 06/26/2018
• Est. Priority Date: 06/07/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bushell, John Samuel, Concion, Davide, Saracino, David, Turnquist, Erik
• Primary Examiner(s): Atala, Jamie
• Assistant Examiner(s): Boylan, James
• Application Number:
  US13/972,434
• Publication Number:
  US 20140362918A1
• Time in Patent Office: 1,770 Days
• Field of Search: 37524012, 37524015
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/16 for a given display mode, e.g. for interlaced or progressive display mode
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

Video data with high frame rates may be displayed on devices with limited resources (e.g., decoder and/or display resources). These devices may have their resources devoted to other tasks or may not be capable to display the video data at the high frame rates. The coding method may include coding the frames such that additional droppable frames are included in the encoded video data. The decoding method may include dropping droppable frames before the encoded video data is decoded to reduce the number of frames that will be decoded and displayed. These methods may be applied to video data that has a variable frame rate and may be combined with processing the image sequence for slow motion playback.

Claims

1. A coding method, comprising:

coding, by the encoder, an image sequence according to predictive coding techniques in which select coded frames serve as prediction references for other coded frames, wherein the coding comprises;

distinguishing first frames from the image sequence that are to be displayed according to a display rate of a normal playback mode at a decoder from additional droppable frames of the same image sequence associated with a slow motion playback mode that are to be displayed at the decoder at a display rate that is higher than the display rate of the normal playback mode,

for the first frames, coding the first frames according to the predictive coding techniques in which the respective first frames are candidates to serve as prediction references for other first frames of the image sequence, and

for the additional droppable frames, coding the additional droppable frames according to the predictive coding techniques using reference frames from the first frames as prediction references, wherein the additional droppable frames are prevented from serving as prediction references for any of the first frames of the image sequence, such that at least one additional droppable frame can be retimed and presented in slow motion at a decoder.

2. The method of claim 1, wherein the image sequence includes a plurality of frames having a frame rate that exceeds the display rate of the normal playback mode.

3. The method of claim 1, wherein the image sequence includes a plurality of frames having a frame rate that at least matches the display rate of the enhanced playback mode.

4. The method of claim 1 further comprising transmitting the coded image sequence over a communication channel.

5. The method of claim 1, wherein the image sequence has a variable frame rate.

6. The method of claim 1, wherein the display of the normal playback mode rate is a frame rate used to display the image sequence on a display device.

7. The method of claim 1, wherein the remaining frames include an intra-frame (I-Frame), a predictive frame (P-Frame) and a bidirectional frame (B-Frame).

8. The method of claim 1, further comprising:

transmitting the coded image sequence; and

decoding the coded image sequence.

9. A decoding method, comprising:

decoding a coded image sequence according to predictive decoding techniques in which select coded frames are coded using other coded frames as prediction references, wherein the decoding comprises;

distinguishing first frames from the image sequence that are to be displayed according to a display rate of a normal playback mode at a decoder from additional droppable frames of the same image sequence associated with a slow motion playback mode that are to be displayed at the decoder at a display rate that is higher than the display rate of the normal playback mode,

for the first frames, decoding the first frames according to the predictive coding techniques in which the respective first frames are candidates to serve as prediction references for other first frames of the image sequence,

for the additional droppable frames, decoding the additional droppable frames according to the predictive coding techniques in which the remaining frames are decoded predictively using reference frames from the first frames as prediction references, wherein the additional droppable frames are prevented from serving as prediction references for any of the first frames of the image sequence,

for a slow motion portion of the coded image sequence, retiming frames according to a slow motion playback rate,

presenting decoded first frames from outside the slow motion portion at the display rate of normal playback mode, and

presenting retimed first frames and at least one retimed additional droppable frame from the slow motion portion at the display rate of normal playback mode.

10. The method of claim 9, wherein the coded image sequence includes a plurality of frames having a frame rate that exceeds the display rate of the normal playback mode.

11. The method of claim 9, wherein the coded image sequence has a variable frame rate.

12. The method of claim 9, wherein the remaining frames include an intra-frame (I-Frame), a predictive frame (P-Frame) and a bidirectional frame (B-Frame).

13. The method of claim 9, wherein the display rate of the enhanced playback mode is selected based on resources available for decoding the coded image sequence.

14. The method of claim 9, wherein the display rate of the enhanced playback mode is selected for a portion of the coded image sequence to be displayed with slow motion.

15. The method of claim 9, further comprising:

transmitting the coded image sequence; and

decoding the coded image sequence.

16. A non-transitory storage device that stores a predictively-coded image sequence comprising a plurality of coded frames, wherein select coded frames serve as prediction references for other coded frames, first frames are distinguished from the image sequence that are to be displayed according to a display rate of a normal playback mode from additional droppable frames of the image sequence associated with a slow motion playback mode that are to be displayed at a display rate that is higher than the display rate of the normal playback mode, the first frames are coded according to the predictive coding techniques in which the respective first frames are candidates to serve as prediction references for other first frames of the same image sequence, and the additional droppable frames are coded according to the predictive coding techniques using reference frames from the first frames as prediction references, wherein the additional droppable frames are prevented from serving as prediction references for any of the first frames of the same image sequence such that at least one additional droppable frame can be retimed and presented in slow motion at a decoder.

17. The device of claim 16, wherein the coded frames are transmitted and decoded.

18. A method for processing a coded image sequence for slow motion playback at a display, the method comprising:

receiving a coded image sequence with a frame rate of an enhanced playback mode that is higher than a display rate of a normal playback mode of a display that will display a decoded image sequence;

receiving a command selecting a portion of the coded image sequence for slow motion playback at the display;

decoding the frames in the selected portion according to predictive coding techniques in which the frames in the selected portion are candidates to serve as prediction references for other first frames of the same image sequence; and

retiming durations of the decoded frames within the selected portion for slow motion playback from a duration of the higher frame rate of the enhanced playback mode to a display duration of frames at the display rate of the normal playback mode;

decoding remaining frames outside the selected portion according to the predictive coding techniques using reference frames from the first frames as prediction references, wherein certain of the remaining frames outside the selected portion that do not serve as prediction references for first frames of the image sequence are dropped prior to displaying at the display;

presenting decoded first frames from outside the portion for slow motion at the display rate of the normal playback mode; and

presenting retimed first frames and at least one retimed additional droppable frame from the portion for slow motion at the display rate of normal playback mode.

19. The method of claim 18 further comprising, retiming the remaining frames in the portions outside of the selected portion.

20. The method of claim 18 further comprising storing the decoded image sequence without the remaining frames in memory.

21. The method of claim 18 further comprising decoding the coded image sequence without the decoding the remaining frames.

22. The method of claim 18 further comprising generating an intermediate slow motion portion in a portion of the image sequence adjacent to the selected portion, the intermediate slow motion portion including a playback speed that transitions between a default playback speed and a playback speed of the selected portion.