Encoding time management in parallel real-time video encoding

• US 9,762,931 B2
• Filed: 07/23/2015
• Issued: 09/12/2017
• Est. Priority Date: 12/07/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Jia, Wei, Gu, Qunshan
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Colon, Maria Vazquez
• Application Number:
  US14/807,124
• Publication Number:
  US 20150326888A1
• Time in Patent Office: 782 Days
• CPC Class Codes: H04N 19/102 characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/127 Prioritisation of hardware or computational resources
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/157 Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/463 by compressing encoding parameters before transmission
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/57 Motion estimation characterised by a search window with variable size or shape
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression

Abstract

Techniques for efficient coding of digital media data are presented. A resource controller component dynamically allocates computing resources between an estimator component and a coder component. The estimator component generates an initial motion estimation of a raw video frame of a sequence of raw video frames based on a previous raw video frame. The coder component encodes the previous raw video frame to generate a reconstructed video frame in parallel with the generation of the initial motion estimation. When the previous raw video frame is reconstructed, the coder component dynamically determines whether and/or how to refine the motion estimation to encode the raw video frame based on a predefined encoding criterion. In making that determination, the coder component selects and uses a mode(s) of motion estimation refinement from a plurality of available motion estimation refinement modes to encode the raw video frame.

Claims

1. A method, comprising:

generating an initial motion estimation of a raw video frame of a sequence of video frames, based at least in part on a previous raw video frame;

obtaining encoding related statistics for a plurality of modes of refinement motion search, wherein the encoding related statistics for each of the plurality of modes of refinement motion search are based at least in part on actual amounts of time to encode previously-encoded video segments for using the respective mode of refinement motion search;

deriving predicted amounts of time to encode for at least some of the plurality of modes of refinement motion search based at least in part on the encoding related statistics; and

selecting a mode of refinement motion search from the plurality of modes of refinement motion search, in accordance with at least one predefined encoding criterion, to facilitate encoding of at least a portion of the raw video frame within a target total period of time by comparing the respective predicted amounts of time to encode for the at least some of the plurality of modes of refinement motion search; and

encoding the raw video frame using the selected mode of refinement motion search using at least one processor.

2. The method of claim 1, further comprising:

generating the initial motion estimation of the raw video frame in parallel with encoding the previous raw video frame, wherein the encoding of the previous raw video frame produces a reconstructed reference video frame.

3. The method of claim 1, further comprising:

dividing the raw video frame into a specified number of raw video frame segments; and

for a raw video frame segment of the specified number of raw video frame segments, identifying a target time to complete encoding of the raw video frame segment, wherein the target time is at least a portion of the target total period of time, and selecting the mode of refinement motion search that is able to provide higher compression quality for the raw video frame segment while enabling completion of the encoding of the raw video frame segment within the target time.

4. The method of claim 3, further comprising:

comparing a first predicted amount of time to encode with the target time; and

selecting a mode of refinement motion search for no refinement of the initial motion estimation of the raw video frame segment if the first predicted amount of time to encode is greater than the target time, wherein the first predicted amount of time to encode is associated with the mode of refinement motion search for no refinement of the initial motion estimation of the raw video frame segment.

5. The method of claim 3, further comprising:

comparing a second predicted amount of time to encode with the target time; and

selecting a quarter-pixel mode of refinement motion search to refine the initial motion estimation of the raw video frame segment to quarter pixel if the second predicted amount of time to encode is less than or equal to the target time, wherein the second predicted amount of time to encode is associated with the quarter-pixel mode.

6. The method of claim 5, further comprising:

comparing a third predicted amount of time to encode with the target time, wherein the third predicted amount of time to encode is associated with a full-pixel mode of refinement motion search to refine the initial motion estimation of the raw video frame to full pixel;

comparing a fourth predicted amount of time to encode with the target time, wherein the fourth predicted amount of time to encode is associated with a half-pixel mode of refinement motion search to refine the initial motion estimation of the raw video frame to half pixel; and

at least one of;

selecting the half-pixel mode to encode the raw video frame segment if the fourth predicted amount of time is less than or equal to the target time, based at least in part on the second predicted amount of time associated with the quarter-pixel mode being greater than the target time, or selecting the full-pixel mode to encode the raw video frame segment if the third predicted amount of time is less than or equal to the target time, and the fourth predicted amount of time is greater than the target time.

7. The method of claim 3, further comprising:

identifying a second target time to complete encoding of a second raw video frame segment of the specified number of raw video frame segments, and selecting a second type of mode of refinement motion search from the plurality of modes of refinement motion search, wherein the second type of mode of refinement motion search provides higher compression quality for encoding the second raw video frame segment while also enabling completion of the encoding of the second raw video frame segment within the second target time, and wherein the second type of mode of refinement motion search is same as or different than the mode of refinement motion search used to encode the raw video frame segment.

8. The method of claim 1, wherein selecting the mode of refinement motion search is performed such that the selected mode of refinement motion search provides a highest available compression quality for the portion of the raw video frame that enables completion of the encoding of the portion of the raw video frame within the target total period of time.

9. The method of claim 1, wherein the predefined encoding criterion relates to at least one of available bandwidth associated with a communication channel available for transmission of an encoded video frame, available computing cycles for encoding a raw video frame or performing motion estimation of a raw video frame, available amount of time to refine an initial motion estimation, quantization values associated with the raw video frame, residual data associated with the raw video frame, block types in a segment of the raw video frame, segment position in the raw video frame, or statistical information relating to previously encoded video frames or slices.

10. The method of claim 1, wherein deriving predicted amounts of time to encode for at least some of the plurality of modes of refinement motion search is based on a low time estimate and a high time estimate for the respective refinement mode from the plurality of refinement modes.

11. An apparatus, comprising:

a memory having instructions stored therein; and

a processor associated with the memory, wherein the instructions, when executed by the processor, cause the processor to;

generate an initial motion estimation of a raw video frame of a sequence of video frames, based at least in part on a previous raw video frame,

obtain encoding related statistics for a plurality of modes of refinement motion search, wherein the encoding related statistics for each for each of the plurality of modes of refinement motion search are based at least in part on actual amounts of time to encode previously-encoded video segments for using the respective mode of refinement motion search,

derive predicted amounts of time to encode for at least some of the plurality of modes of refinement motion search based at least in part on the encoding related statistics, and

select a mode of refinement motion search from the plurality of modes of refinement motion search, in accordance with at least one predefined encoding criterion, to facilitate encoding of at least a portion of the raw video frame within a target total period of time by comparing the respective predicted amounts of time to encode for the at least some of the plurality of modes of refinement motion search; and

encode the raw video frame using the selected mode of refinement motion search using at least one processor.

12. The apparatus of claim 11, wherein the instructions further cause the processor to:

generate the initial motion estimation of the raw video frame in parallel with encoding the previous raw video frame, wherein the encoding of the previous raw video frame produces a reconstructed reference video frame.

13. The apparatus of claim 11, wherein the instructions further cause the processor to:

divide the raw video frame into a specified number of raw video frame segments, and

for a raw video frame segment of the specified number of raw video frame segments, identify a target time to complete encoding of the raw video frame segment, wherein the target time is at least a portion of the target total period of time, and selecting the mode of refinement motion search that is able to provide higher compression quality for the raw video frame segment while enabling completion of the encoding of the raw video frame segment within the target time.

14. The apparatus of claim 13, wherein the instructions further cause the processor to:

compare a first predicted amount of time to encode with the target time, and

select a mode of refinement motion search for no refinement of the initial motion estimation of the raw video frame segment if the first predicted amount of time to encode is greater than the target time, wherein the first predicted amount of time to encode is associated with the mode of refinement motion search for no refinement of the initial motion estimation of the raw video frame segment.

15. The apparatus of claim 13, wherein the instructions further cause the processor to:

compare a second predicted amount of time to encode with the target time, and

select a quarter-pixel mode of refinement motion search to refine the initial motion estimation of the raw video frame segment to quarter pixel if the second predicted amount of time to encode is less than or equal to the target time, wherein the second predicted amount of time to encode is associated with the quarter-pixel mode.

16. The apparatus of claim 15, wherein the instructions further cause the processor to:

compare a third predicted amount of time to encode with the target time, wherein the third predicted amount of time to encode is associated with a full-pixel mode of refinement motion search to refine the initial motion estimation of the raw video frame to full pixel,

compare a fourth predicted amount of time to encode with the target time, wherein the fourth predicted amount of time to encode is associated with a half-pixel mode of refinement motion search to refine the initial motion estimation of the raw video frame to half pixel, and

at least one of;

select the half-pixel mode to encode the raw video frame segment if the fourth predicted amount of time is less than or equal to the target time, based at least in part on the second predicted amount of time associated with the quarter-pixel mode being greater than the target time, or select the full-pixel mode to encode the raw video frame segment if the third predicted amount of time is less than or equal to the target time, and the fourth predicted amount of time is greater than the target time.

17. The apparatus of claim 13, wherein the instructions further cause the processor to:

identify a second target time to complete encoding of a second raw video frame segment of the specified number of raw video frame segments, and select a second type of mode of refinement motion search from the plurality of modes of refinement motion search, wherein the second type of mode of refinement motion search provides higher compression quality for encoding the second raw video frame segment while also enabling completion of the encoding of the second raw video frame segment within the second target time, and wherein the second type of mode of refinement motion search is same as or different than the mode of refinement motion search used to encode the raw video frame segment.

18. The apparatus of claim 11, wherein the instructions cause the processor to select the mode of refinement motion search such that the selected mode of refinement motion search provides a highest available compression quality for the portion of the raw video frame that enables completion of the encoding of the portion of the raw video frame within the target total period of time.

19. The apparatus of claim 11, wherein the predefined encoding criterion relates to at least one of available bandwidth associated with a communication channel available for transmission of an encoded video frame, available computing cycles for encoding a raw video frame or performing motion estimation of a raw video frame, available amount of time to refine an initial motion estimation, quantization values associated with the raw video frame, residual data associated with the raw video frame, block types in a segment of the raw video frame, segment position in the raw video frame, or statistical information relating to previously encoded video frames or slices.

20. The apparatus of claim 11, wherein the instructions cause the processor to derive the predicted amounts of time to encode for at least some of the plurality of modes of refinement motion search based on a low time estimate and a high time estimate for the respective refinement mode from the plurality of refinement modes.