CONFIGURABLE MOTION ESTIMATION SEARCH SYSTEMS AND METHODS

• US 20170094293A1
• Filed: 09/30/2015
• Published: 03/30/2017
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Chou, Jim C., Rygh, Mark P., Ct, Guy
• Granted Patent: US 10,225,572 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 19/109 among a plurality of temporal predictive coding modes
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/157 Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/182 the unit being a pixel
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/439 using cascaded computational arrangements for performing a single operation, e.g. filtering

Abstract

System and method for improving operational efficiency of a video encoding pipeline used to encode image data. The video encoding pipeline includes a motion estimation setup block, which dynamically adjusts a setup configuration of the motion estimation block based at least in part on operational parameters of the video encoding pipeline and select an initial candidate inter-frame prediction mode based at least on the setup configuration, a full-pel motion estimation block, which determines an intermediate candidate inter-frame prediction mode by performing a motion estimation search based on the initial candidate inter-frame prediction mode, a sub-pel motion estimation block, which determines a final candidate inter-frame prediction by performing a motion estimation search based on the intermediate candidate inter-frame prediction mode, and a mode decision block, which determines a rate-distortion cost associated with the final candidate inter-frame prediction mode and determines a prediction mode used to prediction encoding the image data.

Claims

1. A computing device comprising a video encoding pipeline configured to encode image data, wherein the video encoding pipeline comprises:

a motion estimation block comprising;

a motion estimation setup block configured to;

dynamically adjust a setup configuration of the motion estimation block based at least in part on operational parameters of the video encoding pipeline; and

select a first initial candidate inter-frame prediction mode based at least on the setup configuration;

a full-pel motion estimation block configured to determine an intermediate candidate inter-frame prediction mode by performing a first motion estimation search based on the first initial candidate inter-frame prediction mode; and

a sub-pel motion estimation block configured to determine a final candidate inter-frame prediction mode by performing a second motion estimation search based on the intermediate candidate inter-frame prediction mode; and

a mode decision block configured to determine a rate-distortion cost associated with the final candidate inter-frame prediction mode and to determine a prediction mode used to prediction encode the image data based at least in part on the rate-distortion cost.

2. The computing device of claim 1, wherein:

the full-pel motion estimation block is configured to perform a third motion estimation search in a first pixel-search-area around a first location indicated by an automatic inter-frame prediction mode; and

the motion estimation block is configured to;

determine a second pixel-search-area around a second location indicated by the first initial candidate inter-frame prediction mode, wherein the second pixel-search-area is searched by the full-pel motion estimation block when the first initial candidate inter-prediction mode is selected;

determine amount of overlap between the first pixel-search-area and the second pixel-search-area; and

adjust the first initial candidate inter-prediction mode to move the second location when the amount of overlap is greater than a threshold.

3. The computing device of claim 1, wherein the video encoding pipeline comprises:

an inter-frame prediction block configured to determine a first luma prediction sample based at least in part on the final candidate inter-frame prediction mode;

an intra-frame prediction block configured to determine a candidate intra-frame prediction mode and to determine a second luma prediction sample based at least in part on the final candidate intra-frame prediction mode;

a chroma reconstruction block configured to determine chroma components of reconstructed image data based at least in part on the prediction mode determined by the mode decision block;

a luma reconstruction block configured to determine a luma component of the reconstructed image data based at least in part on the prediction mode determined by the mode decision block;

a back-end-filter block configured to filter the reconstructed image data; and

a syntax element binarization block configured to generate binarized syntax elements used to indicate the prediction mode determined by the mode decision block; and

a transcode pipeline configured to entropy encode the binarized syntax elements.

4. The computing device of claim 1, comprising a controller configured to:

determine an operation mode of the motion estimation block based at least in part on the operational parameters of the video encoding pipeline, wherein the operation mode is configured to indicate operating duration provided the motion estimation block and the operational parameters comprise image frame resolution, display refresh rate, desired power consumption, or any combination thereof; and

instruct the motion estimation block to dynamically adjust the setup configuration in response to changes in the operation mode.

5. The computing device of claim 1, comprising a controller configured to:

determine the operational parameters of the video encoding pipeline, wherein the operational parameters comprise image frame resolution, display refresh rate, desired power consumption, or any combination thereof; and

instruct the motion estimation block to dynamically adjust the setup configuration in response to changes in the operational parameters.

6. The computing device of claim 1, wherein the motion estimation block is configured to dynamically adjusting the setup configuration by changing setup parameters used to control operation of the motion estimation block;

wherein the setup parameters comprise number of each type of initial candidate inter-frame prediction modes to select, location of initial candidate inter-frame prediction modes to select, pixel-search-area around initial candidate inter-frame prediction modes in the full-pel motion estimation block, pixel-search-area around intermediate candidate inter-frame prediction modes in the sub-pel motion estimation block, or any combination thereof.

7. The computing device of claim 1, wherein:

the full-pel motion estimation block is configured to perform the first motion estimation search to determine a first reference sample at an integer pixel location in a first pixel-search-area around a first location indicated by the first initial candidate inter-frame prediction mode; and

the sub-pel motion estimation block is configured to perform the second motion estimation search to determine a second reference sample at a fractional pixel location in a second pixel-search-area around a second location indicated by the intermediate candidate inter-frame prediction mode.

8. The computing device of claim 1, wherein the first initial candidate inter-frame prediction mode, the intermediate candidate inter-frame prediction mode, and the final candidate inter-frame prediction mode each comprises a motion vector and a reference index to indicate location of a reference sample.

9. The computing device of claim 1, wherein the computing device comprises a portable phone, a media player, a personal data organizer, a handheld game platform, a tablet device, a computer, or any combination thereof.

10. A tangible, non-transitory, computer-readable medium configured to store instructions executable by one or more processors in a video encoding pipeline, wherein the instructions comprise instructions to:

determine, using the one or more processors, first operational parameters of a video encoding pipeline used to encode a first coding unit in a first image frame, wherein the first operational parameters comprise resolution of the first image frame, refresh rate used to display the first image frame, desired power consumption to encode the first coding unit, or any combination thereof;

determine, using the one or more processors, a first operation mode of a motion estimation block in the video encoding pipeline based at least in part on the first operational parameters, wherein the first operation mode indicates a first operating duration provided the motion estimation block to perform motion estimation searches on the first coding unit;

determine, using the one or more processors, a first setup configuration based at least in part on the first operation mode, wherein the first setup configuration comprises first setup parameters used to control operation of the motion estimation block; and

instruct, using the one or more processors, the video encoding pipeline to encode the first coding unit based at least in part on a first setup configuration.

11. The computer-readable medium of claim 10, comprising instructions to:

determine, using the one or more processors, second operational parameters of the video encoding pipeline used to encode a second coding unit in a second image frame, wherein the second operational parameters differ from the first operational parameters;

determine, using the one or more processors, a second operation mode of the motion estimation block based at least in part on the second operational parameters, wherein the second operation mode is shorter than the first operating duration and indicates a second operating duration provided the motion estimation block to perform motion estimation searches on the second coding unit;

determine, using the one or more processors, a second setup configuration based at least in part on the second operation mode, wherein the second setup configuration comprises second setup parameters used to control operation of the motion estimation block; and

instruct, using the one or more processors, the video encoding pipeline to encode the second coding unit based at least in part on a second setup configuration.

12. The computer-readable medium of claim 11, wherein the first operation mode comprises a normal mode and the second operation mode comprises a medium mode or a turbo mode, wherein:

the normal mode comprises a first operating duration up to ninety-two 16×

16 full-pel motion estimation searches per coding unit;

the medium mode comprises a second operating duration up to fifty-two 16×

16 full-pel motion estimation searches per coding unit; and

the turbo mode comprises a third operating duration up to thirty-six 16×

16 full-pel motion estimation searches per coding unit.

13. The computer-readable medium of claim 11, wherein:

the instruction to instruct the video encoding pipeline to encode the first coding unit comprise instructions to instruct the motion estimation block to select a first plurality of initial candidate inter-frame prediction modes based at least in part on the first setup configuration; and

the instructions to instruct the video encoding pipeline to encode the first coding unit comprising instructions to instruct the motion estimation block to select a second plurality of initial candidate inter-frame prediction modes based at least in part on the second setup configuration;

wherein the first plurality of initial candidate inter-frame prediction modes and the second plurality of initial candidate inter-frame prediction modes include different numbers of one or more types of initial candidate inter-frame prediction modes.

14. The computer-readable medium of claim 10, wherein the instruction to instruct the video encoding pipeline to encode the first coding unit comprise instructions to:

determine a first pixel-search-area around a first location indicated by an automatic inter-frame prediction mode;

determine a first initial candidate inter-frame prediction mode based at least in part on the first setup configuration;

determine a second pixel-search-area around second location indicated by the first initial candidate inter-frame prediction mode;

determine amount of overlap between the first pixel-search-area and the second pixel-search-area;

instruct the motion estimation block to perform a first full-pel motion estimation search based on the first initial candidate inter-frame prediction mode by searching the second pixel-search-area when the amount of overlap is less than a lower threshold;

instruct the motion estimation block to replace the first initial candidate inter-frame prediction mode with a second initial candidate inter-frame prediction and to perform a second full-pel motion estimation search based on the second plurality initial candidate inter-frame prediction mode when the amount of overlap is greater than an upper threshold; and

instruct the motion estimation block to determine an adjusted version of the first initial candidate inter-frame prediction mode that moves the second location and to perform a third full-pel motion estimation search based on the adjusted version of the first initial candidate inter-frame prediction mode when the amount of overlap is between the lower threshold and the upper threshold.

15. The computer-readable medium of claim 10, wherein the instruction to instruct the video encoding pipeline to encode the first coding unit comprise instructions to:

instruct the motion estimation block to select an initial candidate inter-frame prediction mode based at least in part on the first setup configuration;

instruct the motion estimation block to perform a full-pel motion estimation search based at least in part on the initial candidate inter-frame prediction mode to determine an intermediate candidate inter-frame prediction mode that indicates location of a first reference sample at an integer pixel location; and

instruct the motion estimation block to perform a sub-pel motion estimation search based at least in part on the intermediate candidate inter-frame prediction mode to determine a final candidate that indicates location of a second reference sample at a fractional pixel location.

16. The computer-readable medium of claim 10, wherein the first setup parameters comprise number of each type of initial candidate inter-frame prediction modes to be evaluated by a full-pel motion estimation block, location of initial candidate inter-frame prediction modes to be evaluated by the full-pel motion estimation block, pixel-search-area around initial candidate inter-frame prediction modes in the full-pel motion estimation block, pixel-search-area around intermediate candidate inter-frame prediction modes in a sub-pel motion estimation block, or any combination thereof.

17. A method comprising:

determining, using processing circuitry of a video encoding pipeline, a first pixel search based at least in part on a first location indicated by a first candidate inter-frame prediction mode, wherein a full-pel motion estimation block searches a first pixel-search-area to determine a first reference sample;

determining, using the processing circuitry, a second candidate inter-frame prediction mode based at least in part on a setup configuration;

determining, using the processing circuitry, a second pixel-search-area based at least in part on a second location indicated by the second candidate inter-frame prediction mode;

adjusting, using the processing circuitry, the second candidate inter-frame prediction mode to move the second location when overlap between the first pixel-search-area and the second pixel-search-area is between an upper threshold and lower threshold; and

instructing, using the processing circuitry, the video encoding pipeline to encode source image data to generate encoded image data based at least in part on cost associated with the first candidate inter-frame prediction mode, the second candidate inter-frame prediction mode, or both, wherein a size of the encoded image data is smaller than size of the source image data.

18. The method of claim 17, comprising:

determining, using the processing circuitry, operational parameters of the video encoding pipeline used to encode the source image data;

dynamically adjusting, using the processing circuitry, the setup configuration, wherein the setup configuration comprises setup parameters used to control selection of initial candidate inter-frame prediction modes used in the full-pel motion estimation block, search area around initial candidate inter-frame prediction modes searched by the full-pel motion estimation block, or both.

19. The method of claim 17, comprising:

replacing, using the processing circuitry, the second candidate inter-frame prediction mode with a third candidate inter-frame prediction mode when the overlap is greater than the upper threshold; and

instructing, using the processing circuitry, the full-pel motion estimation block to search the second pixel-search-area to determine a second reference sample.

20. The method of claim 17, wherein:

the first candidate inter-frame prediction mode comprises an automatic inter-frame prediction mode, wherein the full-pel motion estimation block searches the first pixel-search-area each time a coding unit of the source image data is encoded in the video encoding pipeline; and

the second candidate inter-frame prediction mode comprises a low resolution inter-frame prediction mode, a predictor inter-frame prediction modes, a controller inter-frame prediction mode, or any combination thereof.