Configurable motion estimation search systems and methods

• US 10,225,572 B2
• Filed: 09/30/2015
• Issued: 03/05/2019
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Chou, Jim C., Rygh, Mark P., Cote, Guy
• Primary Examiner(s): Hallenbeck-Huber, Jeremiah C
• Assistant Examiner(s): Tchoussou, Boubacar Abdou
• Application Number:
  US14/871,848
• Publication Number:
  US 20170094293A1
• Time in Patent Office: 1,252 Days
• CPC Class Codes: H04N 19/109 among a plurality of temporal predictive coding modes
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/157 Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/182 the unit being a pixel
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/439 using cascaded computational arrangements for performing a single operation, e.g. filtering

Abstract

System and method for improving operational efficiency of a video encoding pipeline used to encode image data. The video encoding pipeline includes a motion estimation setup block, which dynamically adjusts a setup configuration of the motion estimation block based at least in part on operational parameters of the video encoding pipeline and select an initial candidate inter-frame prediction mode based at least on the setup configuration, a full-pel motion estimation block, which determines an intermediate candidate inter-frame prediction mode by performing a motion estimation search based on the initial candidate inter-frame prediction mode, a sub-pel motion estimation block, which determines a final candidate inter-frame prediction by performing a motion estimation search based on the intermediate candidate inter-frame prediction mode, and a mode decision block, which determines a rate-distortion cost associated with the final candidate inter-frame prediction mode and determines a prediction mode used to prediction encoding the image data.

Claims

1. A computing device comprising a video encoding pipeline configured to receive source image data corresponding with a first image and to output encoded image data corresponding with the source image data, wherein the video encoding pipeline comprises:

motion estimation setup circuitry configured to;

determine a first pixel search area in a second image based at least in part on a first location indicated by a first initial candidate inter-frame prediction mode;

determine a second pixel search area in the second image based at least in part on a second location indicated by a second initial candidate inter-frame prediction mode different from the first initial candidate inter-frame prediction mode; and

adjust the second initial candidate inter-frame prediction mode to move the second pixel search area in the second image when a first amount of overlap between the first pixel search area and the second pixel search area is between an upper threshold and a lower threshold; and

motion estimation circuitry coupled to the motion estimation setup circuitry, wherein the motion estimation circuitry is configured to perform a first motion estimation search in the first pixel search area and the second pixel search area when the first amount of overlap between the first pixel search area and the second pixel search area is not greater than the upper threshold to facilitate determining a final candidate inter-frame prediction mode; and

mode decision circuitry coupled to the motion estimation circuitry, wherein the mode decision circuitry is configured to;

determine a rate-distortion cost associated with the final candidate inter-frame prediction mode based at least in part on a comparison of the source image data and a prediction sample resulting from application of the final candidate inter-frame prediction mode to other image data; and

determine a prediction mode used to prediction encode the source image data based at least in part on the rate-distortion cost associated with the final candidate inter-frame prediction mode, wherein the encoded image data is generated to indicate the prediction mode.

2. The computing device of claim 1, wherein the motion estimation circuitry is configured to:

perform the first motion estimation search in the first pixel search area and the second pixel search area at a full-pel resolution when the first amount of overlap between the first pixel search area and the second pixel search area is not greater than the upper threshold to facilitate determining an intermediate candidate inter-frame prediction mode;

determine a third pixel search area in the second image based at least in part on a third location indicated by the intermediate candidate inter-frame prediction mode; and

perform a second motion estimation search in the third pixel search area at a sub-pel resolution to facilitate determining the final candidate inter-frame prediction mode.

3. The computing device of claim 1, wherein the video encoding pipeline comprises:

inter-frame prediction circuitry configured to determine a first luma prediction sample based at least in part on the final candidate inter-frame prediction mode;

intra-frame prediction circuitry configured to determine a candidate intra-frame prediction mode and to determine a second luma prediction sample based at least in part on the candidate intra-frame prediction mode;

chroma reconstruction circuitry configured to determine chroma components of reconstructed image data based at least in part on the prediction mode determined by the mode decision circuitry;

luma reconstruction circuitry configured to determine a luma component of the reconstructed image data based at least in part on the prediction mode determined by the mode decision circuitry;

back-end-filter circuitry configured to filter the reconstructed image data; and

syntax element binarization circuitry configured to generate binarized syntax elements used to indicate the prediction mode determined by the mode decision circuitry; and

a transcode pipeline comprising circuitry configured to generate the encoded image data by entropy encoding the binarized syntax elements.

4. The computing device of claim 1, wherein the motion estimation setup circuitry is configured to:

determine a setup configuration comprising a first setup parameter that indicates number of initial candidate inter-frame prediction modes of a first type to be evaluated, a second setup parameter that indicates number of initial candidate inter-frame prediction modes of a second type different from the first type to be evaluated, and a third setup parameter that indicates search area size resulting from each initial candidate inter-frame prediction mode;

select the second initial candidate inter-frame prediction mode from a plurality of initial candidate inter-frame prediction modes based at least in part on the first setup parameter that indicates number of initial candidate inter-frame prediction modes of the first type to be evaluated; and

determine the second pixel search area in the second image based at least in part on the third setup parameter that indicates search area size resulting from each initial candidate inter-frame prediction mode selected.

5. The computing device of claim 4, comprising a controller coupled to the motion estimation setup circuitry, wherein the controller is configured to:

determine operational parameters of the video encoding pipeline, wherein the operational parameters comprise image frame resolution, display refresh rate, remaining battery power, or any combination thereof; and

dynamically adjust the setup configuration in response to changes in the operational parameters of the video encoding pipeline.

6. The computing device of claim 4, comprising a controller coupled to the motion estimation setup circuitry, wherein the controller is configured to:

determine an operation mode of the video encoding pipeline, wherein the operation mode is indicative of operating duration the video encoding pipeline is provided to encode the source image data; and

dynamically adjust the setup configuration in response to changes in the operation mode of the video encoding pipeline.

7. The computing device of claim 1, wherein the motion estimation circuitry is configured to:

replace the second initial candidate inter-frame prediction mode with a third initial candidate inter-frame prediction mode when the first amount of overlap between the first pixel search area and the second pixel search area is greater than the upper threshold;

determine a third pixel search area in the second image based at least in part on a third location indicated by the third initial candidate inter-frame prediction mode; and

perform the first motion estimation search in the first pixel search area and the third pixel search area when a second amount of overlap between the first pixel search area and the third pixel search area is not greater than the upper threshold.

8. The computing device of claim 1, wherein the computing device comprises a portable phone, a media player, a personal data organizer, a handheld game platform, a tablet device, a computer, or any combination thereof.

20. The computing device of claim 1, wherein, when the first amount of overlap between the first pixel search area and the second pixel search area is between the upper threshold and the lower threshold, the motion estimation setup circuitry is configured to adjust the second initial candidate inter-frame prediction mode to move the second pixel search area to reduce the first amount of overlap between the first pixel search area and the second pixel search area below the lower threshold.

9. A tangible, non-transitory, computer-readable medium that store instructions executable by one or more processors in a video encoding pipeline, wherein the instructions comprise instructions to:

instruct, using the one or more processors, the video encoding pipeline to retrieve source image data corresponding with a first image;

instruct, using the one or more processors, the video encoding pipeline to determine a first pixel search area in a second image based at least in part on a first location identified by a first candidate inter prediction mode;

instruct, using the one or more processors, the video encoding pipeline to determine a second pixel search area in the second image based at least in part on a second location identified by a second candidate inter prediction mode different from the first candidate inter prediction mode;

instruct, using the one or more processors, the video encoding pipeline to adjust the second candidate inter prediction mode to move the second pixel search area in the second image when a first amount of overlap between the first pixel search area and the second pixel search area is between an upper threshold and a lower threshold;

instruct, using the one or more processors, the video encoding pipeline to perform a motion estimation search in the first pixel search area and the second pixel search area when the first amount of overlap between the first pixel search area and the second pixel search area is not greater than the upper threshold to facilitate determining a prediction mode to be applied to other image data to determine a prediction sample corresponding with the source image data; and

instruct, using the one or more processors, the video encoding pipeline to output encoded image data generated to indicate the prediction mode and a prediction residual resulting from comparison of the prediction sample and the source image data.

10. The computer-readable medium of claim 9, comprising instructions to:

determine, using the one or more processors, a setup configuration comprising a first setup parameter that indicates number of candidate inter prediction modes of a first type to be evaluated, a second setup parameter that indicates number of candidate inter prediction modes of a second type different from the first type to be evaluated, and a third setup parameter that indicates search area size resulting from each candidate inter prediction mode; and

instruct, using the one or more processors, the video encoding pipeline to implement the setup configuration by;

selecting the second candidate inter prediction mode from a plurality of candidate inter prediction modes based at least in part on the first setup parameter that indicates number of initial candidate inter prediction modes of the first type to be evaluated; and

determining the second pixel search area in the second image based at least in part on the third setup parameter that indicates search area size resulting from each initial candidate inter prediction mode.

11. The computer-readable medium of claim 10, wherein the instructions to determine the setup configuration comprises instructions to:

determine the first setup parameter of the setup configuration to indicate number of 16×

16 initial candidate inter prediction modes to be evaluated; and

determine the second setup parameter of the setup configuration to indicate number of 32×

32 initial candidate inter prediction modes to be evaluated.

12. The computer-readable medium of claim 10, comprising instructions to determine, using the one or more processors, an operation mode of the video encoding pipeline based at least in part on resolution of the first image, refresh rate at which the first image is to be displayed, power allocated to encoding the source image data, or any combination thereof;

wherein the instructions to determine the setup configuration comprise instructions to dynamically adjust the setup configuration based at least in part on the operation mode of video encoding pipeline.

13. The computer-readable medium of claim 9, comprising instructions to:

instruct, using the one or more processors, the video encoding pipeline to replace the second candidate inter prediction mode with a third candidate inter prediction mode when the first amount of overlap between the first pixel search area and the second pixel search area is greater than the upper threshold;

instruct, using the one or more processors, the video encoding pipeline to determine a third pixel search area in the second image based at least in part on a third location indicated by the third candidate inter prediction mode; and

instruct, using the one or more processors, the video encoding pipeline to perform the motion estimation search in the first pixel search area and the third pixel search area when a second amount of overlap between the first pixel search area and the third pixel search area is not greater than the upper threshold to facilitate determining the prediction mode to be used to encode the source image data.

14. A method comprising:

receiving, using a video encoding pipeline, source image data corresponding with a first image;

determining, using the video encoding pipeline, a first pixel search area in a second image based at least in part on a first location indicated by a first candidate inter-frame prediction mode;

determining, using the video encoding pipeline, a second pixel search area in the second image based at least in part on a second location indicated by a second candidate inter-frame prediction mode different from the first candidate inter-frame prediction mode;

adjusting, using the video encoding, the second candidate inter-frame prediction mode to move the second pixel search area in the second image when a first amount of overlap between the first pixel search area and the second pixel search area is between an upper threshold and a lower threshold; and

performing, using the video encoding pipeline, a motion estimation search in the first pixel search area and the second pixel search area when the first amount of overlap between the first pixel search area and the second pixel search area is not greater than the upper threshold to facilitate determining a prediction mode to be applied to other image data to determine a prediction sample corresponding with the source image data; and

outputting, using the video encoding pipeline, encoded image data that indicates the prediction mode and a prediction residual resulting from comparison of the prediction sample and the source image data.

15. The method of claim 14, comprising:

determining, using the video encoding pipeline, operational parameters of the video encoding pipeline used to encode the source image data, wherein the operational parameters of the video encoding pipeline comprise resolution of the first image, refresh rate to be used to display the first image, power consumption to be used to encode the source image data, or any combination thereof;

determining, using the video encoding pipeline a setup configuration based at least in part on the operational parameters of the video encoding pipeline, wherein the setup configuration comprises setup parameters that indicate number of candidate inter-frame prediction modes of a first type to be evaluated and number of candidate inter-frame prediction modes of a second type different from the first type to be evaluated; and

selecting, using the video encoding pipeline, the second candidate inter-frame prediction mode based at least in part on the setup configuration.

16. The method of claim 14, comprising:

replacing, using the video encoding pipeline, the second candidate inter-frame prediction mode with a third candidate inter-frame prediction mode when the first amount of overlap between the first pixel search area and the second pixel search area is greater than the upper threshold;

determining, using the video encoding pipeline, a third pixel search area in the second image based at least in part on a third location indicated by the third candidate inter-frame prediction mode; and

performing, using video encoding pipeline, the motion estimation search in the first pixel search area and the third pixel search area when a second amount of overlap between the first pixel search area and the third pixel search area is not greater than the upper threshold to facilitate determining the prediction mode to be applied to other image data to determine the prediction sample corresponding with the source image data.

17. The method of claim 16, comprising adjusting, using the video encoding pipeline, the third candidate inter-frame prediction mode to move the third pixel search area in the second image when the second amount of overlap between the first pixel search area and the third pixel search area is between the upper threshold and the lower threshold.

18. The method of claim 14, comprising:

determining, using the video encoding pipeline, a third pixel search area in the second image based at least in part on a third location indicated by a third candidate inter-frame prediction mode;

adjusting, using the video encoding pipeline, the third candidate inter-frame prediction mode to move the third pixel search area in the second image when a second amount of overlap between the first pixel search area and the third pixel search area, a third amount of overlap between the second pixel search area and the third pixel search area, or both is between the upper threshold and the lower threshold; and

performing, using the video encoding pipeline, the motion estimation search in the first pixel search area, the second pixel search area, and the third pixel search area when the first amount of overlap between the first pixel search area and the second pixel search area is not greater than the upper threshold, the second amount of overlap between the first pixel search area and the third pixel search area is not greater than the upper threshold, and the third amount of overlap between the second pixel search area and the third pixel search area is not greater than the upper threshold to facilitate determining the prediction mode to be applied to other image data to determine the prediction sample corresponding with the source image data.

19. The method of claim 14, wherein adjusting the second candidate inter-frame prediction mode comprises adjusting the second candidate inter-frame prediction mode to move the second pixel search area in the second image such that the first amount of overlap between the first pixel search area and the second pixel search area is not greater than the lower threshold.