MEMORY-TO-MEMORY LOW RESOLUTION MOTION ESTIMATION SYSTEMS AND METHODS

• US 20170094311A1
• Filed: 09/30/2015
• Published: 03/30/2017
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Chou, Jim C., Rygh, Mark P., Ct, Guy
• Granted Patent: US 10,187,655 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 19/109 among a plurality of temporal predictive coding modes
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/56 Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

System and method for improving operational efficiency of a video encoding pipeline used to encode image data. In embodiments, the video encoding pipeline includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data and determines a low resolution inter-frame prediction mode by performing a motion estimation search using the downscaled image data and previously downscaled image data. The video encoding pipeline also includes a main pipeline in parallel with the low resolution pipeline that includes a motion estimation block, which determines a candidate inter-frame prediction mode based at least in part on the low resolution inter-frame prediction mode, and a mode decision block, which determines a first rate-distortion cost associated with the candidate inter-frame prediction mode and determines prediction mode used to prediction encode the image data based at least in part on the first rate-distortion cost.

Claims

1. A video encoding pipeline configured to encode image data, comprising:

a low resolution pipeline configured to receive the image data, wherein the low resolution pipeline comprises a low resolution motion estimation block configured to;

generate downscaled image data by reducing resolution of the image data; and

determine a low resolution inter-frame prediction mode by performing a motion estimation search using the downscaled image data and previously downscaled image data; and

a main pipeline in parallel with the low resolution pipeline, wherein the main pipeline is configured to receive the image data and to determine encoding operational parameters used to encode the image data, wherein the main pipeline comprises;

a motion estimation block configured to determine a candidate inter-frame prediction mode based at least in part on the low resolution inter-frame prediction mode; and

a mode decision block configured to determine a first rate-distortion cost associated with the candidate inter-frame prediction mode and to determine a prediction mode used to prediction encode the image data based at least in part on the first rate-distortion cost.

2. The video encoding pipeline of claim 1, wherein the low resolution motion estimation block is configured to determine statistics based at least in part on luma of the downscaled image data;

wherein;

the video encoding pipeline is configured to determine when a scene change is expected to occur based at least in part on the statistics; and

the mode decision block is configured to determine the prediction mode based at least in part on when the scene change is expected to occur.

3. The video encoding pipeline of claim 1, wherein the main pipeline comprises:

an inter-frame prediction block configured to determine a first luma prediction sample based at least in part on the candidate inter-frame prediction mode;

an intra-frame prediction block configured to determine a candidate intra-frame prediction mode and to determine a second luma prediction sample based at least in part on the candidate intra-frame prediction mode;

a chroma reconstruction block configured to determine chroma components of reconstructed image data based at least in part on the prediction mode determined by the mode decision block;

a luma reconstruction block configured to determine a luma component of the reconstructed image data based at least in part on the prediction mode determined by the mode decision block;

a back-end-filter block configured to filter the reconstructed image data; and

a syntax element binarization block configured to binarize syntax elements used to indicate the prediction mode determined by the mode decision block.

4. The video encoding pipeline of claim 3, wherein the mode decision block is configured to:

determine the first rate-distortion cost based at least in part on the first luma prediction sample;

determine a second rate-distortion cost associated with the candidate intra-frame prediction mode based at least in part on the second luma prediction sample;

determine a third rate-distortion cost associated with a skip mode; and

determine the prediction mode by comparing the first rate-distortion cost, the second rate-distortion cost, and the third rate-distortion cost.

5. The video encoding pipeline of claim 1, comprising a transcode pipeline in parallel with the low resolution pipeline and the main pipeline, wherein the transcode pipeline is configured to entropy encode binarized syntax elements used to indicate the prediction mode.

6. The video encoding pipeline of claim 1, wherein the low resolution inter-frame prediction mode comprises a motion vector and a reference index configured to indicate where a reference sample in full resolution is expected to occur.

7. The video encoding pipeline of claim 1, wherein:

the low resolution pipeline is configured to receive the image data using direct memory access; and

the main pipeline is configured to receive the image data and the low resolution inter-frame prediction mode using direct memory access.

8. The video encoding pipeline of claim 1, wherein the downscaled image data comprises a resolution that is one-sixteenth a resolution of the image data.

9. A tangible, non-transitory, computer-readable medium configured to store instructions executable by one or more processors in a video encoding pipeline, wherein the instructions comprise instructions to:

downscale, using the one or more processors, image data to be encoded by the video encoding pipeline to generate downscaled image data;

determine, using the one or more processors, a low resolution inter-frame prediction mode by performing a first motion estimation search on previously downscaled image data based on the downscaled image data;

determine, using the one or more processors, a candidate inter-frame prediction mode by performing a second motion estimation search on reconstructed image data based on the low resolution inter-frame prediction mode;

determine, using the one or more processors, encoding operational parameters used to encode the image data based at least in part on a rate-distortion cost associated with the candidate inter-frame prediction mode; and

encode, using the one or more processors, the image data using the encoding operational parameters to generate encoded image data, wherein a size of the encoded image data is smaller than size of the image data.

10. The computer-readable medium of claim 9, comprising instructions to determine, using the one or more processors, statistics based at least in part on luma information of the image data, wherein the statistics comprise an indication of when a scene change is expected to occur;

wherein the instruction to determine the encoding operational parameters comprise instructions to determine a prediction technique used to encode the image based at least in part on when the scene change is expected to occur.

11. The computer-readable medium of claim 9, wherein the instructions to downscale the image data comprise instructions to reduce resolution of the image by one-sixteenth.

12. The computer-readable medium of claim 9, wherein the instructions to determine the low resolution inter-frame prediction mode comprise instructions to:

perform the first motion estimation search to determine a downscaled reference sample in the previously downscaled image data corresponding with the downscaled image data;

determine a motion vector that indicates position of a reference sample corresponding with the downscaled reference sample in a reference image frame relative to position of the image data in a current image frame; and

determine a reference index that indicates display order of the reference image frame relative to the current image frame.

13. The computer-readable medium of claim 9, wherein the instructions to determine a candidate inter-frame prediction mode comprise instructions to:

perform the second motion estimation search to determine a reference sample in the reconstructed image data corresponding with the image data, wherein the low resolution inter-frame prediction mode provides an indication of where the reference sample is expected to be located;

determine a motion vector that indicates spatial position of the reference sample in a reference image frame relative to position of the image data in a current image frame; and

determine a reference index that indicates display order of the reference image frame relative to the current image frame.

14. The computer-readable medium of claim 9, wherein the encoding operational parameters comprise prediction techniques used to encode a coding block, a number of prediction units in the coding block, a size of the prediction units, a prediction mode used to encode each of the prediction units, a number of transform units in the coding block, a size of the transform units, whether to split the coding unit into smaller coding units, or any combination thereof.

15. A video encoding pipeline comprising:

memory configured to store source image data corresponding with a first image frame to be encoded by the video encoding pipeline, wherein the source image data is divided into a plurality of coding units, wherein each of the plurality coding units is configured to be encoded using a same prediction technique and each of the plurality of coding units comprises one or more prediction units each configured to be encoded using a different prediction mode of the same prediction technique;

a low resolution motion estimation circuitry configured to;

retrieve a first coding unit of the plurality of coding units from the memory using direct memory access;

downscale the first coding unit to generate a first downscaled coding unit, wherein the first downscaled coding unit comprises a first downscaled prediction unit corresponding with a first prediction unit in the first coding unit;

determine a first low resolution inter-frame prediction mode by performing a first motion estimation search on second downscaled image data corresponding with a second image frame to determine a first downscaled reference sample corresponding with the first downscaled prediction unit;

a main pipeline configured to retrieve the first coding unit and the first low resolution inter-frame prediction mode from the memory using direct memory access and to determine a first prediction mode used to encode the first prediction unit based at least in part on the first low resolution inter-frame prediction mode.

16. The video encoding pipeline of claim 15, wherein:

the first downscaled coding unit comprises a second downscaled prediction unit corresponding with a second prediction unit in the first coding unit;

the low resolution motion estimation circuitry is configured to determine a second low resolution inter-frame prediction mode by performing a second motion estimation search on the second downscaled image data to determine a second downscaled reference sample corresponding with the second downscaled prediction unit; and

the main pipeline is configured to retrieve the second low resolution inter-frame prediction mode from the memory using direct memory access and to determine a second prediction mode used to encode the second prediction unit based at least in part on the first low resolution inter-frame prediction mode and the second low resolution inter-frame prediction mode.

17. The video encoding pipeline of claim 15, wherein the low resolution motion estimation circuitry is configured to:

retrieve a second coding unit of the plurality of coding units from the memory using direct memory access;

downscale the second coding unit to generate a second downscaled coding unit, wherein the second downscaled coding unit comprises a second downscaled prediction unit corresponding with a second prediction unit in the second coding unit;

determine a second low resolution inter-frame prediction mode by performing a second motion estimation search on the second downscaled image data to determine a second downscaled reference sample corresponding with the second downscaled prediction unit;

wherein the main pipeline configured to retrieve the second coding unit and the second low resolution inter-frame prediction mode from the memory using direct memory access and to determine a second prediction mode used to encode the second prediction unit based at least in part on the second low resolution inter-frame prediction mode.

18. The video encoding pipeline of claim 15, wherein the first coding unit is a 64×

64 sample.

19. The video encoding pipeline of claim 15, wherein the second image frame is configured to be display directly before or directly after the first image frame.

20. A computing device comprising:

an image data source configured to generate first image data corresponding with a first image frame;

a low resolution motion estimation pipeline comprising a low resolution motion estimation block configured to process the first image data to determine a low resolution inter-frame prediction mode and statistics based at least in part on luma of the first image data; and

a main video encoding pipeline configured to generate first encoded image data by encoding the first image data, wherein the video encoding pipeline comprises;

a motion estimation block configured to process the first image data after the low resolution motion estimation block to determine a candidate inter-frame prediction mode based at least in part on the low resolution inter-frame prediction mode; and

a mode decision block configured to determine encoding operational parameters used to encode the first image data based at least in part on the candidate inter-frame prediction mode and the statistics determined by the low resolution motion estimation block; and

an output configured to receive the first encoded image data and to display the first image frame or to transmit the first image frame in real-time or near real-time from when generated.

21. The computing device of claim 20, wherein the statistics comprise:

a luma histogram configured to indicate number of pixels in the first image data at each luma value; and

a zero vector sum of absolute difference between a downscaled image data corresponding with the first image data and a downscaled reference sample at a same position in a second image frame.

22. The computing device of claim 20, wherein:

the motion estimation block is configured to process second image data corresponding with a second image frame while the low resolution motion estimation block processes the first image data, wherein the second image frame is configured to be displayed before the first image frame; and

the main video encoding pipeline is configured to determine where a scene change is expected to occur to facilitate frame-rate conversion.

23. The computing device of claim 20, wherein the low resolution motion pipeline is configured to process image data in a downscaled resolution one or more image frames before the main video encoding pipeline processes the image data in a full resolution.

24. The computing device of claim 20, wherein:

the image data source comprises an image sensor configured to generate a digital representation of proximate physical features as the first image data; and

the output comprises a memory configured to store the first encoded image data, a network interface configured to transmit the first encoded image data to another computing device or a network, an electronic display configured to display the first image frame based on the first encoded image data, or any combination thereof.

25. The computing display of claim 20, wherein the computing device comprises a portable phone, a media player, a personal data organizer, a handheld game platform, a tablet device, a computer, or any combination thereof.

26. A method comprising:

instructing, using a controller in a video encoding pipeline configured to encode image data, a low resolution motion estimation block to process the image data to determine statistics based at least in part on luma of the image data, wherein the statistics comprise a first luma histogram statistic, a zero vector sum of absolute difference statistic, or both;

determining, using the controller, when a scene change is expected to occur based at least in part on the statistics determined by the low resolution motion estimation block;

instructing, using the controller, a motion estimation block to stop determining candidate inter-frame prediction modes when the scene change is expected;

instructing, using the controller, a mode decision block to determine rate-distortion costs associated with a candidate intra-frame prediction mode and a skip mode and to determine a prediction mode based at least in part on the rate-distortion costs; and

instructing, using the controller, the video encoding pipeline to encode the image data using the prediction mode determined by the mode decision block.

27. The method of claim 26, comprising:

instructing, using the controller, the low resolution motion estimation block to process the image data to determine a low resolution inter-frame prediction mode by performing a first motion estimation search on previously downscaled image data based on a downscaled version of the image data and;

instructing, using the controller, the motion estimation block to determine a candidate inter-frame prediction mode by performing a second motion estimation search on reconstructed image data based on the low resolution inter-frame prediction mode; and

instructing, using the controller, the mode decision block to determine rate-distortion cost associated with the candidate inter-frame prediction mode.

28. The method of claim 26, wherein determining when the scene change is expected to occur comprises:

determining that the scene change is expected to occur when difference between the first luma histogram and a second luma histogram corresponding with co-located image data of a different image frame vary by more than a histogram threshold;

determining that the scene change is expected to occur when the zero vector sum of absolute difference is above a threshold;

or both.

29. The method of claim 26, wherein:

the first luma histogram indicates number of pixels in a downscaled version of the image data at each luma value; and

the zero vector sum of absolute difference indicates difference between the downscaled version of the image data and a co-located downscaled reference sample in a different image frame.