Memory-to-memory low resolution motion estimation systems and methods

• US 10,187,655 B2
• Filed: 09/30/2015
• Issued: 01/22/2019
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Chou, Jim C., Rygh, Mark P., Cote, Guy
• Primary Examiner(s): Messmore, Jonathan R
• Application Number:
  US14/871,827
• Publication Number:
  US 20170094311A1
• Time in Patent Office: 1,210 Days
• CPC Class Codes: H04N 19/109 among a plurality of temporal predictive coding modes
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/56 Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

System and method for improving operational efficiency of a video encoding pipeline used to encode image data. In embodiments, the video encoding pipeline includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data and determines a low resolution inter-frame prediction mode by performing a motion estimation search using the downscaled image data and previously downscaled image data. The video encoding pipeline also includes a main pipeline in parallel with the low resolution pipeline that includes a motion estimation block, which determines a candidate inter-frame prediction mode based at least in part on the low resolution inter-frame prediction mode, and a mode decision block, which determines a first rate-distortion cost associated with the candidate inter-frame prediction mode and determines prediction mode used to prediction encode the image data based at least in part on the first rate-distortion cost.

Claims

1. A video encoding pipeline comprising:

a low resolution pipeline coupled to memory that stores source image data to be encoded by the video encoding system, wherein the low resolution pipeline comprises a low resolution motion estimation block configured to;

determine first downscaled source image data generated by reducing a resolution of first source image data corresponding with a first image frame;

generate second downscaled source image data by reducing a resolution of second source image data corresponding with a second image frame when the second source image data is retrieved from the memory;

perform a first motion estimation search in the first downscaled source image data based on the second downscaled source image data to determine a first downscaled reference sample; and

determine a low resolution inter-frame prediction mode that indicates location of the second source image data relative to a first reference sample corresponding with the downscaled reference sample; and

a main pipeline coupled to the memory that stores the source image data to be encoded by the video encoding system, wherein the main pipeline comprises;

a reconstruction block configured to determine first reconstructed image data by decoding encoded image data corresponding with the first source image data;

a motion estimation block configured to determine a candidate inter-frame prediction mode by performing a second motion estimation search in the first reconstructed image data based at least in part on the low resolution inter-frame prediction mode; and

a mode decision block configured to determine encoding parameters to be used to encode the second source image data based at least in part on a first rate-distortion cost associated with the candidate inter-frame prediction mode.

2. The video encoding pipeline system of claim 1, wherein:

the low resolution motion estimation block is configured to determine statistics based at least in part on luma of the second downscaled image data;

the main pipeline is configured to determine when a scene change is expected to occur based at least in part on the statistics; and

the mode decision block is configured to determine the encoding parameters based at least in part on when the scene change is expected to occur.

3. The video encoding pipeline of claim 1, wherein the main pipeline comprises:

an inter-frame prediction block configured to determine a first luma prediction sample based at least in part on the candidate inter-frame prediction mode;

an intra-frame prediction block configured to;

determine a candidate intra-frame prediction mode; and

determine a second luma prediction sample based at least in part on the candidate intra-frame prediction mode;

a back-end-filter block configured to filter second reconstructed image data, wherein the reconstruction block is configured to determine the second reconstructed image data by decoding second encoded image data generated by applying the encoding parameters determined by the mode decision block to the second source image data; and

a syntax element binarization block configured to binarize syntax elements used to indicate the encoding parameters applied by the reconstruction block to determine the second reconstructed image data, filter parameters applied by the back-end-filter block to the second reconstructed image data, or both.

4. The video encoding pipeline of claim 3, wherein the mode decision block is configured to:

determine the first rate-distortion cost based at least in part on the first luma prediction sample;

determine a second rate-distortion cost associated with the candidate intra-frame prediction mode based at least in part on the second luma prediction sample;

determine a third rate-distortion cost associated with a skip mode; and

determine the encoding parameters based at least in part on a comparison between the first rate-distortion cost, the second rate-distortion cost, and the third rate-distortion cost.

5. The video encoding pipeline of claim 1, comprising a transcode pipeline coupled to the memory, wherein:

the main pipeline is configured to;

generate a bin stream by binarizing syntax elements used to indicate the encoding parameters determined by the mode decision block;

output to the bin stream to the memory via direct memory access; and

the transcode pipeline is configured to;

retrieve the bin stream from the memory via direct memory access; and

entropy encode the bin stream to generate a bit stream that indicates one or more of the syntax elements using fractional bits.

6. The video encoding pipeline of claim 1, wherein, to determine the low resolution inter-frame prediction mode, the low resolution motion estimation block is configured:

determine a reference index that indicates temporal position of the first image frame corresponding with the first source image data relative to the second image frame corresponding with the second source image data; and

determine a motion vector that indicates spatial position of the full resolution reference sample in the first image frame relative to the second source image data in the second image frame.

7. The video encoding pipeline of claim 1, wherein:

the low resolution pipeline is configured to retrieve the second source image data from the memory using direct memory access; and

the main pipeline is configured to retrieve the second image data and the low resolution inter-frame prediction mode from the memory using direct memory access.

8. The video encoding pipeline of claim 1, wherein the second downscaled image data comprises a resolution that is one-sixteenth a resolution of the second source image data.

26. The video encoding pipeline of claim 1, wherein the low resolution motion estimation block is configured to:

determine third downscaled source image data generated by reducing resolution of third source image data corresponding with a third image frame;

generate the first downscaled source image data by reducing resolution of the first source image data corresponding with the first image frame when the first source image data is retrieved from the memory;

perform a third motion estimation search in the third downscaled image data based on the first downscaled source image data to determine another downscaled reference sample; and

determine another low resolution inter-frame prediction mode that indicates location of the first source image data relative to another reference sample corresponding with the other downscaled reference sample.

27. The video encoding pipeline of claim 26, wherein:

the reconstruction block configured to determine second reconstructed image data by decoding encoded image data corresponding with the third source image data;

the motion estimation block configured to determine another candidate inter-frame prediction mode by performing a fourth motion estimation search in the second reconstructed image data based at least in part on the first source image data and the other low resolution inter-frame prediction mode; and

the mode decision block configured to determine encoding parameters to be used to encode the first source image data based at least in part on a second rate-distortion cost associated with the other candidate inter-frame prediction mode.

28. The video encoding pipeline of claim 6, wherein, to determine a candidate inter-frame prediction mode, the a motion estimation block is configure to:

determine a search area in the first image frame centered around the spatial position indicated by the motion vector of the low resolution motion estimation block; and

perform the second motion estimation search in the reconstructed image data corresponding with the search area to determine a second reference sample corresponding with the second source image data.

9. A tangible, non-transitory, computer-readable medium that stores instructions executable by one or more processors in a computing device, wherein the instructions comprise instructions to:

instruct, using the one or more processors, a video encoding pipeline implemented in the computing device to generate first downscaled source image data by downscaling first source image data corresponding with a first image frame;

instruct, using the one or more processors, the video encoding pipeline to generate second downscaled source image data by downscaling the second source image data corresponding with a second image frame;

instruct, using the one or more processors, the video encoding pipeline to determine a low resolution inter-frame prediction mode by performing a first motion estimation search in the first downscaled source image data based on the second downscaled source image data to determine a downscaled reference sample;

instruct, using the one or more processors, the video encoding pipeline to determine reconstructed image data by decoding first encoded image data corresponding with the first source image data;

instruct, using the one or more processors, the video encoding pipeline to determine a candidate inter-frame prediction mode by performing a second motion estimation search in the reconstructed image data corresponding with the first source image data based on the low resolution inter-frame prediction mode;

instruct, using the one or more processors, the video encoding pipeline to determine encoding parameters to be used to encode the second source image data based at least in part on a rate-distortion cost associated with the candidate inter-frame prediction mode; and

instruct, using the one or more processors, the video encoding pipeline to generate second encoded image data corresponding with the second source image data by applying the encoding operational parameters.

10. The computer-readable medium of claim 9, comprising instructions to instruct, using the one or more processors, the video encoding pipeline to determine statistics based at least in part on luma of the first source image data and luma of the second source image data, wherein:

the statistics comprise an indication of when a scene change is expected to occur; and

the instruction to instruct the video encoding pipeline to determine the encoding operational parameters comprise instructions to instruct the video encoding pipeline to determine a prediction technique used to encode the second source image data based at least in part on when the scene change is expected to occur.

11. The computer-readable medium of claim 9, wherein the instructions to downscale the image data comprise instructions to reduce resolution of the image by one-sixteenth.

12. The computer-readable medium of claim 9, wherein the instructions to instruct the video encoding pipeline to determine the low resolution inter-frame prediction mode comprise instructions to:

determine a motion vector that indicates position of a reference sample corresponding with the downscaled reference sample in the first image frame relative to position of the second source image data in the second image frame; and

determine a reference index that indicates display order of the first image frame relative to the second image frame.

13. The computer-readable medium of claim 9, wherein the instructions to instruct the video encoding pipeline to determine the candidate inter-frame prediction mode comprise instructions to:

perform the second motion estimation search in the reconstructed image data corresponding with the first source image data based on the second source image data to determine a reference sample, wherein the low resolution inter-frame prediction mode provides an indication of where the reference sample is expected to be located in the first image frame;

determine a motion vector that indicates spatial position of the reference sample in the first image frame relative to position of the second source image data in the second image frame; and

determine a reference index that indicates display order of the first image frame relative to the second image frame.

14. The computer-readable medium of claim 9, wherein the encoding operational parameters comprise prediction techniques used to encode a coding block, a number of prediction units in the coding block, a size of the prediction units, a prediction mode used to encode each of the prediction units, a number of transform units in the coding block, a size of the transform units, whether to split the coding unit into smaller coding units, or any combination thereof.

15. A video encoding system comprising:

memory configured to store source image data corresponding with image frames to be encoded by the video encoding system, wherein each image frame is divided into a plurality of coding units and each of the plurality of coding units comprises one or more prediction units;

a low resolution motion estimation circuitry coupled to the memory, wherein the low resolution motion estimation circuitry is configured to;

retrieve first source image data corresponding with a first coding unit in a first image frame and downscaled source image data corresponding with a second image frame from the memory using direct memory access;

downscale the first source image data to generate a first downscaled coding unit, wherein the first downscaled coding unit comprises a first downscaled prediction unit corresponding with a first prediction unit in the first coding unit; and

determine a first low resolution inter-frame prediction mode by performing a first motion estimation search in the downscaled source image data corresponding with the second image frame based on the first downscaled prediction unit in the first downscaled coding unit to determine a first downscaled reference sample; and

a main pipeline coupled to the memory in parallel with the low resolution motion estimation circuitry, wherein the main pipeline is configured to;

retrieve the first source image data corresponding with the first coding unit and the first low resolution inter-frame prediction mode from the memory using direct memory access; and

determine a first prediction mode to be used to encode the first source image data corresponding with the first prediction unit in the first coding unit based at least in part on the first low resolution inter-frame prediction mode.

16. The video encoding system of claim 15, wherein:

the first downscaled coding unit comprises a second downscaled prediction unit corresponding with a second prediction unit in the first coding unit;

the low resolution motion estimation circuitry is configured to determine a second low resolution inter-frame prediction mode by performing a second motion estimation search in the downscaled source image data corresponding with the second image frame based on the second downscaled prediction unit in the first downscaled coding unit to determine a second downscaled reference sample; and

the main pipeline is configured to;

retrieve the second low resolution inter-frame prediction mode from the memory using direct memory access; and

determine a second prediction mode used to encode the first source image data corresponding with the second prediction unit in the first coding unit based at least in part on the first low resolution inter-frame prediction mode and the second low resolution inter-frame prediction mode.

17. The video encoding system of claim 15, wherein:

the low resolution motion estimation circuitry is configured to;

retrieve second source image data corresponding with a second coding unit in the first image frame from the memory using direct memory access;

downscale the second source image data to generate a second downscaled coding unit, wherein the second downscaled coding unit comprises a second downscaled prediction unit corresponding with a second prediction unit in the second coding unit; and

determine a second low resolution inter-frame prediction mode by performing a second motion estimation search in the downscaled source image data corresponding with the second image frame based on the second downscaled prediction unit in the second downscaled coding unit to determine a second downscaled reference sample; and

the main pipeline configured to;

retrieve the second source image data corresponding with the second coding unit and the second low resolution inter-frame prediction mode from the memory using direct memory access; and

determine a second prediction mode to be used to encode the second source image data corresponding with the second prediction unit in the second coding unit based at least in part on the second low resolution inter-frame prediction mode.

18. The video encoding pipeline of system 15, wherein the first coding unit is a 64×

64 sample.

19. The video encoding pipeline of system 15, wherein the second image frame is configured to be display directly before or directly after the first image frame.

20. A computing device comprising:

an image data source configured to generate first image data corresponding with a first image and second image data corresponding with a second image;

a low resolution motion estimation pipeline comprising first image data processing circuitry programmed to;

generate first downscaled image data by downscaling the first image data generated by the image data source when the low resolution motion estimation pipeline receives the first image data;

when the low resolution motion estimation pipeline receives the second image data;

generate second downscaled image data by downscaling the second image data generated by the image data source;

perform a first motion estimation search in the first downscaled image data based on the second downscaled image data to determine a low resolution inter-frame prediction mode; and

determine statistics based at least in part on luma of the second downscaled image data; and

a main video encoding pipeline configured to generate first encoded image data by encoding the first image data and second encoded image data by encoding the second image data, wherein the video encoding pipeline comprises second processing circuitry programmed to;

determine first reconstructed image data by decoding the first encoded image data; and

when the main video encoding pipeline receives the second image data;

determine a candidate inter-frame prediction mode by performing a second motion estimation search in the first reconstructed image data based at least in part on the low resolution inter-frame prediction mode; and

determine encoding operational parameters to be used to encode the second image data based at least in part on the candidate inter-frame prediction mode and the statistics determined by the low resolution motion estimation pipeline.

21. The computing device of claim 20, wherein the statistics comprise:

a luma histogram configured to indicate number of pixels in the second downscaled image data at each luma value; and

a zero vector sum of absolute difference between the second downscaled image data corresponding with the second image and a downscaled reference sample at a same position in the first image.

22. The computing device of claim 20, wherein:

the low resolution motion estimation pipeline is configured to process the second image data corresponding with the second image frame while the main video encoding pipeline processes the first image data corresponding with the first image frame, wherein the second image frame to be displayed after the first image frame; and

the main video encoding pipeline is configured to determine where a scene change is expected to occur based at least in part on the statistics to facilitate frame-rate conversion.

23. The computing device of claim 20, wherein the low resolution motion estimation pipeline is configured to process image data generated by the image data source in a downscaled resolution one or more image frames before the main video encoding pipeline processes the image data in a full resolution.

24. The computing device of claim 20, wherein the image data source comprises an image sensor configured to generate a digital representation of proximate physical features as source image data.

25. The computing display of claim 20, wherein the computing device comprises a portable phone, a media player, a personal data organizer, a handheld game platform, a tablet device, a computer, or any combination thereof.