Multithread processing of video frames

• US 8,804,849 B2
• Filed: 05/24/2012
• Issued: 08/12/2014
• Est. Priority Date: 03/16/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Duvivier, Christian L.
• Primary Examiner(s): HALLENBECK-HUBER, JEREMIAH CHARLES
• Application Number:
  US13/480,451
• Publication Number:
  US 20120230424A1
• Time in Patent Office: 810 Days
• Field of Search: 375/240.29
• US Class Current: 375/240.29
• CPC Class Codes: H04N 19/127 Prioritisation of hardware or computational resources
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness

Abstract

Method for determining frame slice sizes of a frame for multithreaded decoding. The frame is encoded using at least two different slice types based on size where a large-type slice is at least two times larger than a small-type slice and/or the large-type slices comprise 70-90% of the frame. In some embodiments, the number of large-type slices is equal to the number of threads available for decoding and comprise the beginning slices of the frame to be decoded before the small-type slices. Methods for multithreaded deblocking of the frame under the H.264 codec is provided where first and second threads processes first and second sections of the frame in parallel. The first section comprises macroblocks on one side of a diagonal line and the second section comprises the remainder, the diagonal line extending from a first corner of a sub-frame to a second corner of the sub-frame.

Claims

1. A method for deblock filtering a video frame comprising a plurality of rectangular blocks of pixels, the method comprising:

partitioning the frame along boundaries of a set of diagonally aligned rectangular blocks into a first group of rectangular blocks and a second group of rectangular blocks, the set of diagonally aligned rectangular blocks spans the frame from one border of the frame to another border of the frame, the first group of rectangular blocks comprising rectangular blocks on a first side of the boundaries of the set of diagonally aligned rectangular blocks, the second group of rectangular blocks comprising all remaining rectangular blocks in the frame that are not in the first group of rectangular blocks; and

filtering the first group of rectangular blocks using a first computing resource while filtering the second group of rectangular blocks using a second computing resource.

2. The method of claim 1, wherein the first computing resource comprises a first processor thread and the second computing resource comprises a second processor thread.

3. The method of claim 1, wherein each of the first and second groups of rectangular blocks is divided into a plurality of sub-groups, the method further comprising, for each group of the first and second groups of rectangular blocks, assigning each rectangular block to a sub-group of the group based on a position of the rectangular block in the group of rectangular blocks.

4. The method of claim 3, wherein each sub-group of a group of rectangular blocks comprises all rectangular blocks on a same row of the group.

5. The method of claim 3, wherein the second computing resource waits for the first computing resource to complete filtering rectangular blocks in a particular row before starting filtering rectangular blocks in the same particular row.

6. The method of claim 3, wherein filtering the first group of rectangular blocks using the first computing resource comprises processing sub-groups of the first group of rectangular blocks according to a first processing order using the first computing resource, wherein filtering the second group of rectangular blocks using the second computing resource comprises processing sub-groups of the second group of rectangular blocks according to a second processing order using the second computing resource.

7. The method of claim 1, wherein the frame comprises a rectangular sub-frame, wherein the set of diagonally aligned rectangular blocks extends from a first corner of the rectangular sub-frame to a second corner of the rectangular sub-frame.

8. The method of claim 7, wherein the rectangular sub-frame is a square sub-frame, wherein each side of the square sub-frame is equal in length to at least one side of the frame.

9. The method of claim 1, wherein the frame is a luma frame that is associated with a chroma frame, the method further comprising processing the associated chroma frame using the second computing resource while the second computing resource is waiting for the first computing resource to finish processing a particular set of rectangular blocks of the first group of rectangular blocks of the luma frame, the particular set of rectangular blocks being necessary for processing a next rectangular block in the second group of rectangular blocks of the luma frame.

10. The method of claim 1, wherein each rectangular block is a macroblock.

11. A non-transitory computer readable medium storing a computer program for execution by at least one processing unit, the computer program for deblock filtering a video frame comprising a plurality of groups of pixels, the computer program comprising sets of instructions for:

identifying first and second pluralities of groups of pixels in the frame by partitioning the frame along a diagonal line that spans the frame from one border of the frame to another border of the frame, wherein the groups of pixels in the first plurality of groups of pixels do not depend on any group of pixels in the second plurality of groups of pixels for filtering, wherein a first group of pixels in the second plurality of groups of pixels depends on a second group of pixels in the first plurality of groups of pixels for filtering; and

filtering the first plurality of groups of pixels using a first computing resource while filtering the second pluralities of groups of pixels using a second computing resource.

12. The non-transitory computer readable medium of claim 11, wherein the first plurality of groups of pixels is on a first side of the diagonal line and the second plurality of groups of pixels is on a second side of the diagonal line.

13. The non-transitory computer readable medium of claim 12, wherein the first plurality of groups of pixels further comprises groups of pixels containing a set of pixels aligned in a diagonal direction of the frame and the second plurality of groups of pixels comprises remaining groups of pixels of the frame that are not in the first plurality of groups of pixels.

14. The non-transitory computer readable medium of claim 12, wherein the frame comprises a rectangular sub-frame, wherein the set of pixels aligned in the diagonal direction extends from a first corner of the rectangular sub-frame to a second corner of the rectangular sub-frame.

15. The non-transitory computer readable medium of claim 11, wherein each group of pixels comprises a macroblock.

16. The non-transitory computer readable medium of claim 11, wherein the video frame comprises first and second sections, the first section comprising the first plurality of groups of pixels and the second section comprising the second plurality of groups of pixels wherein each of the first and second sections is divided into a plurality of sub-sections, the computer program further comprising a set of instructions for assigning, for each of the first and second sections, each group of pixels of the section to a sub-section of the section based on a position of the group of pixels in the section.

17. The non-transitory computer readable medium of claim 16, wherein each sub-section of a section comprises all groups of pixels on a same row of the section.

18. The non-transitory computer readable medium of claim 16, the computer program further comprising a set of instructions for determining, for each of the first and second sections, a sequential processing order according to which the sub-sections of the section are assigned to a computing resource for processing.

19. The non-transitory computer readable medium of claim 11, wherein the video frame comprises first and second sections, the first section comprising the first plurality of groups of pixels and the second section comprising the second plurality of groups of pixels, wherein each of the first and second sections is divided into a plurality of sub-sections, wherein the set of instructions for filtering the first plurality of groups of pixels using the first computing resource comprises a set of instructions for processing sub-sections of the first section according to a first processing order using the first computing resource, wherein the set of instructions for filtering the second plurality of groups of pixels using the second computing resource comprises a set of instructions for processing sub-sections of the second section according to a second processing order using the second computing resource.

20. A decoder for decoding a video frame, the decoder comprising:

a module for partitioning the frame along boundaries of a set of diagonally aligned rectangular blocks into a first group of rectangular blocks and a second group of rectangular blocks, the set of diagonally aligned rectangular blocks spans the frame from one border of the frame to another border of the frame, the first group of rectangular blocks comprising rectangular blocks on a first side of the boundaries of the set of diagonally aligned rectangular blocks, the second group of rectangular blocks comprising all remaining rectangular blocks in the frame that are not in the first group of rectangular blocks;

a first computing resource for deblock filtering the first group of rectangular blocks; and

a second computing resource for deblock filtering the second group of rectangular blocks, wherein the first and second computing resources perform deblock filtering of the first and second groups of rectangular blocks in parallel.

21. The decoder of claim 20, wherein each of the first and second groups of rectangular blocks is divided into a plurality of sub-groups, wherein the module for partitioning the frame assigns, for each of the first and second groups of rectangular blocks, each rectangular block of the group of rectangular blocks to a sub-group of the group of rectangular blocks based on a position of the rectangular block in the group of rectangular blocks.

22. The decoder of claim 21, wherein each sub-group of a group of rectangular blocks comprises all rectangular blocks on a same row of the group.

23. The decoder of claim 21, wherein the second computing resource waits for the first computing resource to complete filtering rectangular blocks in a particular row before starting filtering rectangular blocks in the same particular row.

24. The decoder of claim 21, wherein each rectangular block comprises a macroblock.

25. A system for deblock filtering a video frame, the system comprising:

a module for identifying first and second pluralities of groups of pixels in the frame by partitioning the frame along a diagonal line that spans the frame from one border of the frame to another border of the frame, wherein the groups of pixels in the first plurality of groups of pixels do not depend on any group of pixels in the second plurality of groups of pixels for filtering, wherein a first group of pixels in the second plurality of groups of pixels depends on a second group of pixels in the first plurality of groups of pixels for filtering; and

a first computing resource for deblock filtering the first plurality of groups of pixels; and

a second computing resource for deblock filtering the second plurality of groups of pixels, wherein the first and second computing resources perform deblock filtering of the first and second pluralities of groups of pixels in parallel.

26. The system of claim 25, wherein the first plurality of groups of pixels is on a first side of the diagonal line and the second plurality of groups of pixels is on a second side of the diagonal line.

27. The system of claim 25, wherein the first computing resource comprises a first CPU and the second computing resource comprises a second CPU.

28. The system of claim 25, wherein the first computing resource comprises a first computing thread and the second computing resource comprises a second computing thread.

29. The system of claim 25, wherein a group of pixels is a macroblock.