Multi-threaded streaming data decoding

• US 8,213,518 B1
• Filed: 01/26/2007
• Issued: 07/03/2012
• Est. Priority Date: 10/31/2006
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Wang, Jason N., Mehta, Milan
• Primary Examiner(s): Davis, Zachary A
• Application Number:
  US11/627,867
• Time in Patent Office: 1,985 Days
• Field of Search: 375/240.25, 375/240.16, 375/240.23, 712/215, 340/240.28, 348/721
• US Class Current: 375/240.25
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness

Abstract

Streaming data may be decoded by dividing a process for decoding the streaming data into two or more tasks based on data dependencies between the two or more tasks. The two or more tasks may be executed in parallel on three or more processors in a way that balances a processing load of executing the two or more tasks among the three or more processors.

Claims

1. A method for decoding streaming data, comprising:

dividing a process for decoding the streaming data into two or more tasks based on data dependencies between the two or more tasks; and

executing the two or more tasks in parallel on three or more processors in a way that balances a processing load of executing the two or more tasks among the three or more processors by determining a section size for the streaming data, dividing a portion of the streaming data into two or more sections of the section size and processing the two or more sections in parallel on two or more processors, wherein the section size is chosen such that a hardware clock rate divided by a data transfer latency is greater than a data dependency frequency for the section.

2. The method of claim 1 wherein dividing the process into two or more tasks based on data dependencies includes dividing the process into two or more tasks based on data frequency dependencies between the two or more tasks.

3. The method of claim 1 wherein the streaming data includes one or more pictures, wherein each picture is divided into one or more sections.

4. The method of claim 1, further comprising, adapting the section size to facilitate transferring data across different decoding tasks within the decoding process.

5. The method of claim 1 wherein the section size is chosen such that each section may fit into a memory space available for buffering input data for one of the tasks.

6. The method of claim 1, wherein the streaming data includes one or more pictures, wherein each picture is divided into one or more slices, wherein executing the two or more tasks in parallel on three or more processors includes processing the same task on two or more different slices within the same picture using two or more different processors.

7. The method of claim 1 wherein executing the two or more tasks in parallel on three or more processors in a way that balances the processing load includes overlapping two or more different tasks.

8. The method of claim 7, wherein overlapping two or more different tasks includes overlapping execution of a first task on a first processor with the execution of a second task on second processor.

9. The method of claim 8 wherein the streaming data is video data divided into one or more pictures, wherein each picture is partitioned into n sections, where n is greater than 1.

10. The method of claim 9 wherein the first task is a pixel prediction and reconstruction task and the second task is a de-blocking task.

11. The method of claim 9 wherein overlapping two or more different tasks includes overlapping execution of the first task on a section of one picture with execution of the second task on a different section of a different picture.

12. The method of claim 1 wherein executing the two or more tasks in parallel on three or more processors in a way that balances the processing load includes overlapping two tasks in the same picture and in different pictures by use of picture sections.

13. The method of claim 12 wherein a first processor and a second processor overlap processing of a first task for picture (i−

1) and processing of a second task for picture (i) on a section-by-section basis, wherein i is a picture index.

14. The method of claim 13 wherein the second processor and a third processor further overlap processing of the first task for picture (i) with processing of the second task for picture (i) on a section-by-section basis.

15. The method of claim 13 wherein the first task is a pixel prediction and reconstruction task and the second task is a de-blocking task.

16. The method of claim 1 wherein executing the two or more tasks in parallel on three or more processors in a way that balances the processing load includes dividing one or more decoding tasks into two or more sub-tasks and assigning two or more different processors to process the sub-tasks.

17. The method of claim 16 wherein dividing one or more decoding tasks into two or more sub-tasks includes dividing the streaming data into two or more groups and processing each of the two or more groups with a different one of the two or more sub-tasks.

18. The method of claim 17 wherein processing each of the two or more groups with a different one of the two or more sub-tasks includes:

processing a first group with a first sub-task,

subsequently processing a second group with the first sub-task in parallel with processing a third group with a second sub-task.

19. The method of claim 18, wherein processing each of the two or more groups with a different one of the two or more sub-tasks further includes:

processing a fourth group with a third sub-task after completing processing of the second group with the first sub-task and processing the third group with the second sub-task.

20. The method of claim 19 wherein the first, second and third sub-tasks are implemented on different processors.

21. The method of claim 19, further comprising completing processing of first section of the streaming data containing the first group with a first task before processing the first group with the first-sub task, and completing processing of a second section of the streaming data containing the second group with the first task before processing the second group with the second sub-task and completing processing of a third section of the streaming data with the first task before processing the third group with the third sub-task.

22. The method of claim 1 wherein the streaming data is streaming video data.

23. The method of claim 22 wherein the streaming video data has been encoded according to the AVC (H.264) coding standard.

24. The method of claim 22 wherein dividing the process for decoding the streaming data into two or more tasks includes dividing the decoding process into a video coded layer (VCL) decoding task, a motion vector reconstruction task, a pixel prediction/reconstruction task, and a de-blocking task.

25. The method of claim 1, wherein the streaming data is divided into two or more sections, wherein dividing the process for decoding the streaming data into two or more tasks includes dividing the decoding process into one or more producer tasks and one or more consumer tasks wherein each consumer task start after a corresponding producer task has been completed for one section, whereby, a consumer task synchronizes (syncs) only once per section with a producer task or tasks upon which the consumer task depends for input data.

26. The method of claim 1, further comprising receiving a stream of encoded data elements and metadata associated with each encoded data element, wherein executing the tasks in parallel on three or more processors in a way that balances a processing load of executing the two or more tasks among the three or more processors includes utilizing the metadata to facilitate balancing of the processing load.

27. The method of claim 26 wherein the metadata includes statistical information regarding a number of intra macroblocks in a picture or slice.

28. The method of claim 27 wherein utilizing the metadata to facilitate balancing of the processing load includes estimating a number of processor cycles required for one or more tasks and determining a section size or a group size for parallel processing of the streaming data.

29. The method of claim 26 wherein the metadata includes a number of skipped macroblocks, a number of blocks having coded DCT coefficients, the number of motion vectors, or a motion vector range.

30. A video decoder apparatus, comprising:

three or more processors, wherein each of the three or more processors has a separate dedicated associated local memory; and

a memory coupled to the three or more processors storing processor readable instructions operable on the three or more processors, the instructions being configured to implement a method for decoding streaming data, the method comprising;

dividing a process for decoding the streaming data into two or more tasks based on data dependencies between the two or more tasks; and

executing the two or more tasks in parallel on the three or more processors in a way that balances a processing load of executing the two or more tasks among the three or more processors by determining a section size for the streaming data, dividing a portion of the streaming data into two or more sections of the section size and processing the two or more sections in parallel on two or more processors, wherein the section size is chosen such that a hardware clock rate divided by a data transfer latency is greater than a data dependency frequency for the section.

31. The apparatus of claim 30, wherein the instructions are adapted to decode video streaming data.

32. The apparatus of claim 31 wherein the instructions include instructions configured to operate in parallel on the three or more processors at one or more levels of abstraction for tasks selected from the group consisting of video coded layer (VCL) decoding, motion vector (MV) reconstruction, pixel prediction and reconstruction, and de-blocking.

33. The apparatus of claim 30 wherein each of the processors has an associated local memory, wherein executing the two or more tasks in parallel on the three or more processors includes dividing the streaming data into two or more sections, processing a first section with a first task running on a first processor and buffering an output of the first task in a local memory, wherein the section size is selected such that the output will fit into available space in the local memory.