Frame level multimedia decoding with frame information table

• US 8,358,704 B2
• Filed: 04/03/2007
• Issued: 01/22/2013
• Est. Priority Date: 04/04/2006
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Shi, Fang, Oguz, Seyfullah Halit, Raveendran, Vijayalakshmi R.
• Primary Examiner(s): Pearson, David
• Assistant Examiner(s): PLECHA, THADDEUS J
• Application Number:
  US11/696,071
• Publication Number:
  US 20070291836A1
• Time in Patent Office: 2,121 Days
• Field of Search: 375/240.27
• US Class Current: 375/240.27
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 19/895 in combination with error concealment
• CPC Class Codes: H04N 21/234381 by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream

Abstract

Apparatus and method to decode video data while maintaining a target video quality using an integrated error control system including error detection, resynchronization and error recovery are described. Robust error control can be provided by a joint encoder-decoder functionality including multiple error resilience designs. In one aspect, error recovery may be an end-to-end integrated multi-layer error detection, resynchronization and recovery mechanism designed to achieve reliable error detection and error localization. The error recovery system may include cross-layer interaction of error detection, resynchronization and error recovery subsystems. In another aspect, error handling of a scalable coded bitstream is coordinated across a base-layer and enhancement layer of scalable compressed video.

Claims

1. A method of processing multimedia data comprising:

receiving the multimedia data;

organizing descriptive information about the multimedia data in a first layer, wherein the descriptive information is related to the processing of the multimedia data in a second layer; and

providing instructions related to the processing of the multimedia data in the second layer based at least in part on the descriptive information, the providing instructions including determining a method of error control based at least in part on error distribution information and providing instructions to the second layer related to the determined method of error control.

2. The method of claim 1, further comprising passing the descriptive information to the second layer.

3. The method of claim 1, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

4. The method of claim 1, wherein the multimedia data comprises some erroneous data, the method further comprising organizing the descriptive information to include information representing an error distribution of the erroneous data within the multimedia data.

5. The method of claim 4, further comprising determining the instructions based at least in part on the error distribution information.

6. The method of claim 1, further comprising changing the processing of the multimedia data in the second layer based at least in part on the instructions.

7. The method of claim 1, wherein the descriptive information comprises metadata.

8. The method of claim 1, wherein the determined method of error control comprises one or more of error recovery, error concealment, and interpolation of a frame.

81. The method of claim 1, wherein organizing descriptive information comprises organizing the descriptive information into a data structure.

82. The method of claim 1, wherein organizing descriptive information comprises organizing the descriptive information on at least one of a frame level, a slice level, a pixel block level or a pixel level.

9. An apparatus for processing multimedia data comprising:

a receiver implemented in hardware configured to receive the multimedia data;

an information organizer configured to organize descriptive information about the multimedia data in a first layer, wherein the descriptive information is related to the processing of the multimedia data in a second layer; and

an error control decision subsystem configured to provide instructions related to the processing of the multimedia data in the second layer based at least in part on the descriptive information, the error control decision subsystem configured to determine a method of error control based at least in part on error distribution information, the instructions provided to the second layer being related to the determined method of error control.

10. The apparatus of claim 9, wherein the information organizer is further configured to pass the descriptive information to the second layer.

11. The apparatus of claim 9, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

12. The apparatus of claim 9, wherein the multimedia data comprises some erroneous data, and the multimedia data processor is further configured to organize the descriptive information to include information representing an error distribution of the erroneous data within the multimedia data.

13. The apparatus of claim 12, wherein the error control decision subsystem is further configured to determine the instructions based at least in part on the error distribution information.

14. The apparatus of claim 9, further comprising a multimedia decoder configured to change the processing of the multimedia data in the second layer based at least in part on the instructions.

15. The apparatus of claim 9, wherein the descriptive information comprises metadata.

16. The apparatus of claim 9, wherein the determined method of error control comprises one or more of error recovery, error concealment, and interpolation of a frame.

17. An apparatus for processing multimedia data comprising:

means for receiving the multimedia data;

means for organizing descriptive information about the multimedia data in a first layer, wherein the descriptive information is related to the processing of the multimedia data in a second layer; and

means for providing instructions related to the processing of the multimedia data in the second layer based at least in part on the descriptive information, the means for providing instructions including means for determining a method of error control based at least in part on error distribution information, the instructions provided to the second layer being related to the determined method of error control.

18. The apparatus of claim 17, further comprising means for passing the descriptive information to the second layer.

19. The apparatus of claim 17, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

20. The apparatus of claim 17, wherein the multimedia data comprises some erroneous data, wherein the organizing means organizes the descriptive information to include information representing an error distribution of the erroneous data within the multimedia data.

21. The apparatus of claim 20, further comprising means for determining the instructions based at least in part on the error distribution information.

22. The apparatus of claim 17, further comprising means for changing the processing of the multimedia data in the second layer based at least in part on the instructions.

23. The apparatus of claim 17, wherein the descriptive information comprises metadata.

24. The apparatus of claim 17, wherein the determined method of error control comprises one or more of error recovery, error concealment, and interpolation of a frame.

25. A non-transitory machine readable medium comprising program code, which, when executed on one or more machines, causes the one or more machines to perform program operations, the program code comprising:

code for receiving multimedia data;

code for organizing descriptive information about the multimedia data in a first layer, wherein the descriptive information is related to the processing of the multimedia data in a second layer;

code for providing instructions related to the processing of the multimedia data in the second layer based at least in part on the descriptive information; and

code for determining a method of error control based at least in part on error distribution information, the instructions provided to the second layer being related to the determined method of error control.

26. The machine readable medium of claim 25, further comprising code for passing the descriptive information to the second layer.

27. The machine readable medium of claim 25, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

28. The machine readable medium of claim 25, wherein the multimedia data comprises some erroneous data, and further comprising code for organizing the descriptive information to include information representing an error distribution of the erroneous data within the multimedia data.

29. The machine readable medium of claim 28, further comprising code for determining the instructions based at least in part on the error distribution information.

30. The machine readable medium of claim 25, further comprising code for changing the processing of the multimedia data in the second layer based at least in part on the instructions.

31. The machine readable medium of claim 25, wherein the descriptive information comprises metadata.

32. The machine readable medium of claim 25, wherein the determined method of error control comprises one or more of error recovery, error concealment, and interpolation of a frame.

33. A method of processing multimedia data comprising:

receiving the multimedia data;

processing the multimedia data in an upper layer, the processing including determining a method of error control based at least in part on error distribution information;

instructing a lower layer based at least in part on information associated with the processing of the multimedia data in the upper layer, the instructing related to the determined method of error control; and

processing the multimedia data in the lower layer based at least in part on the information associated with the processing of the multimedia data in the upper layer.

34. The method of claim 33, further comprising organizing descriptive information in the lower layer about the multimedia data based at least in part on the information associated with the processing of the multimedia data in the upper layer.

35. The method of claim 34, further comprising providing instructions related to the processing of the multimedia data in the upper layer based at least in part on the descriptive information.

36. The method of claim 34, wherein the descriptive information comprises metadata.

37. The method of claim 34, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

38. The method of claim 33, wherein instructing the lower layer comprises passing information comprising one or more of processing time, processing action and processing status.

39. An apparatus for processing multimedia data comprising:

a receiver implemented in hardware configured to receive the multimedia data;

an upper layer decoder subsystem configured to process the multimedia data in an upper layer, and to instruct a lower layer based at least in part on information associated with the processing of the multimedia data in the upper layer, the processing including determining a method of error control based at least in part on error distribution information, the instructing related to the determined method of error control; and

a lower layer decoder subsystem configured to process the multimedia data in the lower layer based at least in part on the information associated with the processing of the multimedia data in the upper layer.

40. The apparatus of claim 39, further comprising an information organizer configured to organize descriptive information in the lower layer about the multimedia data based at least in part on the information associated with the processing of the multimedia data in the upper layer.

41. The apparatus of claim 40, further comprising an error control decision subsystem configured to provide instructions related to the processing of the multimedia data in the upper layer based at least in part on the descriptive information.

42. The apparatus of claim 40, wherein the descriptive information comprises metadata.

43. The apparatus of claim 40, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

44. The apparatus of claim 39, wherein the upper layer decoder subsystem is further configured to instruct the lower layer by passing information comprising one or more of processing time, processing action and processing status.

45. An apparatus for processing multimedia data comprising:

means for receiving the multimedia data;

means for processing the multimedia data in an upper layer, the processing including determining a method of error control based at least in part on error distribution;

means for instructing a lower layer based at least in part on information associated with the processing of the multimedia data in the upper layer, the instructing related to the determined method of error control; and

means for processing the multimedia data in the lower layer based at least in part on the information associated with the processing of the multimedia data in the upper layer.

46. The apparatus of claim 45, further comprising means for organizing descriptive information in the lower layer about the multimedia data based at least in part on the information associated with the processing of the multimedia data in the upper layer.

47. The apparatus of claim 46, further comprising means for providing instructions related to the processing of the multimedia data in the upper layer based at least in part on the descriptive information.

48. The apparatus of claim 46, wherein the descriptive information comprises metadata.

49. The apparatus of claim 46, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

50. The apparatus of claim 45, wherein the means for instructing the lower layer comprises means for passing information comprising one or more of processing time, processing action and processing status.

51. A non-transitory machine readable medium comprising program code, which, when executed on one or more machines, causes the one or more machines to perform program operations, the program code comprising:

code for receiving multimedia data;

code for processing the multimedia data in an upper layer;

code for determining a method of error control in the upper layer based at least in part on error distribution information;

code for instructing a lower layer based at least in part on information associated with the processing of the multimedia data in the upper layer, the instructing related to the determined method of error control; and

code for processing the multimedia data in the lower layer based at least in part on the information associated with the processing of the multimedia data in the upper layer.

52. The machine readable medium of claim 51, further comprising code for organizing descriptive information in the lower layer about the multimedia data based at least in part on the information associated with the processing of the multimedia data in the upper layer.

53. The machine readable medium of claim 52, further comprising code for providing instructions related to the processing of the multimedia in the upper layer based at least in part on the descriptive information.

54. The machine readable medium of claim 52, wherein the descriptive information comprises metadata.

55. The machine readable medium of claim 52, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

56. The machine readable medium of claim 51, further comprising code for instructing the lower layer by passing information comprising one or more of processing time, processing action and processing status.

57. A method of processing multimedia data comprising:

receiving the multimedia data;

receiving descriptive information about the multimedia data from a first layer, wherein the descriptive information is related to the processing of the multimedia data in a second layer;

receiving an instruction in the second layer, the instruction based on a method of error control determined based at least in part on error distribution information; and

processing the multimedia data in the second layer based at least in part on the received descriptive information and the received instruction.

58. The method of claim 57,

wherein the instruction is based at least in part on the descriptive information.

59. The method of claim 58, wherein the received instruction is related to a method of error control.

60. The method of claim 59, wherein the method of error control comprises one or more of error recovery, error concealment, and interpolation of a frame.

61. The method of claim 57, wherein the descriptive information comprises metadata.

62. The method of claim 57, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

63. An apparatus for processing multimedia data comprising:

a receiver implemented in hardware configured to receive the multimedia data; and

a decoder configured to receive descriptive information about the multimedia data from a first layer, wherein the descriptive information is related to the processing of the multimedia data in a second layer, the decoder configured to receive an instruction in the second layer, the instruction based on a method of error control determined based at least in part on error distribution information, and to process the multimedia data in the second layer based at least in part on the received descriptive information and the received instruction.

64. The apparatus of claim 63, wherein the instruction is based at least in part on the descriptive information.

65. The apparatus of claim 64, wherein the received instruction is related to a method of error control.

66. The apparatus of claim 65, wherein the method of error control comprises one or more of error recovery, error concealment, and interpolation of a frame.

67. The apparatus of claim 63, wherein the descriptive information comprises metadata.

68. The apparatus of claim 63, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

69. An apparatus for processing multimedia data comprising:

means for receiving the multimedia data;

means for receiving descriptive information about the multimedia data from a first layer, wherein the descriptive information is related to the processing of the multimedia data in a second layer;

means for receiving an instruction in the second layer, the instruction based on a method of error control determined based at least in part on error distribution information; and

means for processing the multimedia data in the second layer based at least in part on the received descriptive information and the received instruction.

70. The apparatus of claim 69,

wherein the instruction is based at least in part on the descriptive information.

71. The apparatus of claim 70, wherein the received instruction is related to a method of error control.

72. The apparatus of claim 71, wherein the method of error control comprises one or more of error recovery, error concealment, and interpolation of a frame.

73. The apparatus of claim 69, wherein the descriptive information comprises metadata.

74. The apparatus of claim 69, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.

75. A non-transitory machine readable medium comprising program code, which, when executed on one or more machines, causes the one or more machines to perform program operations, the program code comprising:

code for receiving multimedia data;

code for receiving descriptive information about the multimedia data from a first layer, wherein the descriptive information is related to the processing of the multimedia data in a second layer;

code for receiving an instruction in the second layer, the instruction based on a method of error control determined based at least in part on error distribution information, and

code for processing the multimedia data in the second layer based at least in part on the received descriptive information and the received instruction.

76. The machine readable medium of claim 75,

wherein the instruction is based at least in part on the descriptive information.

77. The machine readable medium of claim 76, wherein the received instruction is related to a method of error control.

78. The machine readable medium of claim 77, wherein the method of error control comprises one or more of error recovery, error concealment, and interpolation of a frame.

79. The machine readable medium of claim 75, wherein the descriptive information comprises metadata.

80. The machine readable medium of claim 75, wherein the descriptive information comprises one or more of frame characteristic information, base or enhancement data identifying information, timing information, an encoding type, a frame type, synchronization information, and predictive encoding related information.