Content driven transcoder that orchestrates multimedia transcoding using content information

• US 8,879,856 B2
• Filed: 09/26/2006
• Issued: 11/04/2014
• Est. Priority Date: 09/27/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Raveendran, Vijayalakshmi R., Walker, Gordon Kent, Tian, Tao, Bhamidipati, Phanikumar, Shi, Fang, Chen, Peisong, Subramania, Sitaraman Ganapathy, Oguz, Seyfullah Halit
• Primary Examiner(s): Seth, Manav
• Application Number:
  US11/528,141
• Publication Number:
  US 20070081587A1
• Time in Patent Office: 2,961 Days
• Field of Search: 382232-253, 375240-254
• US Class Current: 382/232
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/114 Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames H04N19/107 takes precedence
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/18 the unit being a set of transform coefficients
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/194 involving only two passes
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/65 : using error resilience
• CPC Class Codes: H04N 19/86 : involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/87 : involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 19/89 : involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 21/234309 : by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo conversion of standards in analog television systems H04N7/01
• CPC Class Codes: H04N 21/2353 : specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/2389 : Multiplex stream processing, e.g. multiplex stream encrypting
• CPC Class Codes: H04N 5/144 : Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/147 : Scene change detection
• CPC Class Codes: H04N 5/21 : Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 7/0115 : with details on the detection of a particular field or frame pattern in the incoming video signal, e.g. 3:2 pull-down pattern
• CPC Class Codes: H04N 7/012 : Conversion between an interlaced and a progressive signal for graphics images G09G2310/0229

Abstract

Apparatus and methods of using content information for encoding multimedia data are described. A method of processing multimedia data includes receiving multimedia data, and encoding the multimedia data into a first data group and a second data group based on content of the multimedia data, the first data group being configured to be independently decodable from the second data group, and wherein the first and second data groups are encoded at different quality levels. The method can also include classifying the content of the multimedia data and encoding the multimedia data based on the content classification.

Claims

1. A method of processing multimedia data in an encoder that is implemented at least partially in hardware or a processor, the method comprising:

receiving multimedia data in the encoder;

classifying content of the multimedia data via the encoder based on spatial and temporal complexity of the multimedia data; and

encoding, via the encoder, the multimedia data into a first data group and a second data group based on the content classification of the multimedia data, the first data group being configured to be independently decodable from the second data group, wherein the first data group defines a base layer and the second data group defines an enhancement layer.

2. The method of claim 1, wherein the first data group comprises I-frames and P-frames, and wherein the second data group comprises I-frames, P-frames and B-frames.

3. The method of claim 1, wherein the first data group comprises the base layer and the second data group comprises the enhancement layer such that the first data group can be decoded as a single layer or the first and second data groups can be combined when decoded.

4. The method of claim 1, wherein the encoding comprises determining a first quantization parameter of the multimedia data for encoding the first data group and determining a second quantization parameter for encoding the second data group, wherein the determination of the first and second quantization parameter is based on the content classification.

5. The method of claim 1, wherein the encoding comprises allocating a bit rate for at least a portion of the multimedia data based on the content classification.

6. The method of claim 1, wherein the encoding further comprises:

using the content classification to detect scene changes; and

determining whether to include I-frames in the first data group and the second data group based on the detected scene changes.

7. The method of claim 1, wherein the encoding further comprises determining a frame rate for encoding the multimedia data based on the content classification.

8. The method of claim 1, wherein the encoding comprises performing motion estimation of the multimedia data based on the content classification.

9. The method of claim 1, further comprising determining a first frame rate for encoding the first data group and determining a second frame rate for encoding the second data group, wherein the first frame rate is less than the second frame rate.

10. The method of claim 1, wherein the encoding comprises performing error resilience processing on the multimedia data based on the content classification.

11. The method of claim 1, wherein the encoding comprises encoding the first data group and the second data group such that the first data group can be decoded to form displayable multimedia data if the second data group is not available, and the first data group and the second data group can be decoded in combination to form displayable multimedia data if both the first data group and the second data group are available.

12. The method of claim 1, wherein the first quantization parameter comprises a first step size for encoding data, and wherein the second quantization parameter comprises a second step size for encoding data, wherein the first step size is larger than the second step size.

13. The method of claim 1, wherein the encoding comprises reducing noise in the multimedia data based on the content classification.

14. The method of claim 13, wherein reducing noise comprises performing artifact removal.

15. The method of claim 13, wherein reducing noise comprises processing at least a portion of the multimedia data with a de-ringing filter, wherein the strength of the de-ringing filter is based on the content classification of the multimedia data.

16. The method of claim 13, wherein reducing noise comprises processing at least a portion of the multimedia data with a de-blocking filter, wherein the strength of the de-blocking filter is based on the content classification of the multimedia data.

17. The method of claim 13, wherein reducing noise comprises filtering selected frequencies of the multimedia data.

18. The method of claim 15, wherein the strength of the de-ringing filter is based on content classification of the multimedia data.

19. The method of claim 16, wherein the strength of the de-blocking filter is based on content classification of the multimedia data.

20. The method of claim 1, wherein encoding comprises down-sampling the multimedia data.

21. The method of claim 1, wherein encoding comprises associating a quality level with the multimedia data and using the quality level and the content classification of the multimedia data to determine a bit rate for encoding the multimedia data.

22. An apparatus for processing multimedia data comprising an encoder implemented at least partially in hardware or a processor and configured to receive multimedia data, classify content of the multimedia data based on spatial and temporal complexity of the multimedia data, and encode the multimedia data into a first data group and an second data group based on the content classification of the multimedia data, the first data group configured to be independently decodable from the second data group, wherein the first data group defines a base layer and the second data group defines an enhancement layer.

23. The apparatus of claim 22, wherein the encoder comprises a content classification module configured to determine the content classification of the multimedia data.

24. The apparatus of claim 23, wherein the encoder is further configured to determine a first quantization parameter of the multimedia data for encoding the first data group and a second quantization parameter for encoding the second data group, wherein determining the first and second quantization parameters is based on the content classification of the multimedia data.

25. The apparatus of claim 23, wherein the encoder comprises a motion estimation module configured to perform motion estimation of the multimedia data based on the content classification and generate data motion compensation information, and wherein the encoding module is further configured to use the motion compensation information to encode the multimedia data.

26. The apparatus of claim 23, wherein the encoder further comprises a quantization module for determining a quantization parameter of the multimedia data based on the content classification, and wherein the encoder is further configured to use the quantization parameter to encode the multimedia data.

27. The apparatus of claim 23, wherein the encoder further comprises a bit allocation module configured to provide a bit rate for at least a portion of the multimedia data based on the content classification.

28. The apparatus of claim 23, wherein the encoder further comprises a scene change detection module configured to detect scene changes, and wherein the encoding module is further configured to include I-frames in encoded multimedia data based on detected scene changes.

29. The apparatus of claim 23, wherein the encoder further comprises a frame rate module configured to determine a frame rate of the multimedia data based on the content classification, and wherein the encoding module encodes the multimedia data based on the frame rate.

30. The apparatus of claim 23, wherein the encoder is further configured to perform error processing on the multimedia data based on the content classification.

31. An apparatus for processing multimedia data comprising:

means for receiving multimedia data;

means for classifying content of the multimedia data based on spatial and temporal complexity of the multimedia data; and

means for encoding the multimedia data into an encoded first data group and an encoded second data group based on the content classification of the multimedia data, said first data group being configured to be independently decodable from the second data group, wherein the first data group defines a base layer and the second data group defines an enhancement layer.

32. The apparatus of claim 31, wherein the receiving means comprises an encoder.

33. The apparatus of claim 31, wherein the means for encoding comprises an encoder.

34. The apparatus of claim 31, wherein the encoding means comprises a transcoder comprising an encoder.

35. A non-transitory computer readable medium comprising instructions that upon execution cause a processor to:

receive multimedia data;

classify content of the multimedia data based on spatial and temporal complexity of the multimedia data; and

encode the multimedia data into an encoded first data group and an encoded second data group based on the content classification of the multimedia data, the first data group configured to be independently decodable from the second data group, wherein the first data group defines a base layer and the second data group defines an enhancement layer.

36. The computer readable medium of claim 35, wherein the instructions to encode comprise instructions to determine a first quantization parameter of the multimedia data for encoding the first data group and a second quantization parameter for encoding the second data group based on the content classification.

37. The computer readable medium of claim 35, wherein the instructions to encode comprise instructions to allocate a bit rate for at least a portion of the multimedia data based on the content classification of the multimedia data.

38. A processor comprising a configuration to:

receive multimedia data;

classify content of the multimedia data based on spatial and temporal complexity of the multimedia data; and

encode the multimedia data into an encoded first data group and an encoded second data group based on the content classification of the multimedia data, the first data group configured to be independently decodable from the second data group, wherein the first data group defines a base layer and the second data group defines an enhancement layer.

39. The processor of claim 38, wherein the processor further comprises a configuration to determine a first quantization parameter of the multimedia data for encoding the first data group and a second quantization parameter for encoding the second data group, wherein the first and second quantization parameters are based on the content classification.