VIDEO PROCESSING WITH SCALABILITY

• US 20070230564A1
• Filed: 11/21/2006
• Published: 10/04/2007
• Est. Priority Date: 03/29/2006
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Chen, Peisong, Shi, Fang, Raveendran, Vijayalakshmi R., Tian, Tao
• Application Number:
  US11/562,360
• Publication Number:
  US 20070230564A1
• Time in Patent Office: Days
• US Class Current: 375/240.10
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/29 involving scalability at the object level, e.g. video object layer [VOL]
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/434 Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream demultiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00

Abstract

In general, this disclosure describes video processing techniques that make use of syntax elements and semantics to support low complexity extensions for multimedia processing with video scalability. The syntax elements and semantics may be added to network abstraction layer (NAL) units and may be especially applicable to multimedia broadcasting, and define a bitstream format and encoding process that support low complexity video scalability. In some aspects, the techniques may be applied to implement low complexity video scalability extensions for devices that otherwise conform to the H.264 standard. For example, the syntax element and semantics may be applicable to NAL units conforming to the H.264 standard.

Claims

1. A method for transporting scalable digital video data, the method comprising:

including enhancement layer video data in a network abstraction layer (NAL) unit; and

including one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data.

2. The method of claim 1, further comprising including one or more syntax elements in the NAL unit to indicate a type of raw byte sequence payload (RBSP) data structure of the enhancement layer data in the NAL unit.

3. The method of claim 1, further comprising including one or more syntax elements in the NAL unit to indicate whether the enhancement layer video data in the NAL unit includes intra-coded video data.

4. The method of claim 1, wherein the NAL unit is a first NAL unit, the method further comprising including base layer video data in a second NAL unit, and including one or more syntax elements in at least one of the first and second NAL units to indicate whether a decoder should use pixel domain or transform domain addition of the enhancement layer video data with the base layer video data.

5. The method of claim 1, wherein the NAL unit is a first NAL unit, the method further comprising including base layer video data in a second NAL unit, and including one or more syntax elements in at least one of the first and second NAL units to indicate whether the enhancement layer video data includes any residual data relative to the base layer video data.

6. The method of claim 1, further comprising including one or more syntax elements in the NAL unit to indicate whether the NAL unit includes a sequence parameter, a picture parameter set, a slice of a reference picture or a slice data partition of a reference picture.

7. The method of claim 1, further comprising including one or more syntax elements in the NAL unit to identify blocks within the enhancement layer video data containing non-zero transform coefficient syntax elements.

8. The method of claim 1, further comprising including one or more syntax elements in the NAL unit to indicate a number of nonzero coefficients in intra-coded blocks in the enhancement layer video data with a magnitude larger than one.

9. The method of claim 1, further comprising including one or more syntax elements in the NAL unit to indicate coded block patterns for inter-coded blocks in the enhancement layer video data.

10. The method of claim 1, wherein the NAL unit is a first NAL unit, the method further comprising including base layer video data in a second NAL unit, and wherein the enhancement layer video data is encoded to enhance a signal-to-noise ratio of the base layer video data.

11. The method of claim 1, wherein including one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data comprises setting a NAL unit type parameter in the NAL unit to a selected value to indicate that the NAL unit includes enhancement layer video data.

12. An apparatus for transporting scalable digital video data, the apparatus comprising:

a network abstraction layer (NAL) unit module that includes encoded enhancement layer video data in a NAL unit, and includes one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data.

13. The apparatus of claim 12, wherein the NAL unit module includes one or more syntax elements in the NAL unit to indicate a type of raw byte sequence payload (RBSP) data structure of the enhancement layer data in the NAL unit.

14. The apparatus of claim 12, wherein the NAL unit module includes one or more syntax elements in the NAL unit to indicate whether the enhancement layer video data in the NAL unit includes intra-coded video data.

15. The apparatus of claim 12, wherein the NAL unit is a first NAL unit, wherein the NAL unit module incluees base layer video data in a second NAL unit, and wherein the NAL unit module includes one or more syntax elements in at least one of the first and second NAL units to indicate whether a decoder should use pixel domain or transform domain addition of the enhancement layer video data with the base layer video data.

16. The apparatus of claim 12, wherein the NAL unit is a first NAL unit, the NAL unit module includes base layer video data in a second NAL unit, and wherein the NAL unit module includes one or more syntax elements in at least one of the first and second NAL units to indicate whether the enhancement layer video data includes any residual data relative to the base layer video data.

17. The apparatus of claim 12, wherein the NAL unit module includes one or more syntax elements in the NAL unit to indicate whether the NAL unit includes a sequence parameter, a picture parameter set, a slice of a reference picture or a slice data partition of a reference picture.

18. The apparatus of claim 12, wherein the NAL unit module includes one or more syntax elements in the NAL unit to identify blocks within the enhancement layer video data containing non-zero transform coefficient syntax elements.

19. The apparatus of claim 12, wherein the NAL unit module includes one or more syntax elements in the NAL unit to indicate a number of nonzero coefficients in intra-coded blocks in the enhancement layer video data with a magnitude larger than one.

20. The apparatus of claim 12, wherein the NAL unit module includes one or more syntax elements in the NAL unit to indicate coded block patterns for inter-coded blocks in the enhancement layer video data.

21. The apparatus of claim 12, wherein the NAL unit is a first NAL unit, the NAL unit module includes base layer video data in a second NAL unit, and wherein the encoder encodes the enhancement layer video data to enhance a signal-to-noise ratio of the base layer video data.

22. The apparatus of claim 12, wherein the NAL unit module sets a NAL unit type parameter in the NAL unit to a selected value to indicate that the NAL unit includes enhancement layer video data.

23. A processor for transporting scalable digital video data, the processor being configured to include enhancement layer video data in a network abstraction layer (NAL) unit, and include one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data.

24. An apparatus for transporting scalable digital video data, the method comprising:

means for including enhancement layer video data in a network abstraction layer (NAL) unit; and

means for including one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data.

25. The apparatus of claim 24, further comprising means for including one or more syntax elements in the NAL unit to indicate a type of raw byte sequence payload (RBSP) data structure of the enhancement layer data in the NAL unit.

26. The apparatus of claim 24, further comprising means for including one or more syntax elements in the NAL unit to indicate whether the enhancement layer video data in the NAL unit includes intra-coded video data.

27. The apparatus of claim 24, wherein the NAL unit is a first NAL unit, the apparatus further comprising means for including base layer video data in a second NAL unit, and means for including one or more syntax elements in at least one of the first and second NAL units to indicate whether a decoder should use pixel domain or transform domain addition of the enhancement layer video data with the base layer video data.

28. The apparatus of claim 24, wherein the NAL unit is a first NAL unit, the apparatus further comprising means for including base layer video data in a second NAL unit, and means for including one or more syntax elements in at least one of the first and second NAL units to indicate whether the enhancement layer video data includes any residual data relative to the base layer video data.

29. The apparatus of claim 24, further comprising means for including one or more syntax elements in the NAL unit to indicate whether the NAL unit includes a sequence parameter, a picture parameter set, a slice of a reference picture or a slice data partition of a reference picture.

30. The apparatus of claim 24, further comprising means for including one or more syntax elements in the NAL unit to identify blocks within the enhancement layer video data containing non-zero transform coefficient syntax elements.

31. The apparatus of claim 24, further comprising means for including one or more syntax elements in the NAL unit to indicate a number of nonzero coefficients in intra-coded blocks in the enhancement layer video data with a magnitude larger than one.

32. The apparatus of claim 24, further comprising means for including one or more syntax elements in the NAL unit to indicate coded block patterns for inter-coded blocks in the enhancement layer video data.

33. The apparatus of claim 24, wherein the NAL unit is a first NAL unit, the apparatus further comprising means for including base layer video data in a second NAL unit, and wherein the enhancement layer video data enhances a signal-to-noise ratio of the base layer video data.

34. The apparatus of claim 24, wherein the means for including one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data comprises means for setting a NAL unit type parameter in the NAL unit to a selected value to indicate that the NAL unit includes enhancement layer video data.

35. A computer program product for transport of scalable digital video data comprising:

a computer-readable medium comprising codes for causing a computer to;

include enhancement layer video data in a network abstraction layer (NAL) unit; and

include one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data.

36. A method for processing scalable digital video data, the method comprising:

receiving enhancement layer video data in a network abstraction layer (NAL) unit;

receiving one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data; and

decoding the digital video data in the NAL unit based on the indication.

37. The method of claim 36, further comprising detecting one or more syntax elements in the NAL unit to determine a type of raw byte sequence payload (RBSP) data structure of the enhancement layer data in the NAL unit.

38. The method of claim 36, further comprising detecting one or more syntax elements in the NAL unit to determine whether the enhancement layer video data in the NAL unit includes intra-coded video data.

39. The method of claim 36, wherein the NAL unit is a first NAL unit, the method further comprising:

receiving base layer video data in a second NAL unit;

detecting one or more syntax elements in at least one of the first and second NAL units to determine whether the enhancement layer video data includes any residual data relative to the base layer video data; and

skipping decoding of the enhancement layer video data if it is determined that the enhancement layer video data includes no residual data relative to the base layer video data.

40. The method of claim 36, wherein the NAL unit is a first NAL unit, the method further comprising:

receiving base layer video data in a second NAL unit;

detecting one or more syntax elements in at least one of the first and second NAL units to determine whether the first NAL unit includes a sequence parameter, a picture parameter set, a slice of a reference picture or a slice data partition of a reference picture;

detecting one or more syntax elements in at least one of the first and second NAL units to identify blocks within the enhancement layer video data containing non-zero transform coefficient syntax elements; and

detecting one or more syntax elements in at least one of the first and second NAL units to determine whether pixel domain or transform domain addition of the enhancement layer video data with the base layer data should be used to decode the digital video data

41. The method of claim 36, further comprising detecting one or more syntax elements in the NAL unit to determine a number of nonzero coefficients in intra-coded blocks in the enhancement layer video data with a magnitude larger than one.

42. The method of claim 36, further comprising detecting one or more syntax elements in the NAL unit to determine coded block patterns for inter-coded blocks in the enhancement layer video data.

43. The method of claim 36, wherein the NAL unit is a first NAL unit, the method further comprising including base layer video data in a second NAL unit, and wherein the enhancement layer video data is encoded to enhance a signal-to-noise ratio of the base layer video data.

44. The method of claim 36, wherein receiving one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data comprises receiving a NAL unit type parameter in the NAL unit that is set to a selected value to indicate that the NAL unit includes enhancement layer video data.

45. An apparatus for processing scalable digital video data, the apparatus comprising:

a network abstraction layer (NAL) unit module that receives enhancement layer video data in a NAL unit, and receives one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data; and

a decoder that decodes the digital video data in the NAL unit based on the indication.

46. The apparatus of claim 45, wherein the NAL unit module detects one or more syntax elements in the NAL unit to determine a type of raw byte sequence payload (RBSP) data structure of the enhancement layer data in the NAL unit.

47. The apparatus of claim 45, wherein the NAL unit module detects one or more syntax elements in the NAL unit to determine whether the enhancement layer video data in the NAL unit includes intra-coded video data.

48. The apparatus of claim 45, wherein the NAL unit is a first NAL unit, wherein the NAL unit module receives base layer video data in a second NAL unit, and wherein the NAL unit module detects one or more syntax elements in at least one of the first and second NAL units to determine whether the enhancement layer video data includes any residual data relative to the base layer video data, and the decoder skips decoding of the enhancement layer video data if it is determined that the enhancement layer video data includes no residual data relative to the base layer video data.

49. The apparatus of claim 45, wherein the NAL unit is a first NAL unit, wherein the NAL unit module:

receives base layer video data in a second NAL unit;

detects one or more syntax elements in at least one of the first and second NAL units to determine whether the first NAL unit includes a sequence parameter, a picture parameter set, a slice of a reference picture or a slice data partition of a reference picture;

detects one or more syntax elements in at least one of the first and second NAL units to identify blocks within the enhancement layer video data containing non-zero transform coefficient syntax elements; and

detects one or more syntax elements in at least one of the first and second NAL units to determine whether pixel domain or transform domain addition of the enhancement layer video data with the base layer data should be used to decode the digital video data.

50. The apparatus of claim 45, wherein the NAL processing module detects one or more syntax elements in the NAL unit to determine a number of nonzero coefficients in intra-coded blocks in the enhancement layer video data with a magnitude larger than one.

51. The apparatus of claim 45, wherein the NAL processing module detects one or more syntax elements in the NAL unit to determine coded block patterns for inter-coded blocks in the enhancement layer video data.

52. The apparatus of claim 45, wherein the NAL unit is a first NAL unit, the NAL unit module including base layer video data in a second NAL unit, and wherein the enhancement layer video data is encoded to enhance a signal-to-noise ratio of the base layer video data.

53. The apparatus of claim 45, wherein the NAL unit module receives a NAL unit type parameter in the NAL unit that is set to a selected value to indicate that the NAL unit includes enhancement layer video data.

54. A processor for processing scalable digital video data, the processor being configured to:

receive enhancement layer video data in a network abstraction layer (NAL) unit;

receive one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data; and

decode the digital video data in the NAL unit based on the indication.

55. An apparatus for processing scalable digital video data, the apparatus comprising:

means for receiving enhancement layer video data in a network abstraction layer (NAL) unit;

means for receiving one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data; and

means for decoding the digital video data in the NAL unit based on the indication.

56. The apparatus of claim 55, further comprising means for detecting one or more syntax elements in the NAL unit to determine a type of raw byte sequence payload (RBSP) data structure of the enhancement layer data in the NAL unit.

57. The apparatus of claim 55, further comprising means for detecting one or more syntax elements in the NAL unit to determine whether the enhancement layer video data in the NAL unit includes intra-coded video data.

58. The apparatus of claim 55, wherein the NAL unit is a first NAL unit, the apparatus further comprising:

means for receiving base layer video data in a second NAL unit;

means for detecting one or more syntax elements in at least one of the first and second NAL units to determine whether the enhancement layer video data includes any residual data relative to the base layer video data; and

means for skipping decoding of the enhancement layer video data if it is determined that the enhancement layer video data includes no residual data relative to the base layer video data.

59. The apparatus of claim 55, wherein the NAL unit is a first NAL unit, the apparatus further comprising:

means for receiving base layer video data in a second NAL unit;

means for detecting one or more syntax elements in at least one of the first and second NAL units to determine whether the first NAL unit includes a sequence parameter, a picture parameter set, a slice of a reference picture or a slice data partition of a reference picture;

means for detecting one or more syntax elements in at least one of the first and second NAL units to identify blocks within the enhancement layer video data containing non-zero transform coefficient syntax elements; and

means for detecting one or more syntax elements in at least one of the first and second NAL units to determine whether pixel domain or transform domain addition of the enhancement layer video data with the base layer data should be used to decode the digital video data

60. The apparatus of claim 55, further comprising means for detecting one or more syntax elements in the NAL unit to determine a number of nonzero coefficients in intra-coded blocks in the enhancement layer video data with a magnitude larger than one.

61. The apparatus of claim 55, further comprising means for detecting one or more syntax elements in the NAL unit to determine coded block patterns for inter-coded blocks in the enhancement layer video data.

62. The apparatus of claim 55, wherein the NAL unit is a first NAL unit, the apparatus further comprising means for including base layer video data in a second NAL unit, and wherein the enhancement layer video data is encoded to enhance a signal-to-noise ratio of the base layer video data.

63. The apparatus of claim 55, wherein the means for receiving one or more syntax elements in the NAL unit to indicate whether the respective NAL unit includes enhancement layer video data comprises means for receiving a NAL unit type parameter in the NAL unit that is set to a selected value to indicate that the NAL unit includes enhancement layer video data.

64. A computer program product for processing of scalable digital video data comprising:

a computer-readable medium comprising codes for causing a computer to;

receive enhancement layer video data in a network abstraction (NAL) unit;

receive one or more syntax elements in the NAL unit to indicate whether the NAL unit includes enhancement layer video data; and

decode the digital video data in the NAL unit based on the indication.