System and method for management of scalability information in scalable video and audio coding systems using control messages

• US 8,761,263 B2
• Filed: 03/27/2007
• Issued: 06/24/2014
• Est. Priority Date: 03/27/2006
• Status: Active Grant

• Current Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Original Assignee: Vidyo Incorporated (Enghouse Systems Limited)
• Inventors: Eleftheriadis, Alexandros
• Primary Examiner(s): Homayounmehr, Farid
• Assistant Examiner(s): SHOLEMAN, ABU S
• Application Number:
  US11/691,621
• Publication Number:
  US 20070291837A1
• Time in Patent Office: 2,646 Days
• Field of Search: 375/240.2, 375/240.26, 375/242.01, 375/242.02, 375/242.04, 375/242.05, 375/242.06, 375/242.08, 375/242.28, 370/389, 370/394
• US Class Current: 375/240.2
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4621 Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/643 Communication protocols network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP], H04L65/65
• CPC Class Codes: H04N 21/8451 using Advanced Video Coding [AVC]

Abstract

Systems and methods are provided for communicating timely information related to the scalability layer structure of signals received by decoders and other components in a video and/or audio communication system. For a communication system, which uses the Standard H.264 SVC coding format, standard SSEI messages are modified or supplemented to include the ability to signal scalability layer structure information and changes thereof. Recipients can use the signal scalability layer information to properly process or decode received signals.

Claims

1. A digital media communication system comprising:

a media server that transmits media encoded in a scalable media coding format;

at least one receiver; and

an electronic communication network linking the transmitting media server and the at least one receiver,

wherein the media server is configured to transmit to the at least one receiver scalability information pertaining to the transmitted media including initializing scalability information and changes to the scalability information with time, and wherein the transmitted changes to the scalability information are encoded in a differential form and the changes to the scalability information indicate that at least one of;

a scalability layer is added to the bitstream from that time forward,

a scalability layer is removed from the bitstream from that time forward, or

the scalability information for at least one layer present in the bitstream is replaced with

the scalability information contained in a Supplemental Enhancement Information (SEI) message that contains the changes to the scalability information from that time forward, and

wherein any scalability information for a layer which is not explicitly coded in the changes to the scalability information is assumed to retain the scalability information'"'"'s previous value.

2. The system of claim 1, wherein the scalable media coding format is H.264 Scalable Video Coding (SVC), the initializing scalability infounation is the SVC scalability_info Supplemental Enhancement Information (SEI) message, and wherein the transmitted changes to the scalability information are applied cumulatively at the receiver.

3. The system of claim 2, wherein the changes to the scalability information further include a scalability_info_add SEI message indicating that at least one scalability layer is added to the bitstream from that time forward.

4. The system of claim 2, wherein the changes to the scalability information further include a scalability_info_replace SEI message indicating that the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in the said scalability_info_replace SEI message from that time forward, and wherein any scalability information for a layer which is not explicitly coded in the scalability_info_replace SEI message is assumed to retain the scalability information'"'"'s previous value.

5. The system of claim 2, wherein the changes to the scalability information further include a scalability_info_del SEI message indicating that at least one layer is removed from the bitstream from that time forward.

6. The system of claim 2, wherein the changes to the scalability information appear in any access unit of the coded video signal.

7. The system of claim 2, wherein the receiver is a media gateway.

8. The system of claim 1, wherein the scalable media coding format is H.264 SVC, the initializing scalability information is the SVC scalability_info SEI message, and wherein the changes to the scalability information are applied at the receiver individually using the most recent scalability_info SEI message as a reference.

9. The system of claim 8, wherein the changes to the scalability information further include a scalability_info_layers_notpresent SEI message indicating that at least one scalability layer indicated in the initializing scalability information is not present in the bitstream from that time forward.

10. The system of claim 8, wherein the changes to the scalability information further include a scalability_info _dependency_change SEI message indicating that the layer dependency information for at least one layer indicated in the most recent initializing scalability information is replaced with the layer dependency information contained in the said scalability_—

info_dependency_change SEI message from that time forward.

11. The system of claim 8, wherein the changes to the scalability infoimation appear in any access unit of the coded video signal.

12. The system of claim 8, wherein the receiver is a media gateway.

13. A system comprising:

a digital media decoding device configured to process coded media data, the coded media data comprising;

scalably coded media data;

initializing scalability information data; and

changes to initializing scalability information data with time,

wherein the scalability information data is used to inform the decoding device about the presence and dependency structure of the layers contained in the coded media data, and wherein the changes to initializing scalability information data are encoded in a differential form and the changes to the scalability information indicate that at least one of;

a scalability layer is added to the bitstream from that time forward,

a scalability layer is removed from the bitstream from that time forward, or

the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in a Supplemental Enhancement Information (SEI) message that contains the changes to the scalability information from that time forward, and

wherein any scalability information for a layer which is not explicitly coded in the changes to the scalability information is assumed to retain the scalability information'"'"'s previous value.

14. The system of claim 13, wherein the scalable media data is coded according to H.264 Scalable Video Coding (SVC), the initializing scalability information is the SVC scalability_info Supplemental Enhancement Information (SEI) message, and wherein the changes to the scalability information are applied at the decoding device cumulatively.

15. The system of claim 14, wherein the changes to the scalability information further include a scalability_info_add SEI message indicating that at least one scalability layer is added to the bitstream from that time forward.

16. The system of claim 14, wherein the changes to the scalability information further include a scalability_info_replace SEI message indicating that the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in the said scalability_info_replace SEI from that time forward, and wherein any scalability information for a layer which is not explicitly coded in the scalability_info_replace SEI message is assumed to retain the scalability information'"'"'s previous value.

17. The system of claim 14, wherein the changes to the scalability information further include a scalability_info_del SEI message indicating that at least one layer is removed from the bitstream from that time forward.

18. The system of claim 14, wherein the changes to the scalability information may appear in any access unit of the coded video signal.

19. The system of claim 13, wherein the scalable media data is coded according to H.264 SVC, the initializing scalability information is the SVC scalability_info SEI message, and wherein the changes to the scalability information are applied at the decoding device individually using the most recent scalability_info SEI message as a reference.

20. The system of claim 19, wherein the changes to the scalability information further include a scalability_info_layers_notpresent SEI message indicating that one or more of the scalability layers indicated in the initializing scalability information are not present in the bitstream from that time forward.

21. The system of claim 19, wherein the changes to the scalability information further include a scalability_info_dependency_change SEI message indicating that the layer dependency information for at least one layer indicated in the most recent initializing scalability information is replaced with the layer dependency information contained in the said scalability_info_dependency_change SEI message from that time forward.

22. The system of claim 19, wherein the changes to the scalability information may appear in any access unit of the coded video signal.

23. A method for digital media communication over an electronic communication network linking at least one receiver to a transmitting media server which transmits media encoded in a scalable media coding format, the method comprising:

transmitting from the media server to the at least one receiver scalability information pertaining to the transmitted media including initializing scalability information and changes to the scalability information with time,

wherein the transmitted changes to the scalability information are encoded in a differential form and the changes to the scalability information indicate that at least one of;

a scalability layer is added to the bitstream from that time forward,

a scalability layer is removed from the bitstream from that time forward, or

the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in a Supplemental Enhancement Information (SEI) message that contains the changes to the scalability information from that time forward, and

wherein any scalability information for a layer which is not explicitly coded in the changes to the scalability information is assumed to retain the scalability information'"'"'s previous value.

24. The method of claim 23, wherein the scalable media coding format is H.264 Scalable Video Coding (SVC), the initializing scalability information is the SVC scalability_info Supplemental Enhancement Information (SEI) message, the method further comprising:

applying the transmitted changes to the scalability information cumulatively at the receiver.

25. The method of claim 24, further comprising including in the changes to the scalability information a scalability_info_add SEI message indicating that at least one scalability layer is added to the bitstream from that time forward.

26. The method of claim 24, further comprising including in the changes to the scalability information a scalability_info_replace SEI message indicating that the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in the said scalability_info_replace SEI message from that time forward,

wherein any scalability information for a layer which is not explicitly coded in the scalability_info_replace SEI message is assumed to retain the scalability information'"'"'s previous value.

27. The method of claim 24, further comprising including in the changes to the scalability information a scalability_info_del SEI message indicating that at least one layer is removed from the bitstream from that time forward.

28. The method of claim 24, wherein the changes to the scalability information appear in any access unit of the coded video signal.

29. The method of claim 24, wherein the receiver is a media gateway.

30. The method of claim 23, wherein the scalable media coding format is H.264 SVC, the initializing scalability information is the SVC scalability_info SEI message, the method further comprising applying the changes to the scalability information at the receiver individually using the most recent scalability_info SEI message as a reference.

31. The method of claim 30, further comprising including in the changes to the scalability information a scalability_info_layers_notpresent SEI message indicating that at least one scalability layer indicated in the initializing scalability information is not present in the bitstream from that time forward.

32. The method of claim 30, further comprising including in the changes to the scalability information a scalability_info _dependency_change SEI message indicating that the layer dependency information for at least one layer indicated in the most recent initializing scalability information is replaced with the layer dependency information contained in the said scalability_info_dependency_change SEI message from that time forward.

33. The method of claim 30, wherein the changes to the scalability information appear in any access unit of the coded video signal.

34. The method of claim 30, wherein the receiver is a media gateway.

35. A method for decoding digital media, wherein the digital media includes scalably coded media data and scalability information data including initializing scalability information data and changes to initializing scalability information data with time encoded in a differential form, the method comprising:

at a decoder, using the scalability information data to inform the decoder about the presence and dependency structure of the layers contained in the coded media data,

wherein the changes to initializing scalability information data indicates that at least one of;

a scalability layer is added to the bitstream from that time forward,

a scalability layer is removed from the bitstream from that time forward, or

the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in a Supplemental Enhancement Information (SEI) message that contains the changes to initializing scalability information data from that time forward, and

wherein any scalability information data for a layer which is not explicitly coded in the changes to initializing scalability information data is assumed to retain the scalability information data'"'"'s previous value.

36. The method of claim 35, wherein the scalable media data is coded according to H.264 Scalable Video Coding (SVC), the initializing scalability information is the SVC scalability_info Supplemental Enhancement Information (SEI) message, the method further comprising applying the changes to the scalability information at the decoder cumulatively.

37. The method of claim 36, wherein the changes to the scalability information further include a scalability_info_add SEI message indicating that at least one scalability layer is added to the bitstream from that time forward.

38. The method of claim 36, wherein the changes to the scalability information further include a scalability_info_replace SEI message indicating that the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in the said scalability_info_replace SEI from that time forward, the method further comprising:

assuming that any scalability information for a layer which is not explicitly coded in the scalability_info_replace SEI message retains the scalability information'"'"'s previous value.

39. The method of claim 36, wherein the changes to the scalability information further include

a scalability_info_del SEI message indicating that at least one layer is removed from the bitstream from that time forward.

40. The method of claim 36, wherein the changes to the scalability information may appear in any access unit of the coded video signal.

41. The method of claim 35, wherein the scalable media data is coded according to H.264 SVC, and the initializing scalability information is the SVC scalability _info SEI message, the method further comprising:

applying the changes to the scalability information at the decoder individually and using the most recent scalability_info SEI message as a reference.

42. The method of claim 41, wherein the changes to the scalability information further include a scalability_info_layers_notpresent SEI message indicating that one or more of the scalability layers indicated in the initializing scalability information are not present in the bitstream from that time forward.

43. The method of claim 41, wherein the changes to the scalability information further include a scalability_info _dependency_change SEI message indicating that the layer dependency information for at least one layer indicated in the most recent initializing scalability information is replaced with the layer dependency information contained in the said scalability_info _dependency_change SEI message from that time forward.

44. The method of claim 41, wherein the changes to the scalability information may appear in any access unit of the coded video signal.

45. A non-transitory computer readable medium comprising an executable set of instructions to direct a processor to:

transmit from a media server to at least one receiver scalability information pertaining to transmitted media including initializing scalability information and changes to the scalability information with time,

wherein an electronic communication network links the at least one receiver to the transmitting media server which transmits media encoded in a scalable media coding format,

wherein the transmitted changes to the scalability information are encoded in a differential form and the changes to the scalability information indicate that at least one of;

a scalability layer is added to the bitstream from that time forward,

a scalability layer is removed from the bitstream from that time forward, or

the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in a Supplemental Enhancement Information (SEI) message that contains the changes to the scalability information from that time forward, and

wherein any scalability information for a layer which is not explicitly coded in the changes to the scalability information is assumed to retain the scalability information'"'"'s previous value.

46. The non-transitory computer readable medium of claim 45, wherein the scalable media coding format is H.264 Scalable Video Coding (SVC), the initializing scalability information is the SVC scalability_info Supplemental Enhancement Information (SEI) message, further comprising executable instructions to direct the processor to:

apply the transmitted changes to the scalability information cumulatively at the receiver.

47. The non-transitory computer readable medium of claim 46, further comprising executable instructions to direct the processor to include in the changes to the scalability information a scalability_info_add SEI message indicating that at least one scalability layer is added to the bitstream from that time forward.

48. The non-transitory computer readable medium of claim 46, further comprising executable instructions to direct the processor to include in the changes to the scalability information a scalability_info_replace SEI message indicating that the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in the said scalability_—

info_replace SEI message from that time forward,

wherein any scalability information for a layer which is not explicitly coded in the scalability_info_replace SEI message is assumed to retain the scalability information'"'"'s previous value.

49. The non-transitory computer readable medium of claim 46, further comprising executable instructions to direct the processor to include in the changes to the scalability information a scalability_info_del SEI message indicating that at least one layer is removed from the bitstream from that time forward.

50. The non-transitory computer readable medium of claim 46, wherein the changes to the scalability information appear in any access unit of the coded video signal.

51. The non-transitory computer readable medium of claim 46, wherein the receiver is a media gateway.

52. The non-transitory computer readable medium of claim 45, wherein the scalable media coding format is H.264 SVC, the initializing scalability information is the SVC scalability_info SEI message, further comprising executable instructions to direct the processor to apply the changes to the scalability information at the receiver individually using the most recent scalability_info SEI message as a reference.

53. The non-transitory computer readable medium of claim 52, further comprising executable instructions to direct the processor to include in the changes to the scalability information a scalability_info_layers_notpresent SEI message indicating that at least one scalability layer indicated in the initializing scalability information is not present in the bitstream from that time forward.

54. The non-transitory computer readable medium of claim 52, further comprising executable instructions to direct the processor to include in the changes to the scalability information a scalability_info_dependency_change SEI message indicating that the layer dependency information for at least one layer indicated in the most recent initializing scalability information is replaced with the layer dependency information contained in the said scalability_info_dependency_change SEI message from that time forward.

55. The non-transitory computer readable medium of claim 52, wherein the changes to the scalability information appear in any access unit of the coded video signal.

56. The non-transitory computer readable medium of claim 52, wherein the receiver is a media gateway.

57. A non-transitory computer readable medium comprising an executable set of instructions to direct a processor to:

use scalability information data to inform a decoder about a presence and a dependency structure of layers contained in coded digital media data,

wherein the coded digital media data includes scalably coded media data and the scalability information data, which includes initializing scalability information data and changes to initializing scalability information data with time encoded in a differential form,

wherein the changes to initializing scalability information data indicates that at least one of;

a scalability layer is added to the bitstream from that time forward,

a scalability layer is removed from the bitstream from that time forward, or

the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in a Supplemental Enhancement Information (SEI) message that contains the changes to initializing scalability information data from that time forward, and

wherein any scalability information data for a layer which is not explicitly coded in the changes to initializing scalability information data is assumed to retain the scalability information data'"'"'s previous value.

58. The non-transitory computer readable medium of claim 57, wherein the scalable media data is coded according to H.264 Scalable Video Coding (SVC), the initializing scalability information is the SVC scalability_info Supplemental Enhancement Information (SEI) message, further comprising executable instructions to direct the processor to apply the changes to the scalability information at the decoder cumulatively.

59. The non-transitory computer readable medium of claim 58, wherein the changes to the scalability information further include a scalability_info_add SEI message indicating that at least one scalability layer is added to the bitstream from that time forward.

60. The non-transitory computer readable medium of claim 58, wherein the changes to the scalability information further include a scalability_info_replace SEI message indicating that the scalability information for at least one layer present in the bitstream is replaced with the scalability information contained in the said scalability_info_replace SEI from that time forward,

further comprising executable instructions to direct the processor to:

assume that any scalability information for a layer which is not explicitly coded in the scalability_info_replace SEI message retains the scalability information'"'"'s previous value.

61. The non-transitory computer readable medium of claim 58, wherein the changes to the scalability information further include

a scalability_info_del SEI message indicating that at least one layer is removed from the bitstream from that time forward.

62. The non-transitory computer readable medium of claim 58, wherein the changes to the scalability information may appear in any access unit of the coded video signal.

63. The non-transitory computer readable medium of claim 57, wherein the scalable media data is coded according to H.264 SVC, and the initializing scalability information is the SVC scalability_info SEI message, further comprising executable instructions to direct the processor to:

applying the changes to the scalability information at the decoder individually and using the most recent scalability_info SEI message as a reference.

64. The non-transitory computer readable medium of claim 63, wherein the changes to the scalability information further include a scalability_info_layers_notpresent SEI message indicating that one or more of the scalability layers indicated in the initializing scalability information are not present in the bitstream from that time forward.

65. The non-transitory computer readable medium of claim 63, wherein the changes to the scalability information further include a scalability_info_dependency_change SEI message indicating that the layer dependency information for at least one layer indicated in the most recent initializing scalability information is replaced with the layer dependency information contained in the said scalability_info_dependency_change SEI message from that time forward.

66. The non-transitory computer readable medium of claim 63, wherein the changes to the scalability information may appear in any access unit of the coded video signal.