User and content aware object-based data stream transmission methods and arrangements

• US 7,093,028 B1
• Filed: 12/15/1999
• Issued: 08/15/2006
• Est. Priority Date: 12/15/1999
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shao, Huai-Rong, Zhang, Ya-Qin
• Primary Examiner(s): EL HADY, NABIL M
• Application Number:
  US09/464,671
• Time in Patent Office: 2,435 Days
• Field of Search: 709/204, 709/207, 709/219, 709/200, 709/231, 709/232, 709/235, 709/236, 709/240, 709/243
• US Class Current: 709/240
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/2408 for supporting different services, e.g. a differentiated services [DiffServ] type of service
• CPC Class Codes: H04L 47/2441 relying on flow classification, e.g. using integrated services [IntServ]
• CPC Class Codes: H04L 47/326 with random discard, e.g. random early discard [RED]
• CPC Class Codes: H04L 47/50 Queue scheduling
• CPC Class Codes: H04L 47/521 Static queue service slot or fixed bandwidth allocation
• CPC Class Codes: H04L 47/564 Attaching a deadline to packets, e.g. earliest due date first
• CPC Class Codes: H04L 47/6215 Individual queue per QOS, rate or priority
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A scalable video transmission scheme is provided in which client interaction and video content itself are taken into consideration during transmission. Methods and arrangements are provided to prioritize/classify different types of information according to their importance and to packetize or otherwise arrange the prioritized information in a manner such that lower priority information may be dropped during transmission. Thus, when network congestion occurs or there is not enough network bandwidth to transmit all of the prioritized information about an object, some (e.g., lower priority) information may be dropped at the server or at an intermediate network node to reduce the bit rate. Thus, when the server transmits multiple video objects over a channel of limited bandwidth capacity, the bit rate allocated to each object can be adjusted according to several factors, such as, e.g., information importance and client interaction.

Claims

1. A method comprising:

receiving a data bitstream that includes object-based media information;

associating portions of the object-based media information with a plurality of different transmission priority levels;

selectively transmitting the portions of the object-based media information along with the associated plurality of different transmission priority levels over a network that is configured to provide differential services based at least on the plurality of different transmission priority levels; and

selectively discarding, within the network that is configured to provide differential services, one or more of the portions of the object-based media information based at least on the associated plurality of different transmission priority levels and on at least one input received from a downstream client.

2. The method as recited in claim 1, wherein the data bitstream includes object-based media information for a single object.

3. The method as recited in claim 2, wherein the single object is a video object.

4. The method as recited in claim 2, wherein the single object is an audio object.

5. The method as recited in claim 1, wherein associating portions of the object-based media information with the plurality of different transmission priority levels further includes:

placing the portions of the object-based media information in a plurality of data packets, wherein each data packet is associated with a specific transmission priority of the plurality of different transmission priority levels.

6. The method as recited in claim 5, wherein at least one of the plurality of data packets includes non-contiguous portions of data from within the data bitstream.

7. The method as recited in claim 5, wherein selectively transmitting the portions of the object-based media information over the network further includes:

causing the network to selectively halt the transmission of a first data packet carrying object-based media information that is associated with a first priority level prior to halting the transmission of a second data packet carrying object-based media information that is associated with a second priority level if the second priority level is higher than the first priority level, should a need arise while transmitting the first and second data packets.

8. The method as recited in claim 1, wherein the differential services provide different substantially guaranteed Quality of Service (QoS) transmission capabilities for different transmission priority levels.

9. The method as recited in claim 3, wherein the object-based media information includes a plurality of different types of video frame layers selected from a group that includes Intra (I) coded frame layers, Predicted (P) frame layers, Bi-directionally (B) predicted frame layers, Intra (I) coded frame enhancement layers, Predicted (P) frame enhancement layers, and Bi-directionally (B) predicted frame enhancement layers.

10. The method as recited in claim 9, wherein associating portions of the object-based media information with the plurality of different transmission priority levels further includes:

setting the transmission priority levels based at least in part on the type of video frame layer.

11. The method as recited in claim 10, wherein setting the transmission priority levels based at least in part on the type of video frame layer further includes:

causing Intra (I) coded frame layer data to have a higher transmission priority level than Predicted (P) frame layer data;

causing Predicted (P) frame layer data to have a higher transmission priority level than Bi-directionally (B) predicted frame layer data;

causing Bi-directionally (B) predicted frame layer data to have a higher transmission priority level than Intra (I) coded frame enhancement layer data;

causing Intra (I) coded frame enhancement layer data to have a higher transmission priority level than Predicted (P) frame enhancement layer data; and

causing Predicted (P) frame enhancement layer data to have a higher transmission priority level than Bi-directionally (B) predicted frame enhancement layer data.

12. The method as recited in claim 3, wherein the object-based media information further includes a plurality of different types of video object information selected from a group that includes control information, shape information, motion information and texture information.

13. The method as recited in claim 12, wherein associating portions of the object-based media information with the plurality of different transmission priority levels further includes:

setting the transmission priority levels based at least in part on the type of video object information.

14. The method as recited in claim 13, wherein setting the transmission priority levels based at least in part on the type of video object information further includes:

causing at least a portion of the control information to have a higher transmission priority level than at least a portion of the shape information.

15. The method as recited in claim 13, wherein setting the transmission priority levels based at least in part on the type of video object information further includes:

causing at least a portion of the shape information to have a higher transmission priority level than at least a portion of the motion information.

16. The method as recited in claim 13, wherein setting the transmission priority levels based at least in part on the type of video object information further includes:

causing at least a portion of the motion information to have a higher transmission priority level than at least a portion of the texture information.

17. The method as recited in claim 13, wherein setting the transmission priority levels based at least in part on the type of video object information further includes:

causing at least a portion of the texture information to have a higher transmission priority level than at least a portion of the shape information.

18. The method as recited in claim 3, wherein:

the object-based media information includes a plurality of different types of video frame layers selected from a group that includes Intra (I) coded frame layers, Predicted (P) frame layers, Bi-directionally (B) predicted frame layers, Intra (I) coded frame enhancement layers, Predicted (P) frame enhancement layers, and Bi-directionally (B) predicted flume enhancement layers;

the object-based media information further includes a plurality of different types of video object information selected from a group that includes control information, shape information, motion information and texture information; and

wherein associating portions of the object-based media information with the plurality of different transmission priority levels further includes setting the transmission priority levels based at least in part on the type of video frame layer and the type of video object information.

19. The method as recited in claim 18, wherein setting the transmission priority levels based at least in part on the type of video frame layer and the type of video object information further includes:

setting control information to a class 0 transmission priority level;

setting shape information and texture DC information of at least one Intra (I) coded frame layer to a class 1 transmission priority level;

setting texture AC information of the Intra (I) coded frame base layer to a class 2 transmission priority level;

setting shape information and motion information of at least one Predicted (P) frame layer to a class 3 transmission priority level;

setting texture information of the Predicted (P) frame layer to a class 4 transmission priority level; and

setting shape information, motion information and texture information of at least one Bi-directionally (B) predicted frame base layer to a class 5 transmission priority level, and

wherein the class 0 transmission priority level is higher than the class 1 transmission priority level, the class 1 transmission priority level is higher than the class 2 transmission priority level, the class 2 transmission priority level is higher than the class 3 transmission priority level, the class 3 transmission priority level is higher than the class 4 transmission priority level, and the class 4 transmission priority level is higher than the class 5 transmission priority level.

20. The method as recited in claim 1, further comprising:

receiving at least one down-stream preference with regard to the object-based media information; and

selectively transmitting at least one of the portions of the object-based media information over the network based on the down-stream preference.

21. The method as recited in claim 1, further comprising:

receiving at least one down-stream preference with regard to the object-based media information; and

selectively halting the transmission of at least one of the portions of the object-based media information over the network based on the down-stream preference.

22. The method as recited in claim 1, wherein the data bitstream includes MPEG-4 encoded video data.

23. The method as recited in claim 1, wherein the network is an Internet Protocol (IP) based network.

68. A computer-readable medium having computer-executable instructions for performing the steps recited in claim 1.

24. An arrangement comprising:

a server device configured to provide a data bitstream that includes object-based media information having portions of the object-based media information associated with a plurality of different transmission priority levels and that includes identifications of the associated plurality of different transmission priority levels;

at least one client device configured to receive portions of the data bitstream that includes object-media information; and

at least one communication network operatively coupled between the server device and the client device, the communication network being configured to provide selective differential services by discarding, within the network, one or more of the portions of the object-based information, based at least on the identifications of the associated plurality of different transmission priority levels of the portions of the object-based media in formation and on at least one input received from the at least one client device.

25. The arrangement as recited in claim 24, wherein the data bitstream includes object-based media information for a single object.

26. The arrangement as recited in claim 25, wherein the single object is a video object.

27. The arrangement as recited in claim 25, wherein the single object is an audio object.

28. The arrangement as recited in claim 24, wherein the server device is further configured to place the portions of the object-based media information in a plurality of data packets, wherein each data packet is associated with a specific transmission priority of the plurality of different transmission priority levels.

29. The arrangement as recited in claim 28, wherein at least one of the plurality of data packets includes non-contiguous portions of data from within the data bitstream.

30. The arrangement as recited in claim 28, wherein the communication network is further configured to selectively halt the transmission of a first data packet carrying object-based media information that is associated with a first priority level prior to halting the transmission of a second data packet carrying object-based media information that is associated with a second priority level if the second priority level is higher than the first priority level, should a need arise while transmitting the first and second data packets.

31. The arrangement as recited in claim 24, wherein the selective differential services provide different substantially guaranteed Quality of Service (QoS) transmission capabilities for different transmission priority levels.

32. The arrangement as recited in claim 26, wherein the object-based media information includes a plurality of different types of video frame layers selected from a group that includes Intra (I) coded frame layers, Predicted (P) frame layers, Bi-directionally (B) predicted frame layers, Intra (I) coded frame enhancement layers, Predicted (P) frame enhancement layers, and Bi-directionally (B) predicted frame enhancement layers.

33. The arrangement as recited in claim 32, wherein the server device is further configured to set the transmission priority levels based at least in part on the type of video frame layer.

34. The arrangement as recited in claim 33, wherein the server device is further configured to:

set Intra (I) coded frame layer data to a higher transmission priority level than Predicted (P) frame layer data;

set Predicted (P) frame layer data to a higher transmission priority level than Bi-directionally (B) predicted fame layer data;

set Bi-directionally (B) predicted frame layer data to a higher transmission priority level than Intra (I) coded frame enhancement layer data;

set Intra (I) coded frame enhancement layer data to a higher transmission priority level than Predicted (P) frame enhancement layer data; and

set Predicted (P) frame enhancement layer data to a higher transmission priority level than Bi-directionally (B) predicted frame enhancement layer data.

35. The arrangement as recited in claim 26, wherein the object-based media information further includes a plurality of different types of video object information selected from a group that includes control information, shape information, motion information and texture information.

36. The arrangement as recited in claim 35, wherein the server device is further configured to set the transmission priority levels based at least in part on the type of video object information.

37. The arrangement as recited in claim 36, wherein the server device is further configured to set at least a portion of the control information to a higher transmission priority level than at least a portion of the shape information.

38. The arrangement as recited in claim 36, wherein the server device is further configured to set at least a portion of the shape information to a higher transmission priority level than at least a portion of the motion information.

39. The arrangement as recited in claim 36, wherein the server device is further configured to set at least a portion of the motion information to a higher transmission priority level than at least a portion of the texture information.

40. The arrangement as recited in claim 36, wherein the server device is further configured to set at least a portion of the texture information to a higher transmission priority level than at least a portion of the shape information.

41. The arrangement as recited in claim 26, wherein:

the object-based media information includes a plurality of different types of video frame layers selected from a group that includes Intra (I) coded frame layers, Predicted (P) frame layers, Bi-directionally (B) predicted frame layers, Intra (I) coded frame enhancement layers, Predicted (P) frame enhancement layers, and Bi-directionally (B) predicted frame enhancement layers;

the object-based media information further includes a plurality of different types of video object information selected from a group that includes control information, shape information, motion information and texture information; and

wherein the server device is further configured to set the transmission priority levels based at least in part on the type of video frame layer and the type of video object information.

42. The arrangement as recited in claim 41, wherein the server device is further configured to;

set control information to a class 0 transmission priority level;

set shape information and texture DC information of at least one Intra (I) coded frame layer to a class 1 transmission priority level;

set texture AC information of the Intra (I) coded frame base layer to a class 2 transmission priority level;

set shape information and motion information of at least one Predicted (P) frame layer to a class 3 transmission priority level;

set texture information of the Predicted (P) frame layer to a class 4 transmission priority level; and

set shape information, motion information and texture information of at least one Bi-directionally (B) predicted frame base layer to a class 5 transmission priority level, and

where the class 0 transmission priority level is higher than the class 1 transmission priority level, the class 1 transmission priority level is higher than the class 2 transmission priority level, the class 2 transmission priority level is higher than the class 3 transmission priority level, the class 3 transmission priority level is higher than the class 4 transmission priority level, and the class 4 transmission priority level is higher than the class 5 transmission priority level.

43. The arrangement as recited in claim 24, wherein the network is further configured to:

receive at least one down-stream preference generated within the communication network or by the client device with regard to the object-based media information; and

selectively transmit at least one of the portions of the object-based media information based on the down-stream preference.

44. The arrangement as recited in claim 24, wherein the network is further configured to:

receive at least one down-stream preference generated within the communication network or by the client device with regard to the object-based media information; and

selectively halt the transmission at least one of the portions of the object-based media information based on the down-stream preference.

45. The arrangement as recited in claim 24, wherein the data bitstream includes MPEG-4 encoded video data.

46. The arrangement as recited in claim 24, wherein the network is an Internet Protocol (IP) based network.

47. A method for use in a communications node within a network, the method comprising:

receiving data at the communications node that includes object-based media information that is packetized according to different transmission priority levels, the data including indications of the different transmission priority levels; and

selectively outputting from the communications node portions of the object-based media information and discarding one or more of the portions of the object-based information based at least on the indications of the different transmission priority levels included in the received data and on at least one input received from a downstream client with regard to the object-based media information, wherein the downstream client is configured to receive the outputted portions of the object-based media information, and

wherein the communication node is configured to support differential services that provide different substantially guaranteed Quality of Service(QoS) transmission capabilities for the different transmission priority levels.

48. The method as recited in claim 47, wherein the data bitstream includes object-based media information for a single video object.

49. The method as recited in claim 47, wherein the data bitstream includes object-based media information for a single audio object.

50. The method as recited in claim 47, wherein the object-based media information includes a plurality of different types of video frame layers selected from a group that includes Intra (I) coded frame layers, Predicted (P) frame layers, Bi-directionally (B) predicted frame layers, Intra (I) coded frame enhancement layers, Predicted (P) frame enhancement layers, and Bi-directionally (B) predicted frame enhancement layers.

51. The method as recited in claim 50, wherein the received data is packetized according to different transmission priority levels based at least in part on the type of video frame layer.

52. The method as recited in claim 51, wherein, within the received data, at least one of the following statements is true:

the Intra (I) coded frame layer data has a higher transmission priority level than Predicted (P) frame layer data;

the Predicted (P) frame layer data has a higher transmission priority level than Bi-directionally (B) predicted frame layer data;

the Bi-directionally (B) predicted frame layer data has a higher transmission priority level than Intra (I) coded frame enhancement layer data;

the Intra (I) coded frame enhancement layer data has a higher transmission priority level than Predicted (P) frame enhancement layer data; and

the Predicted (P) frame enhancement layer data has a higher transmission priority level than Bi-directionally (B) predicted frame enhancement layer data.

53. The method as recited in claim 47, wherein the object-based media information further includes a plurality of different types of video object information selected from a group that includes control information, shape information, motion information and texture information.

54. The method as recited in claim 53, wherein the received data is packetized according to different transmission priority levels based at least in part on the type of video object information.

55. The method as recited in claim 54, wherein at least a portion of the control information has a higher transmission priority level than at least a portion of the shape information.

56. The method as recited in claim 54, wherein at least a portion of the shape information has a higher transmission priority level than at least a portion of the motion information.

57. The method as recited in claim 54, wherein at least a portion of the motion information has a higher transmission priority level than at least a portion of the texture information.

58. The method as recited in claim 54, wherein at least a portion of the texture information has a higher transmission priority level than at least a portion of the shape information.

59. The method as recited in claim 47, wherein:

the object-based media information includes a plurality of different types of video frame layers selected from a group that includes Intra (I) coded frame layers, Predicted (P) frame layers, Bi-directionally (B) predicted frame layers, Intra (I) coded frame enhancement layers, Predicted (P) frame enhancement layers, and Bi-directionally (B) predicted frame enhancement layers;

the object-based media information further includes a plurality of different types of video object information selected from a group that includes control information, shape information, motion information and texture information; and

wherein the received data is packetized according to different transmission priority levels based at least in part on the type of video frame layer and the type of video object information.

60. The method as recited in claim 59, wherein:

control information has a class 0 transmission priority level;

shape information and texture DC information of at least one Intra (I) coded frame layer each have a class 1 transmission priority level;

texture AC information of the Intra (I) coded frame base layer has a class 2 transmission priority level;

shape information and motion information of at least one Predicted (P) frame layer each have a class 3 transmission priority level;

texture information of the Predicted (P) frame layer has a class 4 transmission priority level; and

shape information, motion information and texture information of at least one Bi-directionally (B) predicted frame base layer each have a class 5 transmission priority level, and

wherein the class 0 transmission priority level is higher than the class 1 transmission priority level, the class 1 transmission priority level is higher than the class 2 transmission priority level, the class 2 transmission priority level is higher than the class 3 transmission priority level, the class 3 transmission priority level is higher than the class 4 transmission priority level, and the class 4 transmission priority level is higher than the class 5 transmission priority level.

61. The method as recited in claim 47, further comprising:

receiving at least one down-stream preference with regard to the object-based media information; and

selectively outputting at least one of the portions of the object-based media information based on the downstream preference.

62. The method as recited in claim 47, wherein the received data includes MPEG-4 encoded video data.

63. The method as recited in claim 47, wherein the received data includes Internet Protocol (IP) data.

69. A computer-readable medium having computer-executable instructions for performing the steps recited in claim 47.

64. A system comprising:

at least one client device configured to receive prioritized video object-based data packets and output control requests relating to a video object;

at least one server device configured to output prioritized object-based data packets representing the video object, the prioritized object-based data packets being prioritized based at least on part on the type of data as selected from a group comprising control data, shape data, motion data, and texture data; and

at least one video transmission agent (ETA) that is part of a network linking the at least one client device to the at least one server device, the VTA coupled to receive the prioritized object-based data packets from the server device and the control requests from the client device, the VTA adapted to selectively output at least a portion of the received prioritized object-based data packets to the client device based on the prioritization and in response to the control requests.

65. The system as recited in claim 64, further comprising:

a network operatively coupled between the server device and the client device, and wherein the video transmission agent (VTA) is operatively configured within the network.

66. The system as recited in claim 65, wherein the network is further configured to provide differential services to the prioritized object-based data packets, such that prioritized object-based data packets having lower priority levels are selectively dropped should the network become congested.

67. A computer-readable medium having a data structure, comprising:

a first field containing identifying data associated with a portion of a data bitstream that represents a video object;

at least one second field that is derived from the first field and includes data representing object-based video information for the video object that has been classified as having a specific transmission priority level based on at least one type of object-based video information selected from a group comprising control information, shape information, motion information, and texture information; and

a third field comprising a network packet header and containing data identifying the specific transmission priority level of the data in the at least one second field;

wherein a network node is enabled to selectively filter packets based on the specific transmission priority level identified in the third field and responsive to control requests from at least one downstream client that is receiving the video object.

70. A method comprising:

receiving a data bitstream that includes object-based media information;

associating portions of the object-based media information with a plurality of different transmission priority levels based, at least in part, on whether a given portion of the object-based media information comprises shape information or texture information;

wherein shape information is associated with a higher transmission priority level than texture information within a single frame; and

selectively transmitting the portions of the object-based media information over a network that is configured to provide differential services based at least on the plurality of different transmission priority levels; and

selectively discarding, within the network that is configured to provide differential services, one or more of the portions of the object-based media information based at least on the plurality of different transmission priority levels and on at least one input received from a downstream client.