Apparatus and method for streaming multimedia data

• US 20050076136A1
• Filed: 08/25/2003
• Published: 04/07/2005
• Est. Priority Date: 09/17/2002
• Status: Abandoned Application

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Mi-Young, Lee, Sang-Jo, Kim, Sang-Wook, Cho, Dae-Sung
• Application Number:
  US10/646,831
• Publication Number:
  US 20050076136A1
• Time in Patent Office: Days
• US Class Current: 709/231
• CPC Class Codes: H04L 47/24 Traffic characterised by specific attributes, e.g. priority or QoS
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/80 Responding to QoS

Abstract

A multimedia streaming apparatus and method by which multimedia data is adaptively transmitted according to the conditions of a network are provided. The multimedia streaming apparatus streams multimedia data corresponding to a predetermined QoS level in response to a parsing result of metadata corresponding to multimedia data intended to be provided and information on a network bandwidth measured by a client which receives the multimedia data. Accordingly, adaptive multimedia streaming corresponding to the bandwidth change of the network can be performed without imposing an additional burden on a server.

Claims

1. A multimedia streaming apparatus comprising:

a multimedia streaming server which streams multimedia data corresponding to a predetermined quality of service (QoS) level in response to a parsing result of metadata corresponding to multimedia data intended to be provided for service and network bandwidth information which is input form the outside; and

a multimedia streaming client which measures the bandwidth of a network to which the server is connected, by using a time when multimedia data is received and information on the size of the multimedia data, and transmits the measured bandwidth information to the server.

2. The multimedia streaming apparatus of claim 1, wherein the multimedia streaming server comprises:

a data storage unit which stores multimedia data desired to be provided for service and metadata related to the multimedia data;

a metadata parsing unit which parses the metadata and outputs the parsing result in the form of a descriptor;

a message receiving unit which receives the network bandwidth information from a client;

a quality of service (QoS) processing unit which selects a QoS level available for service in response to the descriptor information and the network bandwidth information, and extracts multimedia data corresponding to the selected QoS level, from the data storage unit;

a buffer which stores the extracted data;

a packet generation unit which packetizes the data stored in the buffer; and

a packet transmission unit which transmits the data stored in the buffer to the client in each predetermined time interval.

3. The multimedia streaming apparatus of claim 2, wherein the QoS processing unit comprises:

a service level selection unit which compares a target bit rate for each QoS level with the bandwidth and selects a predetermined QoS level; and

a frame selection unit which extracts frames corresponding to the QoS level from the multimedia data stored in the data storage unit and stores the extracted frames in the buffer.

4. The multimedia streaming apparatus of claim 2, wherein the buffer comprises:

a packet storage buffer which stores the packet; and

a packet transmission buffer which transmits the packet.

5. The multimedia streaming apparatus of claim 2, wherein the multimedia data has any one form of audio data, moving picture data, still picture data, text data, and graphic data.

6. The multimedia streaming apparatus of claim 2, wherein the multimedia data is formed with a bit stream which has any one of a spatial scalable function, a quality scalable function, a temporal scalable function, and fine grain scalable (FGS) function.

7. The multimedia streaming apparatus of claim 2, wherein the metadata is defined based on an extensible markup language (XML).

8. The multimedia streaming apparatus of claim 2, wherein the metadata has a hierarchical structure in the form of a tree having the multimedia data and streaming-related information.

9. The multimedia streaming apparatus of claim 8, wherein the metadata comprises:

a STREAMING HINT node which specifies the control type of the metadata and the type of the hierarchical structure of a node;

a HEADER GROUP HINT node which is connected to the STREAMING HINT node as a subordinate node of the STREAMING HINT node and contains header information of the multimedia data;

at least one or more SEGMENT GROUP HINT nodes, each of which is connected to the STREAMING HINT node as a subordinate node of the STREAMING HINT node and contains segment information when the multimedia is divided into segments of a predetermined time interval;

at least one or more FRAME HEADER HINT nodes, each of which is connected to the HEADER GROUP HINT node as a subordinate node of the HEADER GROUP HINT node, and contains an attribute value indicating unique information of each node;

at least one or more MEDIA SEGMENT HINT nodes, each of which is connected to the SEGMENT GROUP HINT node as a subordinate node of the SEGMENT GROUP HINT node, and contains attribute information on each QoS level; and

at least one or more MEDIA FRAME HINT nodes, each of which is connected to the MEDIA SEGMENT HINT node as a subordinate node of the MEDIA SEGMENT HINT node and contains multimedia frame information to be actually transmitted.

10. The multimedia streaming apparatus of claim 9, wherein the STREAMING HINT node comprises:

a target bit rate adjuster which adjusts a transmission bit rate to the change of the network bandwidth;

a target quality adjuster which adjusts QoS of multimedia data to be provided for service;

a target complexity adjuster which supports differentiated services according to resource complexity of the client;

a target profile adjuster which supports differentiated services according to the compression format of the multimedia data;

a target speed adjuster which adjusts a service speed according to a reproduction speed adjusting request from the client;

a target direction adjuster which adjusts a service direction according to a reproduction direction adjusting request from the client; and

a target device adjuster which supports differentiated services according to the type of the client terminal.

11. The multimedia streaming apparatus of claim 9, wherein the HEADER GROUP HINT node comprises:

a stream identifier which distinguishes each multimedia stream when a plurality of multimedia objects are provided at the same time for service;

a stream type identifier which distinguishes the type of the multimedia data;

a scalable function identifier which distinguishes the type of a scalable function for the multimedia data;

a source location identifier which indicates location information of the multimedia data stored in the data storage unit;

a frame rate identifier which indicates the frame rate of the multimedia data; and

an average bit rate identifier which indicates the average bit rate of the multimedia data.

12. The multimedia streaming apparatus of claim 9, wherein the HEADER GROUP HINT node contains the same number of FRAME HEADER HINT nodes as the number of multimedia objects to be provided for service.

13. The multimedia streaming apparatus of claim 9, wherein the SEGMENT GROUP HINT node contains the same number of the MEDIA SEGMENT HINT nodes as the number of the QoS levels.

14. The multimedia streaming apparatus of claim 9, wherein the MEDIA SEGMENT HINT node contains the same number of MEDIA FRAME HINT nodes as the number of entire frames to be provided for service at each QoS level.

15. The multimedia streaming apparatus of claim 9, wherein the MEDIA FRAME HINT node comprises:

a stream identifier which distinguishes each multimedia stream when a plurality of multimedia objects are provided at the same time for service;

a decoding/reproduction time identifier which indicates decoding time information and reproduction time information of a frame;

a coding type identifier which distinguishes frames into I frames, P frames, and B frames according to a method a frame is referred to when the frame is encoded;

a frame offset identifier which indicates location information of each frame of the multimedia data stored in the data storage unit;

a frame length identifier which indicates the size of a corresponding frame; and

a frame number identifier which indicates the number of a corresponding frame.

16. The multimedia streaming apparatus of claim 9, wherein the metadata is broken down into independent-type metadata and dependent-type metadata according to an attribute of the MEDIA SEGMENT HINT node.

17. The multimedia streaming apparatus of claim 16, wherein each node of the independent-type metadata contains frame information of multimedia data corresponding to a service level without reference to an upper node or reuse.

18. The multimedia streaming apparatus of claim 16, wherein each node of the dependent-type metadata refers to an upper node for information shared at a plurality of QoS levels and specifies only additional information in a lower node.

19. The multimedia streaming apparatus of claim 2, wherein if the number of the QoS levels increases, the frame drop rate of the multimedia streaming server gradually decreases and the average bit rate and average peak signal to noise ratio (PSNR) value gradually increase.

20. The multimedia streaming apparatus of claim 1, wherein the multimedia streaming client comprises:

a packet receiving unit which receives the multimedia data from the server;

a buffer which stores the received multimedia data;

a multimedia decoder which reproduces the data stored in the buffer;

a bandwidth measuring unit which measures a network bandwidth by using the time when the multimedia data is received in the packet receiving unit and the size information of the data; and

a message transmission unit which transmits the measured network bandwidth to the server so that the transmission rate of the multimedia data transmitted from the server is adjusted to the network bandwidth.

21. The multimedia streaming apparatus of claim 20, wherein the packet receiving unit distinguishes the first packet and the last packet of each packet group by referring to the packet number of the received multimedia data.

22. The multimedia streaming apparatus of claim 21, wherein assuming that a time when the first packet is received is t1, a time when the last packet is received is t2, and the size of the packet group data is Sp, the network bandwidth is obtained by the following equation:

$E_{\mathrm{BW}}<br><br>\left(\mathrm{Bandwidth}\right)=\frac{S_p\times <br><br>8\times <br><br>1000}{t_2-t_1}$

23. The multimedia streaming apparatus of claim 20, wherein the bandwidth measuring unit feeds the network bandwidth information back to the server through the message transmission unit whenever the network bandwidth varies.

24. A multimedia streaming server comprising:

a data storage unit which stores multimedia data desired to be provided for service and metadata related to the multimedia data;

a metadata parsing unit which parses the metadata and outputs the parsing result in the form of a descriptor;

a message receiving unit which receives network bandwidth information from a client connected to a network;

a quality of service (QoS) processing unit which selects a QoS level available for service in response to the descriptor information and the network bandwidth information, and extracts multimedia data corresponding to the selected QoS level, from the data storage unit;

a buffer which stores the extracted data;

a packet generation unit which packetizes the data stored in the buffer; and

a packet transmission unit which transmits the packet data to the client in each predetermined time interval.

25. The multimedia streaming server of claim 24, wherein the QoS processing unit comprises:

a service level selection unit which compares a target bit rate for each QoS level with the bandwidth and selects a predetermined QoS level; and

a frame selection unit which extracts frames corresponding to the QoS level from the multimedia data stored in the data storage unit and stores the extracted frames in the buffer.

26. The multimedia streaming server of claim 24, wherein the buffer comprises:

a packet storage buffer which stores the packet data generated by the packet generation unit; and

a packet transmission buffer which transmits the packet data.

27. The multimedia streaming server of claim 24, wherein the multimedia data has any one form of audio data, moving picture data, still picture data, text data, and graphic data.

28. The multimedia streaming server of claim 24, wherein the multimedia data is formed with a bit stream which has any one of a spatial scalable function, a quality scalable function, a temporal scalable function, and fine grain scalable (FGS) function.

29. The multimedia streaming server of claim 24, wherein the metadata is defined based on an extensible markup language (XML).

30. The multimedia streaming server of claim 24, wherein the metadata has a hierarchical structure in the form of a tree having the multimedia data and streaming-related information.

31. The multimedia streaming server of claim 30, wherein the metadata comprises:

a STREAMING HINT node which specifies the control type of the metadata and the type of the hierarchical structure of a node;

a HEADER GROUP HINT node which is connected to the STREAMING HINT node as a subordinate node of the STREAMING HINT node and contains header information of the multimedia data;

at least one or more SEGMENT GROUP HINT nodes, each of which is connected to the STREAMING HINT node as a subordinate node of the STREAMING HINT node and contains segment information when the multimedia is divided into segments of a predetermined time interval;

at least one or more FRAME HEADER HINT nodes, each of which is connected to the HEADER GROUP HINT node as a subordinate node of the HEADER GROUP HINT node, and contains an attribute value indicating unique information of each node;

at least one or more MEDIA SEGMENT HINT nodes, each of which is connected to the SEGMENT GROUP HINT node as a subordinate node of the SEGMENT GROUP HINT node, and contains attribute information on each QoS level; and

at least one or more MEDIA FRAME HINT nodes, each of which is connected to the MEDIA SEGMENT HINT node as a subordinate node of the MEDIA SEGMENT HINT node and contains multimedia frame information to be actually transmitted.

32. The multimedia streaming server of claim 31, wherein the STREAMING HINT node comprises:

a target bit rate adjuster which adjusts a transmission bit rate to the change of the network bandwidth;

a target quality adjuster which adjusts QoS of multimedia data to be provided for service;

a target complexity adjuster which supports differentiated services according to resource complexity of the client;

a target profile adjuster which supports differentiated services according to the compression format of the multimedia data;

a target speed adjuster which adjusts a service speed according to a reproduction speed adjusting request from the client;

a target direction adjuster which adjusts a service direction according to a reproduction direction adjusting request from the client; and

a target device adjuster which supports differentiated services according to the type of the client terminal.

33. The multimedia streaming server of claim 31, wherein the HEADER GROUP HINT node comprises:

a stream identifier which distinguishes each multimedia stream when a plurality of multimedia objects are provided at the same time for service;

a stream type identifier which distinguishes the type of the multimedia data;

a scalable function identifier which distinguishes the type of a scalable function for the multimedia data;

a source location identifier which indicates location information of the multimedia data stored in the data storage unit;

a frame rate identifier which indicates the frame rate of the multimedia data; and

an average bit rate identifier which indicates the average bit rate of the multimedia data.

34. The multimedia streaming server of claim 31, wherein the HEADER GROUP HINT node contains the same number of FRAME HEADER HINT nodes as the number of multimedia objects to be provided for service.

35. The multimedia streaming server of claim 31, wherein the SEGMENT GROUP HINT node contains the same number of the MEDIA SEGMENT HINT nodes as the number of the QoS levels.

36. The multimedia streaming server of claim 31, wherein the MEDIA SEGMENT HINT node contains the same number of MEDIA FRAME HINT nodes as the number of entire frames to be provided for service at each QoS level.

37. The multimedia streaming server of claim 31, wherein the MEDIA FRAME HINT node comprises:

a stream identifier which distinguishes each multimedia stream when a plurality of multimedia objects are provided at the same time for service;

a decoding/reproduction time identifier which indicates decoding time information and reproduction time information of a frame;

a coding type identifier which distinguishes frames into I frames, P frames, and B frames according to a method a frame is referred to when the frame is encoded;

a frame offset identifier which indicates location information of each frame of the multimedia data stored in the data storage unit;

a frame length identifier which indicates the size of a corresponding frame; and

a frame number identifier which indicates the number of a corresponding frame.

38. The multimedia streaming server of claim 31, wherein the metadata is broken down into independent-type metadata and dependent-type metadata according to an attribute of the MEDIA SEGMENT HINT node.

39. The multimedia streaming server of claim 38, wherein each node of the independent-type metadata contains frame information of multimedia data corresponding to a service level without reference to an upper node or reuse.

40. The multimedia streaming a server of claim 38, wherein each node of the dependent-type metadata refers to an upper node for information shared at a plurality of QoS levels and specifies only additional information in a lower node.

41. The multimedia streaming server of claim 24, wherein if the number of the QoS levels increases, the frame drop rate of the multimedia streaming server gradually decreases and the average bit rate and average peak signal to noise ratio (PSNR) value gradually increase.

42. A multimedia streaming client comprising:

a packet receiving unit which receives the multimedia data from a server connected to a network;

a buffer which stores the received multimedia data;

a multimedia decoder which reproduces the data stored in the buffer;

a bandwidth measuring unit which measures a network bandwidth by using the time when the multimedia data is received in the packet receiving unit and the size information of the data; and

a message transmission unit which transmits the measured network bandwidth to the server so that the transmission rate of the multimedia data transmitted from the server is adjusted to the network bandwidth.

43. The multimedia streaming client of claim 42, wherein the packet receiving unit distinguishes the first packet and the last packet of each packet group by referring to the packet number of the received multimedia data.

44. The multimedia streaming client of claim 43, wherein assuming that a time when the first packet is received is t1, a time when the last packet is received is t2, and the size of the packet group data is Sp, the network bandwidth is obtained by the following equation:

$E_{\mathrm{BW}}<br><br>\left(\mathrm{Bandwidth}\right)=\frac{S_p\times <br><br>8\times <br><br>1000}{t_2-t_1}$

45. The multimedia streaming client of claim 42, wherein the bandwidth measuring unit feeds the network bandwidth information back to the server through the message transmission unit whenever the network bandwidth varies.

46. A multimedia streaming method to be performed between a server and a client that are connected through a network, the method comprising:

(a) the client transmitting a service request message and session connection request message to the server;

(b) transmitting a service confirmation message on the request message and a pair of dummy packets to the client;

(c) in response to the pair of packets transmitted by the server, determining an initial bandwidth value of the network and transmitting the determined initial bandwidth value to the server;

(d) comparing the initial bandwidth information transmitted by the client with descriptor information obtained as a result of parsing metadata, determining an appropriate QoS level, and starting to provide multimedia streaming service according to a transmission rate corresponding to the QoS level;

(e) in response to packet information transmitted by the streaming service of the server, measuring the network bandwidth periodically and transmitting the measured bandwidth value to the server; and

(f) according to the network bandwidth value transmitted by the client, extracting a predetermined multimedia stream, and transmitting the extracted multimedia stream to the client.

47. The multimedia streaming method of claim 46, wherein the step (e) comprises:

(e-1) setting the size value of an accumulated packet to 0;

(e-2) starting to receive a packet from the server;

(e-3) setting the time when a first packet is received as T1;

(e-4) after the first packet is input till a last packet is input, whenever a packet is input, accumulating the size value of the packet to the size of the accumulated packet;

(e-5) if the last packet is input, setting the time when the last packet is input as T2;

(e-6) measuring the network bandwidth by calculating

$\frac{\mathrm{Accumulated}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{packet}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{size}\times <br><br>1000\times <br><br>8}{\mathrm{TS2}-\mathrm{TS1}};<br><br>$

(e-7) feeding the measured network bandwidth information back to the server.

48. The multimedia streaming method of claim 46, wherein the multimedia data has any one form of audio data, moving picture data, still picture data, text data, and graphic data.

49. The multimedia streaming method of claim 48, wherein the multimedia data is formed with a bit stream which has any one of a spatial scalable function, a quality scalable function, a temporal scalable function, and fine grain scalable (FGS) function.

50. The multimedia streaming method of claim 46, wherein the metadata is defined based on an extensible markup language (XML).

51. The multimedia streaming method of claim 50, wherein the metadata has a hierarchical structure in the form of a tree having the multimedia data and streaming-related information.

52. The multimedia streaming method of claim 46, wherein if the number of the QoS levels increases, the frame drop rate of the multimedia streaming server gradually decreases and the average bit rate and average peak signal to noise ratio (PSNR) value gradually increase.

60. A computer readable medium having embodied thereon a computer program for performing the method of claim 46.

53. A streaming method of a server which is connected to a client through a network, the method comprising:

(a) receiving the bandwidth of the network from the client;

(b) based on a descriptor obtained as a result of parsing metadata corresponding to multimedia data desired to be provided for service, selecting a current time segment;

(c) comparing a target bit rate defined in the descriptor for the selected segment with the network bandwidth, selecting a QoS level available for service; and

(d) extracting frames corresponding to the selected QoS level and transmitting the frames to the client at each predetermined time interval.

54. The streaming method of claim 53, wherein the multimedia data has any one form of audio data, moving picture data, still picture data, text data, and graphic data.

55. The streaming method of claim 54, wherein the multimedia data is formed with a bit stream which has any one of a spatial scalable function, a quality scalable function, a temporal scalable function, and fine grain scalable (FGS) function.

56. The streaming method of claim 53, wherein the metadata is defined based on an extensible markup language (XML).

57. The streaming method of claim 56, wherein the metadata has a hierarchical structure in the form of a tree having the multimedia data and streaming-related information.

58. The streaming method of claim 53, wherein if the number of the QoS levels increases, the frame drop rate of the multimedia streaming server gradually decreases and the average bit rate and average peak signal to noise ratio (PSNR) value gradually increase.

61. A computer readable medium having embodied thereon a computer program for performing the method of claim 53.

59. A network bandwidth measuring method of a client which receives multimedia data from a server through a network, the method comprising:

(a) setting the size value of an accumulated packet to 0;

(b) starting to receive a packet from the server;

(c) setting the time when a first packet is received as T1;

(d) after the first packet is input till a last packet is input, whenever a packet is input, accumulating the size value of the packet to the size of the accumulated packet;

(e) if the last packet is input, setting the time when the last packet is input as T2;

(f) measuring the network bandwidth by calculating

$\frac{\mathrm{Accumulated}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{packet}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{size}\times <br><br>1000\times <br><br>8}{\mathrm{TS2}-\mathrm{TS1}};<br><br>$

(g) feeding the measured network bandwidth information back to the server.

62. A computer readable medium having embodied thereon a computer program for performing the method of claim 59.