Method and apparatus that provides a scalable media delivery system

• US 5,953,506 A
• Filed: 12/17/1996
• Issued: 09/14/1999
• Est. Priority Date: 12/17/1996
• Status: Expired due to Term

• Current Assignee: Adaptive Media Technologies, Inc., Oracle International Corporation (Oracle Corporation)
• Original Assignee: Adaptive Media Technologies, Inc.
• Inventors: Guruswamy, Kowsik, Kalra, Devendra, Krishnamohan, Karnamadakala, Ramamoorthy, Venkatasubbarao, Burr, Timothy J., Balakrishnan, Jeyendran
• Primary Examiner(s): Dinh, Dung C.
• Application Number:
  US08/768,114
• Time in Patent Office: 1,001 Days
• Field of Search: 395/200.61, 370/431, 370/437, 345/302, 345/428, 348/17, 348/18, 348/19, 348/20, 348/384, 348/387, 348/390, 348/420
• US Class Current: 709/231
• CPC Class Codes: G06T 3/4092 Image resolution transcoding, e.g. by using client-server architectures
• CPC Class Codes: H04L 67/75 Indicating network or usage conditions on the user display
• CPC Class Codes: H04N 19/39 involving multiple description coding [MDC], i.e. with separate layers being structured as independently decodable descriptions of input picture data
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4143 embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 7/17336 Handling of requests in head-ends

Abstract

The present invention provides an apparatus and method for encoding, storing, transmitting and decoding multimedia information in the form of scalable, streamed digital data. A base stream containing basic informational content and subsequent streams containing additive informational content are initially created from standard digital multimedia data by a transcoder. Client computers, each of which may have different configurations and capabilities are capable of accessing a stream server that contains the scalable streamed digital data. Each different client computer, therefore, may access different stream combinations according to a profile associated with each different client computer. Thus, the streams accessed from the server are tailored to match the profile of each client computer so that the best combination of streams can be provided to maximize the resolution of the 3D, audio and video components.

Claims

1. A method of communicating multimedia digital data from a server computer to a client computer comprising the steps of:

determining multimedia characteristics of said client computer;

determining available bandwidth of a communication channel linking said server computer and said client computer;

determining a selected set of adaptive streams containing said multimedia digital data relating to at least one of sight and sound and intended to be transmitted to said client computer based upon said determined multimedia characteristics and said determined bandwidth, said adaptive streams being stored on a memory of said server computer and formed of a base stream and a plurality of additive streams, said additive streams containing additive data which corresponds to base data associated with said base stream, such that additional additive data from each additive stream provides increasingly greater resolution of said one sight and sound; and

transmitting said selected set of adaptive streams from said server computer to said client computer.

2. A method according to claim 1 wherein said selected set of adaptive streams relates to sight and is derived from a sequence of digital video pictures, each of said digital video pictures containing a plurality of slices, and each of said slices containing a plurality of blocks, each block representing one of luminance and chrominance information, said luminance and chrominance information in each of said blocks being segmented into a base stream and a plurality of additive streams, each said additive stream containing additive data that corresponds to base data associated with said base stream, such that additional additive data from each additive stream provides increasingly greater resolution of said one luminance and chrominance information for each block.

3. A method according to claim 2 wherein each block is an 8 by 8 array of data and there are 7 additive streams.

4. A method according to claim 3 wherein said base stream comprises a single DCT coefficient, and each additive stream comprises an increasingly greater number of DCT coefficients.

5. A method according to claim 4 wherein said increasingly greater number of DCT coefficients is 3, 5, 7, 9, 11, 13 and 15.

6. A method according to claim 2 wherein said selected set of adaptive streams relates to sight and sound;

and

wherein said step of determining said selected set of adaptive streams further includes the steps of;

receiving at said server computer, from said client computer, a user specified preference for quality of video as compared to quality of audio; and

using said user specified preference to determine said selected set of said adaptive streams for sight.

7. A method according to claim 6 wherein said step of determining said selected set of adaptive streams further includes the steps of:

receiving at said server computer a profile for sight and sound of said client computer; and

using said profile to determine said selected set of adapted streams for sight and sound.

8. A method according to claim 6 wherein said step of determining said selected set of adaptive streams further includes repeating, while said selected set of adaptive streams are being transmitted, said steps of receiving an updated user specified preference for quality of video as compared to quality of audio and using said updated user specified preference to determine an updated selected set of adaptive streams;

and

during said step of transmitting, transmitting said updated selected set of adaptive streams to said client computer.

9. A method according to claim 6 wherein said step of determining said selected set of adaptive streams further includes the step of:

dividing said determined bandwidth into video bandwidth and audio bandwidth;

applying said video bandwidth as a constraint when determining said selected set of adaptive streams; and

during said step of transmitting, transmitting said selected set of adaptive streams using said video bandwidth, and transmitting audio data using said audio bandwidth.

10. A method according to claim 6 wherein said step of determining said selected set of adaptive streams further includes the steps of:

receiving at said server computer a profile for sight and sound of said client computer; and

using said profile to determine said selected set of adapted streams for sight and sound.

11. A method according to claim 9, further including the step of determining a second selected set of adaptive streams containing said multimedia digital data relating to sound that is to be transmitted to said client computer based upon said determined multimedia characteristics, said determined bandwidth and said user specified preference, said second set of adaptive streams being stored on said server computer and formed of a second base stream and at least one second additive stream, each said second additive stream containing second additive data which corresponds to second base data associated with said second base stream, such that additional second additive data from each second additive stream provides increasingly greater resolution of sound;

and

during said step of transmitting, transmitting said second selected set of adaptive streams using said audio bandwidth.

12. A method according to claim 2 further including the step of inserting correction codes into each slice of each additive stream to correct for prediction errors.

13. A method according to claim 2 further comprising the step of decoding said selected set of adaptive streams to reproduce said sight, said base stream and said additive streams in said selected set of adaptive streams each being identified by a distinct stream identification code and each of said base and additive streams including a plurality of picture codes that correlate base data and additive data that relates to the same picture in said sequence of digital video pictures.

14. A method according to claim 13 wherein said base stream and said additive streams in said selected set of adaptive streams further include a slice code that correlates base data and additive data that relates to the same slice in each of said digital video pictures.

15. A method according to claim 14 wherein said step of decoding includes the steps of:

correlating said base and said additive data that relate to the same block using said picture codes and said slice codes; and

reconstructing each of said blocks using said base data and said additive data for each block and said stream identification codes corresponding to said base stream and said additive streams.

16. A method according to claim 2 wherein, for each digital video picture in said sequence of digital video pictures, there is also stored on said memory an associated drop frame code, said drop frame code providing an indication of whether said associated digital video picture should be dropped during transmission of said selected set of adaptive streams for each of a plurality of predetermined frame rates.

17. A method according to claim 16, wherein said drop frame code contains a predetermined plurality of bits and only one of said bits is used to indicate whether to drop said associated digital picture for one of said plurality of predetermined frame rates.

18. A method according to claim 2 wherein:

said digital video pictures in said sequence of digital video pictures are one of an Intra picture, a Bidirectional and a Predicted picture;

each of said Intra, Bidirectional and Predicted pictures containing a respective plurality of Intra, Bidirectional and Predicted slices;

each of said Intra, Bidirectional and Predicted slices containing a respective plurality of Intra, Bidirectional and Predicted blocks; and

each of said Intra, Bidirectional and Predicted blocks representing one of luminance and chrominance information, said luminance and chrominance information in each of said Intra, Bidirectional and Predicted blocks being segmented into a base stream and a plurality of additive streams, each said additive stream containing additive data that corresponds to base data associated with said base stream, such that additional additive data from each additive stream provides increasingly greater resolution of said one luminance and chrominance information for each block.

19. A method according to claim 18 further including the step of inserting correction codes into each slice of each Bidirectional and Predicted additive stream to correct for prediction errors.

20. A method according to claim 1 wherein said selected set of adaptive streams relates to sight and wherein said step of determining said selected set of adaptive streams further includes the steps of:

receiving at said server computer, from said client computer, a user specified preference for quality of vision as compared to quality of audio; and

using said user specified preference to determine said selected set of said adaptive streams for sight.

21. A method according to claim 20 wherein said step of determining said selected set of adaptive streams further includes repeating, while said selected set of adaptive streams are being transmitted, said steps of receiving an updated user specified preference for quality of video as compared to quality of audio and using said updated user specified preference to determine an updated selected set of adaptive streams;

and

during said step of transmitting, transmitting said updated selected set of adaptive streams to said client computer.

22. A method according to claim 21 wherein said step of determining said selected set of adaptive streams further includes the steps of:

receiving at said server computer a profile for sight and sound of said client computer; and

using said profile to determine said selected set of adapted streams for sight and sound.

23. A method according to claim 20 wherein said step of determining said selected set of adaptive streams further includes the steps of:

dividing said determined bandwidth into vision bandwidth and audio bandwidth; and

applying said vision bandwidth as a constraint when determining said selected set of adaptive streams; and

during said step of transmitting, transmitting said selected set of adaptive streams using said vision bandwidth, and transmitting audio data using said audio bandwidth.

24. A method according to claim 23 wherein step of determining said selected set of adaptive streams further includes the steps of:

receiving at said server computer a profile for sight and sound of said client computer; and

using said profile to determine said selected set of adapted streams for sight and sound.

25. A method according to claim 23, further including the step of determining a second selected set of adaptive streams containing said multimedia digital data relating to sound that is to be transmitted to said client computer based upon said determined multimedia characteristics, said determined bandwidth and said user specified preference, said second set of adaptive streams being stored on said server computer and formed of a second base stream and at least one second additive stream, each said second additive stream containing second additive data which corresponds to second base data associated with said second base stream, such that additional second additive data from each second additive stream provides increasingly greater resolution of sound;

and

during said step of transmitting, transmitting said second selected set of adaptive streams using said audio bandwidth.

26. A method according to claim 1 wherein said step of determining said selected set of adaptive streams further includes repeating said steps of determining available bandwidth and determining said selected set of adaptive streams to obtain an updated selected set of adaptive streams;

and

during said step of transmitting, transmitting said updated selected set of adaptive streams to said client computer.

27. A method according to claim 26, wherein said steps of determining available bandwidth and determining said selected set of adaptive streams to obtain said updated selected set of adaptive streams are repeated at a substantially periodic interval.

28. A method according to claim 26 wherein said substantially periodic interval is at least once per minute.

29. A method according to claim 1 wherein said multimedia digital data relates to sight in the form of a rendered graphical image such that at least one object within said rendered image contains a texture in the form of a digital video picture, said digital video picture containing a plurality of slices, and each of said slices containing a plurality of blocks, each block representing one of luminance and chrominance information, said luminance and chrominance information in each of said blocks being segmented into a base stream and a plurality of additive streams, each said additive stream containing additive data that corresponds to base data associated with said base stream, such that additional additive data from each additive stream provides increasingly greater resolution of said one luminance and chrominance information for each block.

30. A method according to claim 29 wherein said texture is in the form of a plurality of digital video pictures.

31. A method according to claim 30 wherein each of said digital video pictures contain a different combination of said additive streams.

32. A method according to claim 1 wherein said selected set of adaptive streams relates to sight, and said selected set of adaptive streams and other graphics data are usable to produce a sequence of graphical image frames from a scene containing a plurality of objects;

wherein said selected set of adaptive streams includes attribute adaptive stream data that includes base attribute adaptive stream data and additive attribute adaptive stream data;

wherein said other graphics data includes scene definition data including global scene data and spatial partitioning data; and

wherein said step of determining said selected set of adaptive streams further includes the steps of;

obtaining a graphic priority table that identifies relative priorities for said attribute adaptive stream data on an object by object basis,

using said relative priorities to determine a priority order of which of said attribute adaptive stream data to transmit more frequently after said transmission step is initiated; and

wherein said transmitting step transmits, from said server computer to said client computer, said global scene data, said spatial partitioning data, and said base and additive attribute adaptive stream data based upon said determined priority order.

33. A method according to claim 1 wherein said attribute adaptive stream data includes geometry attribute adaptive stream data.

34. A method according to claim 33 wherein said attribute adaptive stream data includes texture attribute adaptive stream data.

35. A method according to claim 34 wherein said attribute adaptive stream data includes material attribute adaptive stream data.

36. A method according to claim 32 wherein said attribute adaptive stream data includes texture attribute adaptive stream data.

37. A method according to claim 36 wherein said attribute adaptive stream data includes material attribute adaptive stream data.

38. A method according to claim 34 wherein, for at least one object, said geometry attribute adaptive stream data has a higher priority than said texture attribute adaptive stream data.

39. A method according to claim 33 wherein said spatial partitioning data has at least as high a priority as said geometry attribute adaptive stream data associated with any of said objects.

40. A method according to claim 35 wherein, for at least one object, said material attribute adaptive stream data has a lower priority than said texture attribute adaptive stream data.

41. A method according to claim 32 wherein a plurality of objects each include attribute adaptive stream data, and, within each of said plurality of objects, said relative priorities of said attribute adaptive stream data can be different.

42. A method according to claim 41 wherein a plurality of objects each have a relative priority, and said relative priorities of said objects are used to determine said priority order of which of said attribute adaptive stream data to transmit more frequently after said transmission step is initiated.

43. A method according to claim 42 wherein said relative priority of objects is based upon a distance of said object from a viewpoint.

44. A method according to claim 42 wherein said step of determining said selected set of adaptive streams is based in part upon an associated object'"'"'s distance from said viewpoint so that said attribute adaptive stream data associated with one of said objects that is closer to said viewpoint has a higher relative priority than other attribute adaptive stream data associated with another of said objects that is further from said viewpoint than said one object.

45. A method according to claim 44 wherein geometry attribute adaptive stream data associated with said one object has said higher relative priority than other geometry attribute adaptive stream data associated with said another object.

46. A method according to claim 44 wherein said step of determining further including the step of determining said selected set of additive adaptive streams based upon available bandwidth of a communication channel linking said server computer and said client computer.

47. A method according to claim 46 wherein said step of determining said selected set of adaptive streams further includes the steps of:

repeating said steps of determining available bandwidth and determining said selected set of adaptive streams to obtain an updated selected set of adaptive streams; and

during said step of transmitting, transmitting said updated selected set of adaptive streams to said client computer.

48. A method according to claim 47 wherein said steps of determining available bandwidth and determining said selected set of adaptive streams to obtain said updated selected set of adaptive streams are repeated at a substantially periodic interval.

49. A method according to claim 32 wherein said graphic priority table identifies relative priorities for each of said object attributes on a per object basis such that each of a plurality of different objects in said scene are capable of having some of the same object characteristics at different priorities.

50. A method according to claim 32 further including the steps of:

determining a visible portion of said scene prior to initial transmission of said attribute adaptive stream data; and

wherein during said step of transmitting, initially transmitting, from said selected set of adaptive streams, said base data that substantially corresponds to only said visible portion of said scene.

51. A method according to claim 32 wherein said step of determining further including the step of determining said selected set of additive adaptive streams based upon available bandwidth of a communication channel linking said server computer and said client computer.

52. A method according to claim 51 wherein said step of determining said selected set of adaptive streams further includes the steps of:

repeating said steps of determining available bandwidth and determining said selected set of adaptive streams to obtain an updated selected set of adaptive streams; and

during said step of transmitting, transmitting said updated selected set of adaptive streams to said client computer.

53. A method according to claim 52, wherein said steps of determining available bandwidth and determining said selected set of adaptive streams to obtain said updated selected set of adaptive streams are repeated at a substantially periodic interval.

54. A method according to claim 53 wherein said substantially periodic interval is at least once per minute.

55. A method according to claim 32 wherein one of specular and reflection material object characteristics are contained in one of said additive streams stored on said server computer and said one additive stream is determined not to be part of said selected set of additive streams.

56. A method according to claim 1 wherein said selected set of adaptive streams relates to sight, and said selected set of adaptive streams and other graphics data are usable to produce a sequence of graphical image frames from a scene containing a plurality of objects, wherein said adaptive streams include, for each object, attribute adaptive stream data that includes base attribute adaptive stream data and additive attribute adaptive stream data, wherein said other graphics data includes scene definition data including global scene data and spatial partitioning data;

and said method further comprises the steps of;

retrieving at said client computer said global scene data including a visual portion of the scene data and said spatial positioning data and determining boundaries of said scene and locations of objects in said scene;

drawing a first frame relating to a first visual portion of the scene at said client computer using transmitted base and attribute adaptive stream data; and

determining at said client computer whether to send a message to said server computer indicating that one of updated base attribute adaptive stream data and updated additive attribute adaptive stream data is required due to a change in one of level of detail and said visual portion.

57. A method according to claim 56 wherein said step of determining includes the step of generating performance statistics at said client computer.

58. A method according to claim 56 wherein said step of determining whether one of said updated base attribute adaptive stream data and updated additive attribute adaptive stream data is needed results in the transmission of said message to said server computer indicating that updated additive attribute data is needed, and the subsequent transmission of said updated additive attribute data thereby provides a different level of detail.

59. A method according to claim 58 wherein said attribute adaptive stream data includes geometry attribute adaptive stream data containing vertex information and edge information for each object, and wherein said step of determining results in the transmission of said message indicating that updated additive geometry attribute data is needed, including updated vertex information and updated edge information.

60. A method according to claim 59 wherein said geometry attribute adaptive stream data includes vertex split list information, and said step of determining results in the transmission of said message indicating that updated additive geometry attribute data is needed, including updated vertex split list information.

61. A method according to claim 56 wherein said step of determining whether one of said updated base attribute adaptive stream data and updated additive attribute adaptive stream data is needed results in the transmission of said message to said server computer indicating that updated base attribute data is needed, and the subsequent transmission of said updated base attribute data results in reproducing a second visual portion of the scene that is different from said first visual portion of the scene.

62. A method according to claim 56 wherein said step of determining whether one of said updated base attribute adaptive stream data and updated additive attribute adaptive stream data is needed results in the transmission of said message to said server computer indicating that updated base and additive attribute data is needed, and the subsequent transmission of said updated base attribute data results in reproducing a second visual portion of the scene that is different from said first visual portion of the scene at a different level of detail.

63. A method according to claim 1 wherein said selected set of adaptive streams relates to sight, and said selected set of adaptive streams and other graphics data are usable to produce a sequence of graphical image frames from a scene containing a plurality of objects, wherein said adaptive streams include, for each object, attribute adaptive stream data that includes base attribute adaptive stream data and additive attribute adaptive stream data, wherein said other graphics data includes scene definition data including global scene data and spatial partitioning data;

and said method further comprises the steps of;

retrieving at said client computer said global scene data including a visual portion of the scene data and said spatial positioning data and determining boundaries of said scene and locations of objects in said scene;

drawing a first frame relating to a first visual portion of the scene at said client computer using transmitted base and attribute adaptive stream data, said first visual portion of the scene containing a first object that is further from a camera position than a second object, and said transmitted attribute adaptive stream data associated with said first object having a lower level of detail than said transmitted attribute adaptive stream data associated with said second object.

64. A method according to claim 63 wherein a level of detail of said attribute adaptive stream data transmitted for each of said plurality objects dynamically varies based upon a distance of each of said plurality of objects from a camera position, such that objects closer to said camera position contain said attribute adaptive stream data having a higher level of detail.

65. A method according to claim 63 wherein said attribute adaptive stream data includes geometry attribute adaptive stream data, and wherein a level of detail of said geometry attribute adaptive stream data transmitted for each of said plurality objects dynamically varies based upon a distance of each of said plurality of objects from a camera position, such that objects closer to said camera position contain said geometry attribute adaptive stream data having a higher level of detail.

66. A method according to claim 46 wherein each of said base and additive streams includes a plurality of picture codes that correlate base data and additive data that relates to the same picture in said sequence of digital video pictures.

68. A method according to claim 66 wherein, prior to inputting said sequence of digital video pictures, filtering a plurality of sequential pictures with a plurality of filters to obtain a plurality of filtered picture sequences, each filtered picture sequence corresponding to a different frame rate;

digitizing each of said plurality of filtered picture sequences to obtain a plurality of sequences of digital video pictures, each of said plurality of sequences corresponding to the same video scene at varying degrees of resolution;

performing said inputting, segmenting and storing steps on each of said plurality of sequences of digital video pictures.

69. A method according to claim 68 wherein said plurality of sequences of digital video pictures is at least two.

70. A method according to claim 69 wherein a first of said at least two frame rate sequences of digital video pictures has a frame rate between 15 and 30 frames per second and a second of said at least two frame rate sequences of digital video pictures has a frame rate that is at or below 15 frames per second.

71. A method according to claim 68 wherein, for each digital video picture in each said plurality of sequences of digital video pictures there is also stored on said memory an associated drop frame code, said drop frame code providing an indication of whether said associated digital video picture should be dropped during a subsequent transmission of said adaptive streams when said stored digital video information is read out from said memory.

72. A method according to claim 68 wherein, for each digital video picture in each said plurality of sequences of digital video pictures there is also stored on said memory an associated next picture pointer that points to a next picture in said sequence of digital pictures.

73. A method according to claim 66 wherein, for each digital video picture in said sequence of digital video pictures there is also stored on said memory an associated next picture pointer that points to a next picture in said sequence of digital pictures.

74. A method according to claim 66 wherein, for each digital video picture in said sequence of digital video pictures there is also stored on said memory an associated drop frame code, said drop frame code providing an indication of whether said associated digital video picture should be dropped during a subsequent transmission of said adaptive streams when said stored digital video information is read out at each of a plurality of predetermined frame rates.

75. A method according to claim 66 wherein each of said base and additive streams includes a plurality of picture codes that correlate base data and additive data that relates to the same picture in said sequence of digital video pictures.

67. A method of creating stored digital video information comprising the steps of:

inputting a sequence of digital video pictures, each of said digital video pictures containing a plurality of slices, and each of said slices containing a plurality of blocks, each block representing one of luminance and chrominance information;

segmenting said information in each of said blocks into a base stream and a plurality of additive streams, each said additive stream containing additive data that corresponds to base data associated with said base stream, such that additional additive data from each additive stream provides increasingly greater resolution of said one luminance and chrominance information for each block;

associating one of a plurality of distinct stream identification codes with said base data stream and each of said additive streams;

storing on a memory said plurality of distinct stream identification codes and, for each of said distinct stream identification codes, storing said associated base data or additive data so that said base data and said additive data can be identified when being read out from said memory.

76. A method according to claim 67 wherein said plurality of stream identification codes are stored at least one time for each slice of base and additive data.

77. A method of creating stored digital adaptive stream graphics data representing a scene from digital graphics data that allows for the transmission from a server to a client computer and display of some of said stored digital adaptive stream graphics data as a sequence of graphics image frames, said method of creating stored digital adaptive stream graphics data comprising the steps of:

creating a spatialization of said scene that identifies objects based on location and size;

identifying each of said objects in said scene with an object identifier and storing location data for each object obtained from said digital graphics data with said object identifier;

creating, for each of said objects identified in said scene, a plurality of adaptive object attributes and an associated plurality of adaptive object attribute identifiers, each of said object attributes corresponding to an aspect of said object; and

for each of said objects, correlating different portions of said input digital graphics data with one of said adaptive object attributes; and

storing each said different portion of said input digital graphics data as object attribute data along with one of said adaptive object attribute identifiers.

78. A method according to claim 77 wherein said plurality of object attributes include geometry, material and texture.

79. A method according to claim 77 wherein for each of said plurality of adaptive object attributes for each object, storing an associated priority preference to assist in subsequently determining whether to request said object attribute data associated with that object attribute for each object.

80. A method according to claim 77, wherein said step of creating said spatialization creates a tree structure that bounds said scene and creates bounding boxes within said scene, said tree structure including nodes representing split locations of said scene, said split locations being non-uniform through the use of a split value to assist in placing said split locations at positions that cause objects to be disposed totally within one of said bounding boxes.