Multimedia stream compression

• US 20010026591A1
• Filed: 01/25/2001
• Published: 10/04/2001
• Est. Priority Date: 07/27/1998
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Feder, Meir, Paz, Ofir, Keren, Avishai
• Granted Patent: US 8,259,788 B2
• Time in Patent Office: Days
• US Class Current: 375/240.26
• CPC Class Codes: A63F 2300/409 Data transfer via television network
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/5506 using advertisements
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/134 characterised by the element, parameter or criterion affecting or controlling the adaptive coding
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/234336 by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
• CPC Class Codes: H04N 21/234354 by altering signal-to-noise ratio parameters, e.g. requantization
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/234381 by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/23655 Statistical multiplexing, e.g. by controlling the encoder to alter its bitrate to optimize the bandwidth utilization
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2407 Monitoring of transmitted content, e.g. distribution time, number of downloads arrangements for monitoring programmes for broadcast or for distribution combined with broadcast H04H20/14
• CPC Class Codes: H04N 21/241 Operating system [OS] processes, e.g. server setup
• CPC Class Codes: H04N 21/2543 Billing , e.g. for subscription services payment schemes, architectures or protocols G06Q20/00; e-commerce G06Q30/00; arrangements for billing for the use of broadcast information or broadcast-related information H04H60/21
• CPC Class Codes: H04N 21/262 : Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists scheduling strategies for dispatcher in multiprogramming arrangements G06F9/4881; arrangements for scheduling broadcast services or broadcast-related services H04H60/06; flow control in packet networks H04L47/10; establishing a time schedule or organising the servicing of application requests H04L67/62
• CPC Class Codes: H04N 21/2668 : Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063
• CPC Class Codes: H04N 21/274 : Storing end-user multimedia data in response to end-user request , e.g. network recorder
• CPC Class Codes: H04N 21/478 : Supplemental services, e.g. displaying phone caller identification, shopping application
• CPC Class Codes: H04N 21/488 : Data services, e.g. news ticker systems specially adapted for using meteorological information in broadcast systems H04H60/71
• CPC Class Codes: H04N 21/6168 : involving cable transmission, e.g. using a cable modem
• CPC Class Codes: H04N 21/6187 : involving transmission via a telephone network, e.g. POTS
• CPC Class Codes: H04N 21/6405 : Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 21/6408 : Unicasting
• CPC Class Codes: H04N 21/6547 : comprising parameters, e.g. for client setup
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/812 : involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8173 : End-user applications, e.g. Web browser, game
• CPC Class Codes: H04N 21/8453 : by locking or enabling a set of features, e.g. optional functionalities in an executable program
• CPC Class Codes: H04N 21/8543 : using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML] information retrieval of semistructured data, the underlying structure being taken into account, e.g. mark-up language structure data G06F16/80
• CPC Class Codes: H04N 21/8586 : by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02
• CPC Class Codes: H04N 7/165 : Centralised control of user terminal subsequent to an upstream request signal H04N7/17345; Registering at central by two-way working H04N7/17309

Abstract

A computer may be remotely accessed. At a first location, display commands are generated. The display commands are covered into a compressed video data stream. Each display element (5) is checked if it is encoded (52). If object is encoded, it is transcribed into MPEG (54). The image is adjusted for display (56) and compression (58). Additional steps of motion determination (60), change detection (62), compression depth and frame determination (66, 68) are executed. Then the data is transmitted to a second location. The display commands are decompressed and displayed as an image at the second location.

Claims

1. A method of generating a compressed video stream, comprising:

receiving a plurality of display commands; and

directly converting said commands into a compressed video stream, without first generating a display raster.

2. A method of generating a compressed video stream, comprising:

receiving a plurality of display commands; and

directly converting said commands, into said compressed video stream, wherein said converting comprises motion estimation and wherein said motion estimation is performed directly on said commands without first generating a display raster.

3. A method of generating a compressed video stream, comprising:

receiving a plurality of display commands; and

directly converting said commands into said compressed video stream, wherein said converting comprises change detection and wherein said change detection is performed directly on said commands without first generating a display raster.

4. A method of motion estimation for image stream compression, comprising:

providing a plurality of display commands;

identifying display manipulation commands among said commands which manipulation commands manipulate image portions in said image stream, said portions being generated by other display commands among said plurality of display commands; and

generating at least one motion vector, responsive to said identified display manipulation commands.

5. A method according to

claim 4

, wherein said display manipulation commands include a scroll command.

6. A method according to

claim 4

or

claim 5

, wherein said display manipulation command is generated as a direct result of a display of a downloaded image in a WWW browser.

7. A method according to any of claims 4-5, comprising reducing a difficulty of said motion vector generation by modifying said display manipulation commands.

8. A method according to

claim 7

, wherein scroll-type display modification commands are limited to multiples of a compression block size dimension.

9. A method of change detection for image stream compression, comprising:

providing a plurality of display commands;

identifying commands of said plurality of commands which affect image portions in said image stream;

identifying said image portions; and

determining if a change in one of said image portions requires image information to be included in said image stream.

10. A method according to

claim 9

wherein said determining comprises determining if said change is greater than a change threshold.

11. A method of generating a compressed video stream, comprising:

providing a plurality of display commands;

directly setting values in a transform space, responsive to said display commands; and

creating a compressed video stream utilizing said set values.

12. A method according to

claim 11

, wherein said transform space comprises a DCT space.

13. A method according to

claim 11

, wherein directly setting comprises utilizing transform coefficients from a look-up table.

14. A method according to

claim 11

, wherein directly setting comprises copying transform coefficients from a look-up table of transformed graphical primitives.

15. A method according to

claim 11

, wherein directly setting comprises copying transform coefficients from a cache of transformed image portions.

16. A method according to

claim 11

, wherein directly setting comprises rendering said display command into said transform space utilizing an analytical formula associating a graphical primitive defined by said display command with said transform space.

17. A method according to

claim 11

, comprising, modifying at least one of said display commands so that a graphical primitive defined by said at least one display command does not straddle an image block, for which block a set of transform coefficients are defined according to a compression protocol utilized by said compressed image stream.

18. A method of generating a catalog, comprising:

providing a plurality of display elements;

generating a compressed video representation of each one of said elements; and

storing said compressed representations in a memory, indexed by element.

19. A method according to

claim 18

, wherein said display elements comprise GUI (graphic user interface) objects.

20. A method of generating a catalog, comprising:

receiving, during interaction with an Internet a plurality of objects to be displayed;

generating a compressed video representation of each one of said objects; and

storing said compressed representations in a cache of objects, whereby when one of said objects is received again from said Internet at a later time, a earlier stored compressed representation is used for display generation.

21. A method according to

claim 20

, wherein said stored objects comprise at least one JPEG image.

22. A method according to

claim 20

, wherein said stored objects comprise at least one uncompressed image.

23. A method according to

claim 20

, wherein said stored objects comprise a WWW page.

24. A method of generating a compressed video stream, comprising:

generating a plurality of display commands which represents a display, by a program;

modifying at least one of said display commands, independently of said program; and

generating a compressed video stream utilizing said display commands.

25. A method according to

claim 24

, wherein said modifying comprises modifying said commands to responsive to limitations of a display device on which said compressed video stream is to be displayed.

26. A method according to

claim 24

, wherein modifying comprises modifying said commands to reduce bandwidth requirements of said compressed video stream.

27. A method according to

claim 24

, wherein modifying comprises modifying said commands to reduce resource requirements for compressing said compressed video stream.

28. A method according to any of claims 24-27, wherein at least one of said display commands represents an object and wherein said modifying comprises changing said at least one display command such that the object is moved relative to its original display position.

29. A method according to

claim 28

, wherein said compressed video stream comprises utilizes blocks having boundaries and wherein moving comprises moving said object to match at least one compression block boundary.

30. A method according to any of claims 24-27, wherein at least one of said display commands represents an object and wherein said modifying comprises replacing said object with a different object.

31. A method according to

claim 30

, wherein replacing said object comprises replacing said object with a compressed representation of said different object.

32. A method according to

claim 30

, wherein said object comprises a text object.

33. A method according to

claim 30

, wherein said object comprises a background of said display.

34. A method according to

claim 30

, comprising analyzing said object to determine a closest suitable replacement object.

35. A method according to any of claims 24-27, wherein modifying said object comprises changing a font definition for said object.

36. A method according to any of claims 24-27, wherein modifying said object comprises modifying at least one color of said object.

37. A method according to

claim 36

, wherein modifying a color comprises reducing a spatial resolution of said colors.

38. A method according to

claim 36

, wherein modifying a color comprises reducing a color range resolution of said colors.

39. A method according to any of claims 24-27, wherein modifying said object comprises reducing a spatial resolution of said object.

40. A method according to any of claims 24-27, wherein at least one of said display commands comprises a scrolling command and wherein said modifying comprises increasing a granularity of said scrolling.

41. A method according to

claim 40

, wherein said increasing a granularity comprises limiting said scrolling command to multiples of compression blocks size of said compressed stream.

42. A method of generating a compressed video stream, comprising:

generating a plurality of display commands which represents a display;

generating a plurality of transform coefficients from said display commands, wherein said transform coefficients are quantized and wherein said quantization is modified responsive to limitations a display device on which said compressed video stream is to be displayed; and

creating a compressed video stream utilizing said coefficients.

43. A method according to

claim 42

, wherein said coefficients are quantized at a lower resolution in a vertical direction of said display.

44. A method of generating a compressed video stream, comprising:

providing a display definition including a compressed object; and

converting said display definition into a compressed video stream, wherein converting includes transcribing at least a portion of said compressed object, without decompressing said compressed object.

45. A method according to

claim 44

, wherein transcribing comprises reducing a quality level of said compressed object.

46. A method according to

claim 44

or

claim 45

, wherein said compressed stream and said compressed object are compressed utilizing a same transform and wherein transcribing comprises copying transform coefficients from said compressed object to said compressed video stream.

47. A method according to

claim 44

or

claim 45

, wherein said compressed stream and said compressed object are compressed utilizing a motion estimation representation and wherein transcribing comprises copying motion estimation vectors from said compressed object to said compressed video stream.

48. A method according to

claim 44

or

claim 45

, wherein converting said display definition comprises directly converting said display representation into said compressed video stream, without generating an intermediate image raster.

49. A method according to

claim 44

or

claim 45

, wherein said compressed object comprises a second compressed video stream, covering only a portion of said display.

50. A method according to

claim 44

or

claim 45

, wherein said compressed object comprises a compressed image, covering only a portion of said display.

51. A method according to

claim 44

or

claim 45

, wherein said compressed object comprises a compressed audio stream.

52. A method of image processing, comprising:

providing an image to be compressed for transmission;

simultaneously compressing and processing said image;

transmitting said image to a remote location; and

decompressing said processed image at said remote location.

53. A method according to

claim 52

, wherein said simultaneously compressing and processing comprises processing said image by modifying a quantization of at least one transform coefficient of said image.

54. A method according to

claim 52

, wherein said modifying comprises increasing a quantization granularity for vertical coefficients.

55. A method of generating a compressed video stream, comprising:

accumulating a plurality of display commands;

identifying at least one display command of said plurality, whose effect is mooted by a later accumulated display command; and

converting said commands into said compressed video stream, ignoring the mooted display command.

56. A method according to

claim 55

, wherein said mooted display command is ignored if its effect on said compressed video stream, in view of said later command, is below a threshold value.

57. A method according to

claim 55

, wherein converting said commands comprises combining display commands.

58. A method according to any of claims 55-57, wherein said converting comprises directly converting said commands into said compressed video stream without first generating an image raster.

59. A method of video compression, comprising:

generating a display, which generation is refreshed at a generation refresh frame rate;

compressing, transmitting and decompressing said display; and

displaying said display at a second refresh rate other than said generation refresh rate and wherein said refresh is not synchronized to said display refresh rate.

60. A method according to

claim 59

, wherein said compressing is un-synchronized with said generation refresh frame rate.

61. A method according to

claim 59

, wherein said transmitting is un-synchronized with said compressing.

62. A method of generating a compressed video stream, comprising:

generating a plurality of display commands;

accumulating at least one display command;

generating a compressed video stream from said accumulated display commands, wherein accumulating comprises accumulating a number of display commands responsive to instantaneous available resources.

64. A method according to

claim 62

or

claim 63

, wherein generating a compressed video stream is un-synchronized relative to said generation of display commands.

65. A method according to

claim 64

, wherein said display commands are generated in sets, each set associated with a display frame and wherein said accumulating, in a single accumulation set, display commands originating from different sets, wherein said single accumulation set is utilized for generating a single frame of said compressed video stream.

66. A method according to

claim 62

or

claim 63

, comprising assigning a priority to a display command.

67. A method according to

claim 66

, comprising reordering said accumulated display commands, responsive to said priority, wherein generating a compressed video stream comprises generating a first frame in said compressed video stream utilizing an originally later display command, prior to generating a frame in said stream utilizing an originally earlier display command.

63. A method of generating a compressed video stream, comprising:

generating a plurality of display commands;

accumulating at least one display command;

generating a compressed video stream from said accumulated display commands, wherein accumulating comprises accumulating a number of display commands responsive to a desired output frame rate.

68. A method of generating a compressed video stream, comprising:

generating a plurality of display commands;

generating a compressed image portion responsive to said display commands; and

assembling a compressed video stream from said compressed image portion and from at least one previously compressed image portion, created prior to generation of said display commands.

69. A method according to

claim 68

, wherein said previously compressed image portion comprises an image portion created for a previous image of said video stream.

70. A method of asynchronous video stream compression, comprising:

providing a first data source and a second data source;

converting said first data source into a first portion of a compressed video stream;

converting said second data source into a second portion of a compressed video stream; and

mixing said first portion and said second portions to create a single compressed video stream, wherein said first portion and said second portion temporally overlap in said single compressed video stream.

71. A method according to

claim 70

, wherein said first portion and said second portion utilize different frame rates.

72. A method according to

claim 70

, wherein said first portion and said second portion occupy different spatial areas in a display defined by said single compressed video stream.

73. A method according to any of claims 70-72, wherein said first and said second data sources comprise sets of display commands.

74. A method according to

claim 73

, wherein converting said first data source and converting said second data source comprises directly converting said data sources without generating an intermediate image raster.

75. A method according to any of claims 70-72, wherein at least one of said first data source and said second data source comprises an image source.

76. A method according to any of claims 70-72, wherein said first data source comprises a display portion affected by a command entered by a user who views said single compressed video source.

77. A method according to

claim 76

, comprising identifying portions of a data display which are affected by said user command.

78. A method of generating a compressed video stream, comprising:

executing a program, which program generates a set of display commands, representative of a display;

identifying changes in said display which are responsive to at least one type of user command; and

converting said display commands into a compressed video stream, wherein said changes are inserted into said compressed video stream at a higher frame rate than other changes in said display.

79. A method according to

claim 78

, wherein said type of user command comprises a pointing device command.

80. A method according to

claim 78

, wherein said changes comprises an indication of a selection of a GUI (graphical user interface) element.

81. A method according to any of claims 78-80, comprising analyzing said user command to determine display commands which effect said identified changes.

82. A method of display control, comprising:

transmitting a request, from a generator of a compressed video stream, to a source of display commands, which requests relates to said compression; and

modifying a generation of display commands, at said source, responsive to said request.

83. A method according to

claim 82

, wherein said source of display commands comprises a WWW browser.