System and method for the lossless progressive streaming of images over a communication network

• US 7,454,074 B2
• Filed: 07/18/2005
• Issued: 11/18/2008
• Est. Priority Date: 04/18/2000
• Status: Expired due to Fees

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Dekel, Shai, Goldberg, Nitzan
• Primary Examiner(s): Lu; Tom Y
• Application Number:
  US11/184,365
• Publication Number:
  US 20050271283A1
• Time in Patent Office: 1,219 Days
• Field of Search: 382/240
• US Class Current: 382/240
• CPC Class Codes: H04N 1/3873 defined only by a limited number of coordinate points or parameters, e.g. corners, centre; for trimming
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/129 Scanning of coding units, e.g. zig-zag scan of transform coefficients or flexible macroblock ordering [FMO]
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/1883 the unit relating to sub-band structure, e.g. hierarchical level, directional tree, e.g. low-high [LH], high-low [HL], high-high [HH]
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/635 characterised by filter definition or implementation details
• CPC Class Codes: H04N 21/234345 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4621 Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region

Abstract

A lossless image streaming system for the transmission of images over a communication network. The system eliminates the necessity to store a compressed version of the original image, by losslessly streaming ROI data using the original stored image. The imaging system also avoids the computationally intensive task of compression of the full image. When a user wishes to interact with a remote image, the imaging client generates and sends a ROI request list to the imaging server. The request list can be ordered according to the particular progressive mode selected (e.g., progressive by quality, resolution or spatial order). The imaging server performs a fast preprocessing step in near real time after which it can respond to any ROI requests in near real time. When a ROI request arrives at the server, a progressive image encoding algorithm is performed, but not for the full image. Instead, the encoding algorithm is performed only for the ROI. Since the size of the ROI is bounded by the size and resolution of the viewing device at the client and not by the size of the image, only a small portion of the full progressive coding computation is performed for a local area of the original image.

Claims

1. A system for lossless progressive streaming of images over a communication network, comprising:

an image storage device for storing a digital image;

a client computer coupled to the communication network, wherein said client computer generates and transmits across said communication network a request list containing the coordinates of data blocks required for rendering a region of interest (ROI) within said digital image, wherein said request list is ordered in accordance with a selected progressive mode;

a server computer coupled to said communication network and said image storage device, said server computer adapted to perform the steps of;

preprocessing said digital image through a low pass filter and a lossless wavelet transformation;

receiving said request list from said client computer;

progressively transmitting to said client computer data blocks corresponding to said region of interest in the order they were requested;

wherein said lossless wavelet transform comprises the steps of;

first applying an X-direction wavelet transform to the output of said low pass filter to yield a temporal matrix therefrom;

second applying a low Y-direction wavelet transform to a low portion of said temporal matrix to yield LL and LH subband coefficients; and

third applying a high Y-direction wavelet transform to a high portion of said temporal matrix to yield HL and HH subband coefficients including a half-bit matrix containing half-bits, each half-bit corresponding to an HH subband coefficient.

2. The system according to claim 1, wherein the server computer progressively transmits the region of interest to a selected quality threshold.

3. The system according to claim 1, wherein the server computer progressively transmits the region of interest to lossless quality.

4. The system according to claim 1, wherein the client computer reverse transforms the region of interest received from the server computer, to form a lossless reproduction of the digital image.

5. The system according to claim 4, wherein client computer displays the lossless reproduction of the digital image on a web browser resident on the client computer.

6. The system according to claim 1, wherein the server computer performs the pre-processing step through a lossless wavelet transformation comprising non-identical one-dimensional transforms.

7. The system according to claim 1, wherein said selected progressive mode comprises progressive by quality whereby said request list is ordered in accordance with the absolute value of requested subband coefficients such that subband coefficients with largest absolute values are requested before subband coefficients with smaller absolute values.

8. The system according to claim 1, wherein said selected progressive mode comprises progressive by resolution whereby lower resolution data blocks are requested before higher resolution data blocks.

9. The system according to claim 1, wherein said selected progressive mode comprises progressive by spatial order whereby data blocks are requested from top to bottom.

10. A system for lossless progressive streaming of images over a communication network, comprising:

an image storage device for storing a digital image;

a client computer coupled to the communication network, wherein said client computer generates and transmits across said communication network a request list containing the coordinates of data blocks required for rendering a region of interest (ROI) within said digital image, wherein said request list is ordered in accordance with a selected progressive mode;

a server computer coupled to said communication network and said image storage device, said server computer adapted to perform the steps of;

preprocessing said digital image through a low pass filter and a lossless wavelet transform to yield low pass scaling function data, high pass wavelet coefficient data and halfbit data;

receiving said request list from said client computer;

progressively transmitting to said client computer subband coefficient data blocks corresponding to said region of interest in the order they were requested, said subband coefficient data blocks defined by said coordinates and determined in accordance with said wavelet coefficients and said half-bit matrix;

wherein said lossless wavelet transform comprises the steps of;

first applying an X-direction wavelet transform to the output of said low pass filter to yield a temporal matrix therefrom;

second applying a low Y-direction wavelet transform to a low portion of said temporal matrix to yield LL and LH subband coefficients; and

third applying a high Y-direction wavelet transform to a high portion of said temporal matrix to yield HL and HH subband coefficients including a half-bit matrix containing half-bits, each half-bit corresponding to an HH subband coefficient.

11. The system according to claim 10, wherein the server computer progressively transmits the region of interest to a selected quality threshold.

12. The system according to claim 10, wherein the server computer progressively transmits the region of interest to lossless quality.

13. The system according to claim 10, wherein said client computer reverse transforms the region of interest received from the server computer, to form a lossless reproduction of the digital image.

14. The system according to claim 13, wherein said client computer displays the lossless reproduction of the digital image on a web browser resident on the client computer.

15. The system according to claim 10, wherein the server computer performs the pre-processing step through a lossless wavelet transformation comprising two non-identical one-dimensional transforms.

16. The system according to claim 10, wherein said selected progressive mode comprises progressive by quality whereby said request list is ordered in accordance with the absolute value of requested subband coefficients such that subband coefficients with largest absolute values are requested before subband coefficients with smaller absolute values.

17. The system according to claim 10, wherein said selected progressive mode comprises progressive by resolution whereby lower resolution data blocks are requested before higher resolution data blocks.

18. The system according to claim 10, wherein said selected progressive mode comprises progressive by spatial order whereby data blocks are requested from top to bottom.

19. A method for use on a client computer for lossless progressive streaming of images from a server computer to said client computer over a communication network, said method comprising the steps of:

determining one or more data blocks required for rendering of a region of interest (ROI) within said digital image;

generating a request list of coordinates corresponding to said data blocks, wherein said request list is ordered in accordance with a selected progressive mode;

transmitting said request list to said server computer;

receiving said data blocks from said server computer;

rendering said region of interest utilizing said data blocks;

wherein said data blocks are generated on said server utilizing a lossless wavelet transform comprising the steps of;

first applying an X-direction wavelet transform to the output of a low pass filter to yield a temporal matrix therefrom;

second applying a low Y-direction wavelet transform to a low portion of said temporal matrix to yield LL and LH subband coefficients; and

third applying a high Y-direction wavelet transform to a high portion of said temporal matrix to yield HL and HH subband coefficients including a half-bit matrix containing half-bits, each half-bit corresponding to an HH subband coefficient.

20. The method according to claim 19, wherein the server computer progressively transmits the region of interest to a selected quality threshold.

21. The method according to claim 19, wherein the server computer progressively transmits the region of interest to lossless quality.

22. The method according to claim 19, further comprising the step of reverse transforming at the client computer the region of interest received from the server computer, to form a lossless reproduction of the digital image.

23. The method according to claim 22, further comprising the step of displaying at the client computer the lossless reproduction of the digital image on a web browser resident on the client computer.

24. The method according to claim 19, wherein the pre-processing step is performed through a lossless wavelet transformation comprising two non-identical one-dimensional transforms.

25. The method according to claim 19, wherein said selected progressive mode comprises progressive by quality whereby said request list is ordered in accordance with the absolute value of requested subband coefficients such that subband coefficients with largest absolute values are requested before subband coefficients with smaller absolute values.

26. The method according to claim 19, wherein said selected progressive mode comprises progressive by resolution whereby lower resolution data blocks are requested before higher resolution data blocks.

27. The method according to claim 19, wherein said selected progressive mode comprises progressive by spatial order whereby data blocks are requested from top to bottom.

28. A method for use on a client computer for lossless progressive streaming of images from a server computer to said client computer over a communication network, said method comprising the steps of:

determining one or more data blocks required for rendering of a region of interest (ROI) within said digital image;

generating a request list of coordinates corresponding to said data blocks, wherein said request list is ordered in accordance with the absolute value of requested subband coefficients whereby subband coefficients with larger absolute values are requested before subband coefficients with smaller absolute values;

transmitting said request list to said server computer;

receiving said data blocks from said server computer;

rendering said region of interest utilizing said data blocks;

wherein said data blocks are generated on said server utilizing a lossless wavelet transform comprising the steps of;

first applying an X-direction wavelet transform to the output of a low pass filter to yield a temporal matrix therefrom;

second applying a low Y-direction wavelet transform to a low portion of said temporal matrix to yield LL and LH subband coefficients; and

third applying a high Y-direction wavelet transform to a high portion of said temporal matrix to yield HL and HH subband coefficients including a half-bit matrix containing half-bits, each half-bit corresponding to an HH subband coefficient.

29. The method according to claim 28, wherein the server computer progressively transmits the region of interest to a selected quality threshold.

30. The method according to claim 28, wherein the server computer progressively transmits the region of interest to lossless quality.

31. The method according to claim 28, further comprising the step of reverse transforming at the client computer the region of interest received from the server computer, to form a lossless reproduction of the digital image.

32. The method according to claim 31, further comprising the step of displaying at the client computer the lossless reproduction of the digital image on a web browser resident on the client computer.

33. The method according to claim 28, wherein the pre-processing step is performed through a lossless wavelet transformation comprising non-identical one-dimensional transforms.

34. The server according to claim 28, wherein said selected progressive mode comprises progressive by quality whereby said request list is ordered in accordance with the absolute value of requested subband coefficients such that subband coefficients with largest absolute values are requested before subband coefficients with smaller absolute values.

35. The server according to claim 28, wherein said selected progressive mode comprises progressive by resolution whereby lower resolution data blocks are requested before higher resolution data blocks.

36. The server according to claim 28, wherein said selected progressive mode comprises progressive by spatial order whereby data blocks are requested from top to bottom.

37. A server for lossless progressive streaming of images to a client over a communication network, comprising:

an image storage device for storing said digital image;

a processor adapted to perform the steps of;

preprocessing said digital image through a low pass filter lossless wavelet transformation;

receiving said request list from said client, wherein said request list is ordered in accordance with a selected progressive mode;

progressively transmitting to said client data blocks corresponding to said region of interest in the order they were requested;

wherein said lossless wavelet transform comprises the steps of;

first applying an X-direction wavelet transform to the output of said low pass filter to yield a temporal matrix therefrom;

second applying a low Y-direction wavelet transform to a low portion of said temporal matrix to yield LL and LH subband coefficients; and

third applying a high Y-direction wavelet transform to a high portion of said temporal matrix to yield HL and HH subband coefficients including a half-bit matrix containing half-bits, each half-bit corresponding to an HH subband coefficient.

38. The server according to claim 37, wherein the server computer progressively transmits the region of interest to a selected quality threshold.

39. The server according to claim 37, wherein the server computer progressively transmits the region of interest to lossless quality.

40. The server according to claim 37, wherein the client computer reverse transforms the region of interest received from the server computer, to form a lossless reproduction of the digital image.

41. The server according to claim 40, wherein client computer displays the lossless reproduction of the digital image on a web browser resident on the client computer.

42. The server according to claim 37, wherein the server computer performs the pre-processing step through a lossless wavelet transformation comprising two non-identical one-dimensional transforms.

43. A server for lossless progressive streaming of images to a client over a communication network, comprising:

an image storage device for storing said digital image;

a processor adapted to perform the steps of;

preprocessing said digital image through a lossless wavelet transformation;

generating wavelet coefficients corresponding to said digital image;

receiving said request list from said client computer, wherein said request list is ordered in accordance with the absolute value of requested subband coefficients whereby subband coefficients with larger absolute values are requested before subband coefficients with smaller absolute values;

progressively transmitting to said client computer subband coefficient data blocks corresponding to said region of interest in the order they were requested, said subband coefficient data blocks defined by said coordinates and determined in accordance with said wavelet coefficients;

wherein said lossless wavelet transform comprises the steps of;

first applying an X-direction wavelet transform to the output of said low pass filter to yield a temporal matrix therefrom;

second applying a low Y-direction wavelet transform to a low portion of said temporal matrix to yield LL and LH subband coefficients; and

third applying a high Y-direction wavelet transform to a high portion of said temporal matrix to yield HL and HH subband coefficients including a half-bit matrix containing half-bits, each half-bit corresponding to an HH subband coefficient.

44. The server according to claim 43, wherein said server computer progressively transmits the region of interest to a selected quality threshold.

45. The server according to claim 43, wherein said server computer progressively transmits the region of interest to lossless quality.

46. The server according to claim 43, wherein said client computer reverse transforms the region of interest received from the server computer, to form a lossless reproduction of the digital image.

47. The server according to claim 46, wherein said client computer displays the lossless reproduction of the digital image on a web browser resident on said client computer.

48. The server according to claim 43, wherein said server computer performs said pre-processing step through a lossless wavelet transformation comprising two non-identical one-dimensional transforms.

49. The system according to claim 43, wherein in each layer of precision the order of data block requests is set according to resolution whereby low resolution coefficient data blocks are requested first and high resolution coefficient data blocks are requested last.

50. A system for lossless progressive streaming of images over a communication network, comprising:

an image storage device for storing a digital image;

a client computer coupled to the communication network, wherein said client computer generates and transmits across said communication network a request list containing the coordinates of data blocks required for rendering a region of interest (ROI) within said digital image, wherein said request list is ordered in accordance with the absolute value of requested subband coefficients whereby subband coefficients with larger absolute values are requested before subband coefficients with smaller absolute values;

a server computer, coupled to said communication network and said image storage device, adapted to perform the steps of;

preprocessing the digital image through a low pass filter lossless wavelet transformation;

receiving said request list from said client computer;

progressively transmitting to said client computer data blocks corresponding to said region of interest in the order they were requested;

wherein said lossless wavelet transform comprises the steps of;

first applying an X-direction wavelet transform to the output of said low pass filter to yield a temporal matrix therefrom;

second applying a low Y-direction wavelet transform to a low portion of said temporal matrix to yield LL and LH subband coefficients; and

third applying a high Y-direction wavelet transform to a high portion of said temporal matrix to yield HL and HH subband coefficients including a half-bit matrix containing half-bits, each half-bit corresponding to an HH subband coefficient.

51. A server for lossless progressive streaming of images to a client over a communication network, comprising:

an image storage device for storing a digital image;

a processor in communication with said image storage device and adapted to perform the steps of;

preprocessing said digital image through a low pass filter and a lossless wavelet transform a predetermined number of times to yield low pass scaling function data, high pass wavelet coefficient data and halfbit data;

storing said low pass scaling function data, said high pass wavelet coefficient data and said halfbit data in a memory cache;

receiving a request for one or more data blocks from said client, each data block corresponding to a region of interest;

if a requested data block is not present in said memory cache, performing said step of preprocessing on a minimum portion of the region of interest requiring processing;

transmitting to said client computer subband coefficient data blocks corresponding to said region of interest;

wherein said lossless wavelet transform comprises the steps of;

first applying an X-direction wavelet transform to the output of said low pass filter to yield a temporal matrix therefrom;

second applying a low Y-direction wavelet transform to a low portion of said temporal matrix to yield LL and LH subband coefficients; and

third applying a high Y-direction wavelet transform to a high portion of said temporal matrix to yield HL and HH subband coefficients including a half-bit matrix containing half-bits, each half-bit corresponding to an HH subband coefficient.