Method and system for scalable multi-user interactive visualization

• US 20100278508A1
• Filed: 05/04/2010
• Published: 11/04/2010
• Est. Priority Date: 05/04/2009
• Status: Active Grant

• Current Assignee: Mamigo Incorporated
• Original Assignee: Mamigo Incorporated
• Inventors: Aggarwal, Manoj
• Granted Patent: US 8,639,046 B2
• Time in Patent Office: Days
• US Class Current: 386/109
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/25825 involving client display capabilities, e.g. screen resolution of a mobile phone optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06
• CPC Class Codes: H04N 21/2665 Gathering content from different sources, e.g. Internet and satellite
• CPC Class Codes: H04N 21/4621 Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/631 Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/6377 directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 Transmission by the client directed to the server

Abstract

A scalable method and apparatus is described to provide personalized interactive visualization of a plurality of compressed image data to a plurality of concurrent users. A plurality of image sources are digitally processed in the compressed domain to provide controllable enhanced user-specific interactive visualization with support for adjustment in viewing parameters such frame-rate, field of view, resolution, color format, viewpoint and bandwidth.

Claims

1. A method to synthesize at least one output view from at least one compressed image source comprising:

a) image processing method for decoding and processing the at least one compressed image source to generate multi-resolution coefficients;

b) a set of view synthesis parameters for each output view; and

c) at least one view synthesis pipeline for processing the multi-resolution coefficients to synthesize at least one view based on the corresponding view synthesis parameters.

2. The method of claim 1, wherein the image source is one of JPEG 2000 or Laplacian compressed.

3. The method of claim 2, wherein, (a) image processing method for decoding the at least one compressed image source to generate multi-resolution coefficients, further comprising:

(i) Tier 2 decoding to parse and de-packetize the coded video;

(ii) Tier 1 decoding to perform arithmetic decoding to extract quantized multi-resolution coefficients;

(iii) De-quantization to remove any bias; and

(iv) Resolution normalization to create a common set of resolution bands across all image sources.

4. The method of claim 1, wherein each view synthesis pipeline, further comprising:

(i) module for selection of multi-resolution coefficients;

(ii) a set of synthesis modules for warping, color adjustment and blending each resolution band;

(iii) reassembly of the synthesized resolution bands; and

(iv) either of encoding of synthesized resolution bands to generate compressed view or decoding of synthesized resolution bands to generate un-compressed view.

5. A video processing system comprising:

a computer server and the computer readable medium containing instructions that, when executed by a processor, cause the processor to perform the method according to claim 1.

6. The video processing system of claim 5, further comprising:

at least one compressed video source;

means for receiving the at least one compressed video source;

at least one user device; and

means for transmitting synthesized views to at least one user device.

7. The video processing system of claim 6, further comprising:

a media server with recording, search and playback support.

8. The method of claim 1, further comprising, means for time aligning image sources.

9. The method of claim 1, further comprising, image processing means for updating image source alignment and image source color adjustment parameters.

10. A method to interactively synthesize at least one output view from at least one compressed image source comprising:

a) image processing method for decoding and processing the at least one compressed image source to generate multi-resolution coefficients;

b) a set of view synthesis parameters for each output view;

c) a View Control Manager to dynamically adjust view synthesis parameters based on view control requests; and

d) at least one view synthesis pipeline for processing the multi-resolution coefficients to synthesize at least one view based on the corresponding view synthesis parameters.

11. The method of claim 10, wherein the image source is one of JPEG 2000 or Laplacian compressed.

12. The method of claim 11, wherein (a) image processing method for decoding the at least one compressed image source to generate multi-resolution coefficients, further comprising:

(i) Tier 2 decoding to parse and de-packetize the coded video;

(ii) Tier 1 decoding to perform arithmetic decoding to extract quantized multi-resolution coefficients;

(iii) De-quantization to remove any bias; and

(iv) Resolution normalization to create a common set of resolution bands across all image sources.

13. The method of claim 10, wherein each view synthesis pipeline, further comprising:

(i) module for selection of multi-resolution coefficients;

(ii) a set of synthesis modules for warping, color adjustment and blending each resolution band;

(iii) reassembly of the synthesized resolution bands; and

(iv) either of encoding of synthesized resolution bands to generate compressed view or decoding of synthesized resolution bands to generate un-compressed view.

14. A video processing system comprising:

a computer server and the computer readable medium containing instructions that, when executed by a processor, cause the processor to perform the method according to claim 10.

15. The video processing system of claim 14, further comprising:

at least one compressed video source;

means for receiving the compressed video;

at least one interactive user device;

means for transmitting synthesized views to at least one user device; and

means for receiving view control messages from at least one interactive user device.

16. The video processing system of claim 15, further comprising:

a media server with search and playback support.

17. The method of claim 10, further comprising, image processing means for updating image source alignment and image source color adjustment parameters.

18. The method of claim 10, further comprising, means for time aligning image sources.

19. The method of claim 10, further comprising:

a Data Source Selector to dynamically select one or more of input image sources, and a multi-port switch to dynamically select one or more view synthesis pipelines.

20. The method of claim 10, further comprising:

a data recorder with search and playback support.

21. A computer server for synthesizing at least one output view from at least one compressed image source, the system comprising:

a) image processing module for decoding and processing the at least one compressed image source to generate multi-resolution coefficients;

b) specification of view synthesis parameters;

c) at least one view synthesis pipeline for processing the multi-resolution coefficients to synthesize at least one view based on the corresponding view synthesis parameters; and

d) interfaces for transferring at least one synthesized view to at least one user device.

22. A computer server of claim 21, wherein the image source is one of JPEG 2000 or Laplacian compressed, and (a) image processing module for partially decoding the at least one compressed image source to extract embedded multi-resolution coefficients.

23. The computer server of claim 21, further comprising:

a calibration update module to dynamically update image source alignment, and image source color adjustment parameters.

24. The computer server of claim 21, further comprising:

a module for time aligning image sources.

25. A computer server for interactively synthesizing at least one output view from at least one compressed image source, the system comprising:

a) image processing module for decoding and processing the compressed image source to generate multi-resolution coefficients;

b) interfaces for receiving view control requests;

c) a View Control Manager to dynamically adjust view synthesis parameters based on view control requests;

d) at least one view synthesis pipeline for processing the multi-resolution coefficients to synthesize at least one view based on the corresponding view synthesis parameters; and

e) interfaces for transferring at least one synthesized view to at least one user device.

26. The computer server of claim 25, wherein the image source is one of JPEG 2000 or Laplacian compressed, and (a) image processing module for partially decoding the compressed image source to extract embedded multi-resolution coefficients.

27. The computer server of claim 25, further comprising:

using a calibration update module to dynamically update image source alignment and image source color adjustment parameters.

28. The computer server of claim 25, further comprising:

a module for time aligning image sources.

29. The computer server of claim 25, further comprising:

data recorder with search and playback support.

30. The computer server of claim 25, further comprising, of Data Source Selector to dynamically select one or more of input image sources, and a multi-port switch to dynamically select one or more view synthesis pipelines.