Client-server visualization system with hybrid data processing

• US 9,019,287 B2
• Filed: 11/21/2008
• Issued: 04/28/2015
• Est. Priority Date: 11/23/2007
• Status: Active Grant

• Current Assignee: PME IP Pty., Ltd.
• Original Assignee: PME IP Australia Pty Ltd. (Pro Medicus Limited)
• Inventors: Westerhoff, Malte, Stalling, Detlev
• Primary Examiner(s): MADAMBA, GLENFORD J
• Application Number:
  US12/275,834
• Publication Number:
  US 20090210487A1
• Time in Patent Office: 2,349 Days
• Field of Search: 709/203, 345/501
• US Class Current: 345/522
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G09G 2360/12 Frame memory handling
• CPC Class Codes: G09G 2360/18 Use of a frame buffer in a display terminal, inclusive of the display panel
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 5/001 Arbitration of resources in a display system, e.g. control of access to frame buffer by video controller and/or main processor
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: G09G 5/363 Graphics controllers
• CPC Class Codes: G09G 5/395 Arrangements specially adapted for transferring the contents of the bit-mapped memory to the screen G09G5/399 takes precedence

Abstract

The invention comprises a system of client-server visualization with hybrid data processing, having a server digital data processor, that allows for server side rendering and processing image data, and client digital data processors simultaneously connected to the server, which receives messages from the clients, creates rendered images of data sets or other data processing results and sends those rendered images and results to the clients for display or further processing. Performing certain image rendering operations on either the server or the client according to which is better suited for the tasks requested by the user at any point in time, and possibly adjusting this division of work dynamically, improves rendering speed and application responsiveness on the clients.

Claims

1. A system for rendering images, comprising:

A. a client digital data processor comprising one or more local processing resources,

B. a server digital data processor in communications coupling with said client digital data processor over a network,

C. said client digital data processor executing an application that generates one or more requests for rendering aspects of one or more images, at least one image including at least a first aspect and a second aspect, and

D. the client digital data processor being responsive to the one or more such requests for (i) effecting rendering of one of said first aspect and said second aspect of the at least one image using the local processing resources, and (ii) effecting rendering of the other of said first aspect and said second aspect of the at least one image by the server digital data processor, wherein the other of said first aspect and said second aspect of the at least one image generated by the server digital data processor is relatively unchanging static and is pasted into a buffer, wherein the one of said first aspect and said second aspect of the at least one image rendered by the local processing resources changes and is combined with the entry in the buffer to change the entry in the buffer, wherein the application displays the changing entry in the buffer as the rendered image.

2. The system of claim 1, wherein the first aspect comprises an overlay graphics aspect of the at least one image and the second aspect comprises an image aspect of the same image.

3. The system of claim 2, wherein the image aspect comprises any of an acquired and a synthesized image and wherein the overlay graphics aspect comprises one or more textual or other graphical elements to be presented with the image aspect.

4. The system of claim 1, wherein the local processing resources include any of a central processing unit (CPU), a graphics processing unit (GPU), and/or a graphics library.

5. The system of claim 1, wherein the client digital data processor comprises a render module that responds to the one or more requests by the application by allocating rendering and compute tasks between the local processing resources and the server digital data processor, wherein the render module sends messages to the server digital data processor over the network to effect any of the rendering and compute tasks allocated to the server digital data processor.

6. The system of claim 5, wherein the render module responds to requests from the application by (i) messaging the server digital data processor to render the other of said first aspect and said second aspect of the at least one image by the server digital data processor, and (ii) effecting rendering of an overlay graphics aspect of that same image using the local processing resources.

7. The system of claim 5, wherein the render module combines aspects of a composite image rendered by the local resources with other aspects of the composite image rendered by the server digital data processor.

8. The system of claim 7, wherein the render module pastes into the buffer an image aspect of a composite image generated by the server digital data processor and adds to the buffer another aspect of the composite image selected from the group consisting of one or more of a text, a ruler, a registration, and a cross hairs rendered by the local processing resources.

9. The system of claim 8, wherein the other aspect of the composite image rendered by the local processing resources and added to the buffer by the render module is an overlay graphics aspect of the composite image.

10. The system of claim 5, wherein the render module allocates rendering and compute tasks between the local processing resources and the server digital data processor based on any of (i) a state of the application, (ii) capabilities of the local processing resources, (iii) a load on those resources, (iv) throughput of a network connecting the client digital data processor and the server digital data processor, (v) image rendering capabilities of the server digital data processor, (vi) a load on the server digital data processor, (vii) a locale of data being rendered.

11. The system of claim 8, wherein the render module discerns information on which such allocation is based from any of communications from the server digital data processor and a rapidity with which the server digital data processor responds to messages submitted to it by the client digital data processor.

12. A system for rendering images, comprising:

A. a client digital data processor comprising one or more local processing resources including a central processing unit (CPU), a graphics processing unit (GPU), and/or a graphics library,

B. a server digital data processor in communications coupling with said client digital data processor,

C. said client digital data processor executing an application that generates one or more requests for rendering aspects of one or more composite images, at least one composite image including at least a first aspect, a second aspect, and a third aspect,

D. said client digital data processor comprising a render module that responds to the one or more requests by the application by allocating rendering and compute tasks between the local processing resources and the server digital data processor, thereby (i) effecting rendering of at least one of said first aspect, said second aspect, and said third aspect of the at least one composite image using the local processing resources, and (ii) effecting rendering of at least one of the remaining aspects of said first aspect, said second aspect, and said third aspect of the at least one composite image by the server digital data processor, wherein the at least one of the remaining aspects of said first aspect, said second aspect, and said third aspect of the at least one composite image by the server digital data processor is relatively unchanging static, wherein the at least one of said first aspect, said second aspect, and said third aspect of the at least one composite image using the local processing resources is dynamic, wherein said render module combines aspects of the at least one composite image rendered by the local resources with aspects of the at least one composite image rendered by the server digital data processor by entering into a buffer the at least one composite image rendered by the server digital data processor and adding to the buffer the at least one composite image rendered by the local resources, wherein the application displays the changing entry in the buffer as the rendered image.

13. The system of claim 12, wherein the render module messages the server digital data processor over a network to effect any of rendering and compute tasks allocated to it.

14. The system of claim 12, wherein

(i) the first aspect comprises an image aspect of the at least one composite image, the image aspect representing any of an acquired and a synthesized image,

(ii) the second aspect comprises an overlay graphics aspect of the composite image, the overlay graphics aspect comprising any of textual or other graphical elements to be presented with the acquired or synthesized image, and

(iii) the third aspect comprises a perspective aspect of that same composite image, the perspective aspect comprising any of a viewer vantage point or virtual camera position.

15. The system of claim 12, wherein the render module combines display aspects of the at least one composite image rendered by the local resources with aspects of the composite image rendered by the server digital data processor.

16. The system of claim 15, wherein the render module pastes into a buffer an image aspect of the composite image generated by the server digital data processor and adds to that buffer another aspect of that composite image rendered by the local processing resources.

17. The system of claim 15, wherein the display aspects are selected from the group consisting of one or more of a text, a ruler, a registration, and a cross hairs.

18. The system of claim 12, wherein the render module has access to the internal state of the application.

19. The system of claim 12, wherein the render module allocates rendering and compute tasks between the local processing resources and the server digital data processor based on any of (i) a state of the application, (ii) capabilities of the local processing resources, (iii) a load on those resources, (iv) throughput of a network connecting the client digital data processor and the server digital data processor, (v) image rendering capabilities of the server digital data processor, (vi) a load on the server digital data processor, (vii) a locale of data being rendered.

20. The system of claim 19, wherein the render module discerns information on which such allocation is based from any of communications from the server digital data processor and a rapidity with which the server digital data processor responds to messages submitted to it by the client digital data processor.

21. A method of rendering images, comprising:

A. executing on a client digital data processor an application that generates one or more requests for rendering aspects of at least one composite image,

B. responding, with the client digital data processor, to one or more such requests by (i) effecting rendering of at least one of said aspects of the at least one composite image on local processing resources of the client digital data processor, and (ii) effecting rendering of others of said aspects of the at least one composite image by a server digital data processor that is in communication coupling with the client digital data processor via a network, said effecting rendering of others of said aspects of the composite image by the server digital data processor includes messaging the server digital data processor over the network, wherein the at least one of said aspects of the at least one composite image rendered on local processing resources changes, wherein the others of said aspects of the at least one composite image rendered by the server digital data processor is relatively unchanging static,

C. pasting into a buffer the entry corresponding to the others of said aspects of the at least one composite image rendered by the server digital data processor,

D. adding to the entry in the buffer the entry corresponding to at least one of said aspects of the at least one composite image rendered on local processing resources, and

E. displaying the entry in the buffer as the rendered image.

22. The method of claim 21, wherein said aspects include any of

(i) a first aspect comprising an image aspect of the composite image, the image aspect representing any of an acquired and a synthesized image,

(ii) a second aspect comprising an overlay graphics aspect of the composite image, the overlay graphics aspect comprising any of textual or other graphical elements to be presented with the acquired or synthesized image, and

(iii) the third aspect comprising a perspective aspect of that same composite image, the perspective aspect comprising any of a viewer vantage point or virtual camera position.

23. The method of claim 21, wherein the local processing resources include any of a central processing unit (CPU), a graphics processing unit (GPU), and/or a graphics library.

24. The method of claim 21, comprising combining display aspects of a composite image rendered by the local resources with aspects of the composite image rendered by the server digital data processor.

25. The method of claim 24, comprising pasting into the buffer an image aspect of the composite image rendered by the server digital data processor, and adding to that buffer a dynamic component rendered by the local processing resources, where the dynamic component is selected from the group consisting of one or more of a text, a ruler, a registration, and a cross hairs.

26. The method of claim 21, comprising allocating rendering and compute tasks between the local processing resources and the server digital data processor based on any of (i) a state of the application, (ii) capabilities of the local processing resources, (iii) a load on those resources, (iv) throughput of a network connecting the client digital data processor and the server digital data processor, (v) image rendering capabilities of the server digital data processor, (vi) a load on the server digital data processor, (vii) a locale of data being rendered.

27. The method of claim 26, comprising discerning information on which such allocation is based from any of communications from the server digital data processor and a rapidity with which the server digital data processor responds to messages submitted to it by the client digital data processor.