Client-server visualization system with hybrid data processing

• US 9,595,242 B1
• Filed: 03/06/2015
• Issued: 03/14/2017
• Est. Priority Date: 11/23/2007
• Status: Active Grant

• Current Assignee: PME IP Pty., Ltd.
• Original Assignee: PME IP Pty., Ltd.
• Inventors: Westerhoff, Malte, Stalling, Detlev
• Primary Examiner(s): Madamba, Glenford
• Application Number:
  US14/641,243
• Time in Patent Office: 739 Days
• Field of Search: 709/203, 345/501
• US Class Current: 1/1
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G09G 2360/12 Frame memory handling
• CPC Class Codes: G09G 2360/18 Use of a frame buffer in a display terminal, inclusive of the display panel
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 5/001 Arbitration of resources in a display system, e.g. control of access to frame buffer by video controller and/or main processor
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: G09G 5/363 Graphics controllers
• CPC Class Codes: G09G 5/395 Arrangements specially adapted for transferring the contents of the bit-mapped memory to the screen G09G5/399 takes precedence

Abstract

The invention comprises a system of client-server visualization with hybrid data processing, having a server digital data processor, that allows for server side rendering and processing image data, and client digital data processors simultaneously connected to the server, which receives messages from the clients, creates rendered images of data sets or other data processing results and sends those rendered images and results to the clients for display or further processing. Performing certain image rendering operations on either the server or the client according to which is better suited for the tasks requested by the user at any point in time, and possibly adjusting this division of work dynamically, improves rendering speed and application responsiveness on the clients.

Claims

1. A system for rendering images, comprising:

a server digital data processor;

a server graphics processing unit (GPU);

a client digital data processor including one or more local processing resources, a display device and a main memory; and

a server buffer accessible for processing by the server digital data processor, where the server buffer can be available for processing by the client digital data processor;

a network, where the server digital data processor is in communications coupling with the client digital data processor over the network, where the client digital data processor executes an application residing on the main memory that generates one or more requests for rendering aspects of one or more images, at least one image including at least a first aspect and a second aspect;

where the server digital data processor renders the first aspect of the at least one image and pastes the first aspect into the server buffer;

where the client digital data processor renders the second aspect of the at least one image using the one or more local processing resources and adds the second aspect to the server buffer; and

where the server digital data processor makes the server buffer available for further processing by the client digital data processor.

2. The system of claim 1, where the further processing includes adding one or more textual elements.

3. The system of claim 1, where the further processing includes display by the display device.

4. The system of claim 1, where the first aspect comprises any of an acquired and a synthesized image.

5. The system of claim 1, where the first aspect comprises one or more graphical elements.

6. The system of claim 1, where the second aspect comprises one or more textual elements.

7. The system of claim 1, where the second aspect comprises an overlay aspect.

8. The system of claim 1, where the client digital data processor further comprises a render module that responds to multiple requests by the application by allocating rendering and compute tasks between the one or more local processing resources and the server digital data processor.

9. The system of claim 8, where the render module responds to requests from the application by messaging the server digital data processor to render a first graphics aspect of a composite image, effecting rendering of a second graphics aspect of the composite image using the one or more local processing resources and further processing includes adding one or more textual elements to the composite image.

10. The system of claim 8, where the render module combines a composite image including:

(i) graphics aspects rendered by the server digital data processor;

(ii) graphics aspects rendered by the one or more local processing resources; and

(iii) non graphics elements rendered by the one or more local processing resources during further processing.

11. The system of claim 8, where the render module:

(i) pastes into the render buffer the first aspect of a composite image generated by the server digital data processor;

(ii) adds to the render buffer the second aspect of the composite image rendered by the one or more local processing resources; and

(iii) adds to the render buffer a non graphics aspect of the composite image rendered by the one or more local processing resources during further processing.

12. The system of claim 8, where the render module allocates rendering and compute tasks between the one or more local processing resources and the server digital data processor based on capabilities of the one or more local processing resources.

13. The system of claim 8, where the render module allocates rendering and compute tasks between the one or more local processing resources and the server digital data processor based on one or both load on the one or more local processing resources and load on the server digital data processor.

14. The system of claim 13, where the render module discerns load on a resource based on one or both any communications from the server digital data processor and rapidity with which the server digital data processor responds to messages submitted to it by the client digital data processor.

15. The system of claim 8, where the render module allocates rendering and compute tasks between the one or more local processing resources and the server digital data processor based on throughput of the network connecting the client digital data processor and the server digital data processor.

16. The system of claim 8, where the render module allocates rendering and compute tasks between the one or more local processing resources and the server digital data processor based on image rendering capabilities of the server digital data processor.

17. The system of claim 8, where the render module allocates rendering and compute tasks between the one or more local processing resources and the server digital data processor based on one or both processing state of the application and locale of data being rendered.

18. A method of rendering images comprising:

a server receiving a request from an application residing on a main memory of a client digital data processor that generates one or more requests for rendering aspects of one or more images, at least one image including at least a first aspect and a second aspect, where the server includes a server digital data processor, a server graphics processing unit and a server buffer accessible for processing by the server digital data processor, where the server buffer can be available for processing by the client digital data processor, where the client digital data processor further includes one or more local processing resources and a display device;

rendering the first aspect of the at least one image using the server graphics processing unit;

pasting the first aspect into the server buffer;

receiving the second aspect of the at least one image rendered using the one or more local processing resources;

adding the second aspect to the server buffer; and

making the server buffer available for further processing by the client digital data processor.

19. The method of claim 18, where further processing includes displaying the at least one image on the display device.

20. The method of claim 18, further comprising allocating rendering and computing tasks between the one or more local processing resources and the server digital data processor based on any of (i) a state of the application, (ii) capabilities of the one or more local processing resources, (iii) a load on those resources, (iv) throughput of a network connecting the client digital data processor and the server digital data processor, (v) image rendering capabilities of the server digital data processor, (vi) a load on the server digital data processor, (vii) a locale of data being rendered.