Methods, media, and systems for displaying information on a thin-client in communication with a network

• US 7,730,157 B2
• Filed: 09/08/2005
• Issued: 06/01/2010
• Est. Priority Date: 02/03/2005
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Baratto, Ricardo, Nieh, Jason
• Primary Examiner(s): Avellino; Joseph E
• Assistant Examiner(s): TOWFIGHI, AFSHAWN M
• Application Number:
  US11/221,475
• Publication Number:
  US 20060184614A1
• Time in Patent Office: 1,727 Days
• Field of Search: 709/204, 709201-203, 709217-219, 709238-244, 370229-240
• US Class Current: 709/217
• CPC Class Codes: G06F 2209/544 Remote
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

A method and system are provided for executing an application remotely on a server computer for a client computer in a computer network. The server receives at user input from the client computer associated with the application, and executes the application. The server processes display commands of the application to generate display primitives, and translates the display primitives into lower level display commands defining display updates using semantic information of the display primitives. The lower level display commands are selected from a predetermined set of lower level display commands. The lower level display commands are aggregated and ordered into one or more command queues. Each command queue is associated with a given display region. The server computer transmits the lower level display commands in the one or more command queues over the network to the client computer. The client computer is capable of translating the lower level display commands into hardware calls for causing the client computer to display the display updates.

Claims

1. A method of executing an application remotely on a server computer for a client computer in a computer network, the method comprising:

(a) receiving at the server computer user input from the client computer associated with the application;

(b) executing the application at the server computer, including;

(i) processing display commands of the application to generate display primitives;

(ii) translating the display primitives into lower level display commands defining display updates using semantic information of the display primitives and comprising region information identifying at least one of an onscreen region and an offscreen region, the lower level display commands being selected from a predetermined set of lower level display commands;

(iii) aggregating a plurality of lower level display commands translated in (ii); and

(iv) ordering said plurality of lower level display commands into two or more separate command queues based at least in part on the region information of the lower level display commands, each command queue associated with a given onscreen region or offscreen region, including at least one separate command queue associated with a given onscreen region and at least one separate command queue associated with a given offscreen region; and

(c) transmitting the lower level display commands in the two or more separate command queues over the network to the client computer, including transmitting said lower level display commands to the client computer when offscreen data associated with display commands ordered in command queues associated with offscreen regions are to be copied onscreen, the client computer being capable of translating the lower level display commands into hardware calls for causing the client computer to display the display updates.

2. The method of claim 1 further comprising tracking display commands affecting a given display region from new draw operations, and overwriting one or more display commands in one or more of the two or more command queues made irrelevant by the new draw operations.

3. The method of claim 1 further comprising scheduling delivery of said plurality of lower level display commands to the client computer in accordance with the size of each of said lower level display commands, wherein smaller sized commands are sent before larger sized commands.

4. The method of claim 3 wherein scheduling delivery comprises sorting said plurality of lower level display commands into a plurality of queues, each queue associated with a given command size range, and ordering said plurality of lower level display commands in each said queue in accordance with arrival time of said command.

5. A method for displaying information on a thin-client in communication with a network, the method comprising:

receiving display primitives generated by a window system that processes display commands received from at least one application, the display primitives defining updates to onscreen regions and offscreen regions;

translating the display primitives into thin-client display commands defining display updates, the thin-client display commands being selected from a predetermined set of lower-level display commands and comprising region information identifying at least one of an onscreen region and an offscreen region;

storing the thin-client display commands in a plurality of separate region queues based at least in part on the region information of the display commands, each region queue of the plurality of region queues being associated with a particular onscreen or offscreen region, including at least one separate region queue associated with a given onscreen region and at least one separate region queue associated with a given offscreen region; and

transmitting the thin-client display commands stored in the separate region queues to the thin-client for display, including transmitting the thin-client display commands to the thin-client when offscreen data associated with thin-client display commands stored in region queues associated with offscreen regions are to be copied on screen.

6. The method of claim 5, further comprising storing the thin-client display commands in size-based queues within a region queue of the plurality of region queues based at least in part on a size of the thin-client display commands.

7. The method of claim 6, wherein thin-client display commands in size-based queues corresponding to smaller-sized thin-client display commands are transmitted before thin-client display commands in size-based queues corresponding to larger-sized thin-client display commands.

8. The method of claim 5, further comprising scheduling the transmission of said plurality of lower-level display commands to the thin client based at least in part on a size of each of the thin-client display commands, wherein smaller-sized thin-client display commands are sent before larger-sized thin-client display commands.

9. The method of claim 5, further comprising

storing thin-client display commands designated as real-time display commands in a real-time sub-queue of the plurality of region queues, wherein thin-client display commands in a real-time sub-queue are transmitted with higher priority over thin-client display commands not stored in a real-time sub-queue.

10. The method of claim 5, wherein the storing is further based on the at least one application from which a thin-client display command originated.

11. The method of claim 5, further comprising aggregating a first number of the thin-client display commands into a smaller second number of thin-client display commands prior to the transmitting.

12. The method of claim 5, further comprising overwriting one or more of the thin-client display commands in the region queues made irrelevant by one or more other thin-client display commands.

13. The method of claim 5, wherein the predetermined set of lower-level display commands includes at least one of:

a command to transmit unencoded pixel data to be displayed on a region of a display of the thin client;

a command to instruct the thin client to copy a region of the display from its local frame buffer to another location;

a command to instruct the thin client to fill a significant region of the display with a single color;

a command to instruct the thin client to replicate a tile over a region of the display; and

a command to instruct the thin client to perform a fill of a region of the display using a bitmap as a stipple to apply a foreground or background color.

14. The method of claim 5, wherein said thin-client display commands comprise video commands, and the translating is based at least in part on detected video hardware capabilities of the thin client.

15. The method of claim 5, wherein the plurality of region queues comprises a first queue and a second queue, the first queue being associated with a particular onscreen and the second queue being associated with a particular offscreen region.

16. A non transitory computer-readable medium storing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for displaying information on a thin-client in communication with a network, comprising:

receiving display primitives generated by a window system that processes display commands received from at least one application, the display primitives defining updates to onscreen regions and offscreen regions;

translating the display primitives into thin-client display commands defining display updates, the thin-client display commands being selected from a predetermined set of lower-level display commands and comprising region information identifying at least one of an onscreen region and an offscreen region;

storing the thin-client display commands in a plurality of separate region queues based at least in part on the region information of the display commands, each region queue of the plurality of region queues being associated with a particular onscreen or offscreen region, including at least one separate region queue associated with a given onscreen region and at least one separate region queue associated with a given offscreen region; and

transmitting the thin-client display commands stored in the separate region queues to the thin-client for display, including transmitting the thin-client display commands to the thin-client when offscreen data associated with thin-client display commands stored in region queues associated with offscreen regions are to be copied on screen.

17. The non transitory computer-readable medium of claim 16, the method further comprising storing the thin-client display commands in size-based queues within a region queue of the plurality of region queues based at least in part on a size of the thin-client display commands.

18. The non transitory computer-readable medium of claim 17, wherein thin-client display commands in size-based queues corresponding to smaller-sized thin-client display commands are transmitted before thin-client display commands in size-based queues corresponding to larger-sized thin-client display commands.

19. The non transitory computer-readable medium of claim 16, the method further comprising scheduling the transmission of said plurality of lower-level display commands to the thin client based at least in part on a size of each of the thin-client display commands, wherein smaller-sized thin-client display commands are sent before larger-sized thin-client display commands.

20. The non transitory computer-readable medium of claim 16, the method further comprising storing thin-client display commands designated as real-time display commands in a real-time sub-queue of the plurality of region queues, wherein thin-client display commands in a real-time sub-queue are transmitted with higher priority over thin-client display commands not stored in a real-time sub-queue.

21. The non transitory computer-readable medium of claim 16, wherein the storing is further based on the at least one application from which a thin-client display command originated.

22. The non transitory computer-readable medium of claim 16, the method further comprising aggregating a first number of the thin-client display commands into a smaller second number of thin-client display commands prior to the transmitting.

23. The non transitory computer-readable medium of claim 16, the method further comprising overwriting one or more of the thin-client display commands in the region queues made irrelevant by one or more other thin-client display commands.

24. The non transitory computer-readable medium of claim 16, wherein the predetermined set of lower-level display commands includes at least one of:

a command to transmit unencoded pixel data to be displayed on a region of a display of the thin client;

a command to instruct the thin client to copy a region of the display from its local frame buffer to another location;

a command to instruct the thin client to fill a significant region of the display with a single color;

a command to instruct the thin client to replicate a tile over a region of the display; and

a command to instruct the thin client to perform a fill of a region of the display using a bitmap as a stipple to apply a foreground or background color.

25. The non transitory computer-readable medium of claim 16, wherein said thin-client display commands comprise video commands, and the translating is based at least in part on detected video hardware capabilities of the thin client.

26. The non transitory computer-readable medium of claim 16, wherein the plurality of region queues comprises a first queue and a second queue, the first queue being associated with a particular onscreen and the second queue being associated with a particular offscreen region.

27. A system for displaying information on a thin-client in communication with a network, comprising:

a virtual device driver that;

receives display primitives generated by a window system that processes display commands received from at least one application, the display primitives defining updates to onscreen regions and offscreen regions;

translates the display primitives into thin-client display commands defining display updates, the thin-client display commands being selected from a predetermined set of lower-level display commands and comprising region information identifying at least one of an onscreen region and an offscreen region; and

a command scheduler in communication with the virtual device driver that;

receives the thin-client display commands from the virtual device driver;

stores the thin-client display commands in a plurality of separate region queues based at least in part on the region information of the display commands, each region queue of the plurality of region queues being associated with a particular onscreen or offscreen region, including at least one separate region queue associated with a given onscreen region and one separate region queue associated with a given offscreen region; and

transmits the thin-client display commands stored in the separate region queues to the thin-client for display, including transmitting the thin-client display commands to the thin-client when offscreen data associated with thin-client display commands stored in region queues associated with offscreen regions are to be copied on screen.

28. The system of claim 27, wherein the command scheduler further stores the thin-client display commands in size-based queues within a region queue of the plurality of region queues based at least in part on a size of the thin-client display commands.

29. The system of claim 28, wherein thin-client display commands in size-based queues corresponding to smaller-sized thin-client display commands are transmitted before thin-client display commands in size-based queues corresponding to larger-sized thin-client display commands.

30. The system of claim 27, wherein the command scheduler further schedules the transmission of said plurality of lower-level display commands to the thin client based at least in part on a size of each of the thin-client display commands, wherein smaller-sized thin-client display commands are sent before larger-sized thin-client display commands.

31. The system of claim 27, wherein the command scheduler further stores thin-client display commands designated as real-time display commands in a real-time sub-queue of the plurality of region queues and transmits the thin-client display commands in a real-time sub-queue with higher priority over thin-client display commands not stored in a real-time sub-queue.

32. The system of claim 27, wherein the storing is further based on the at least one application from which a thin-client display command originated.

33. The system of claim 27, wherein the command scheduler further aggregates a first number of the thin-client display commands into a smaller second number of thin-client display commands prior to the transmitting.

34. The system of claim 27, wherein the command scheduler further overwrites one or more of the thin-client display commands in the region queues made irrelevant by one or more other thin-client display commands.

35. The system of claim 27, wherein the predetermined set of lower-level display commands includes at least one of:

a command to transmit unencoded pixel data to be displayed on a region of a display of the thin client;

a command to instruct the thin client to copy a region of the display from its local frame buffer to another location;

a command to instruct the thin client to fill a significant region of the display with a single color;

a command to instruct the thin client to replicate a tile over a region of the display; and

a command to instruct the thin client to perform a fill of a region of the display using a bitmap as a stipple to apply a foreground or background color.

36. The system of claim 27, wherein said thin-client display commands comprise video commands, and the translating is based at least in part on detected video hardware capabilities of the thin client.

37. The system of claim 27, wherein the plurality of region queues comprises a first queue and a second queue, the first queue being associated with a particular onscreen and the second queue being associated with a particular offscreen region.