Thin-client network computing method and system

• US 20060184614A1
• Filed: 09/08/2005
• Published: 08/17/2006
• Est. Priority Date: 02/03/2005
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Nieh, Jason, Baratto, Ricardo
• Granted Patent: US 7,730,157 B2
• Time in Patent Office: Days
• US Class Current: 709/203
• CPC Class Codes: G06F 2209/544 Remote
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

A method and system are provided for executing an application remotely on a server computer for a client computer in a computer network. The server receives at user input from the client computer associated with the application, and executes the application. The server processes display commands of the application to generate display primitives, and translates the display primitives into lower level display commands defining display updates using semantic information of the display primitives. The lower level display commands are selected from a predetermined set of lower level display commands. The lower level display commands are aggregated and ordered into one or more command queues. Each command queue is associated with a given display region. The server computer transmits the lower level display commands in the one or more command queues over the network to the client computer. The client computer is capable of translating the lower level display commands into hardware calls for causing the client computer to display the display updates.

Claims

1. A method of executing an application remotely on a server computer for a client computer in a computer network, the method comprising:

(a) receiving at the server computer user input from the client computer associated with the application;

(b) executing the application at the server computer, including;

(i) processing display commands of the application to generate display primitives;

(ii) translating the display primitives into lower level display commands defining display updates using semantic information of the display primitives, the lower level display commands being selected from a predetermined set of lower level display commands;

(iii) aggregating a plurality of lower level display commands translated in (ii); and

(iv) ordering said plurality of lower level display commands into one or more command queues, each command queue associated with a given display region; and

(c) transmitting the lower level display commands in the one or more command queues over the network to the client computer, the client computer being capable of translating the lower level display commands into hardware calls for causing the client computer to display the display updates.

2. The method of claim 1 further comprising tracking display commands affecting a given display region from new draw operations, and overwriting one or more display commands in one or more command queues made irrelevant by the new draw operations.

3. The method of claim 1 wherein at least some of said one or more command queues are associated with offscreen regions, and wherein (b) further comprises identifying display commands associated with an offscreen drawing operation, translating said display commands associated with said offscreen drawing operation into lower level display commands, ordering said lower level display commands in the command queues associated with said offscreen regions, and transmitting said lower level display commands to the client computer when offscreen data associated with said offscreen drawing operation are to be copied onscreen.

4. The method of claim 1 further comprising scheduling delivery of said plurality of lower level display commands to the client computer in accordance with the size of each of said lower level display commands, wherein smaller sized commands are sent before larger sized commands.

5. The method of claim 4 wherein scheduling delivery comprises sorting said plurality of lower level display commands into a plurality of queues, each queue associated with a given command size range, and ordering said plurality of lower level display commands in each said queue in accordance with arrival time of said command.

6. The method of claim 1 wherein (i) is performed using a window system at the server computer.

7. The method of claim 1 wherein (ii) is performed using a virtual display driver at the server computer.

8. The method of claim 1 wherein (ii) is performed at a video device abstraction layer at the server computer.

9. The method of claim 1 wherein the predetermined set of lower level display commands includes a command to transmit unencoded pixel data to be displayed on a region of a display screen of the client computer.

10. The method of claim 1 wherein the predetermined set of lower level display commands includes a command to instruct the client computer to copy a region of a screen from its local frame buffer to another location.

11. The method of claim 1 wherein the predetermined set of lower level display commands includes a command to instruct the client computer to fill a significant region on the screen with a single color.

12. The method of claim 1 wherein the predetermined set of lower level display commands includes a command to instruct the client computer to replicate a tile over a screen region.

13. The method of claim 1 wherein the predetermined set of lower level display commands includes a command to instruct the client computer to perform a fill using a bitmap of ones and zeros as a stipple to apply a foreground or background color.

14. The method of claim 1 wherein the predetermined set of lower level display commands support graphics compositing operations.

15. The method of claim 1 wherein the network is a local area network or a wide area network.

16. The method of claim 1 wherein said display commands comprise video commands, and wherein the client computer includes hardware to perform video playback, wherein the method further comprises determining video hardware capabilities of the client computer, and wherein (ii) comprises translating the display primitives into lower level display commands in accordance with the video hardware capabilities of the client computer.

17. The method of claim 1 wherein said display updates are sized by the server to fit the size of the client computer display.

18. The method of claim 1 wherein (c) comprises using a server-push architecture to transmit said level display commands to the client computer when generated.

19. A network computing system for executing an application remotely on a server computer for a client computer in a computer network, the system comprising:

a server computer and one or more client computers communicating over a computer network;

wherein said server computer includes;

a server input for receiving user input from one of the one or more client computers, said user input being associated with the application;

a window system for processing display commands of the application to generate display primitives;

a virtual display driver for translating the display primitives into lower level display commands defining display updates using semantic information of the display primitives, the lower level display commands being selected from a predetermined set of lower level display commands;

a command scheduler for aggregating a plurality of lower level display commands translated by the virtual display driver and ordering said plurality of lower level display commands into one or more command queues, each command queue associated with a given display region; and

a server output for transmitting the lower level display commands in the one or more command queues over the network to the one of the one or more client computers, the client computer being capable of translating the lower level display commands into hardware calls for causing the client computer to display the display updates.

20. The system of claim 19 wherein the command scheduler tracks display commands affecting a given display region from new draw operations, and overwrites one or more display commands in one or more command queues made irrelevant by the new draw operations.

21. The system of claim 19 wherein at least some of said command queues are associated with offscreen regions, and wherein the server computer identifies display commands associated with an offscreen drawing operation, translates said display commands associated with said offscreen drawing operation into lower level display commands, orders said lower level display commands in the command queues associated with said offscreen regions, and transmits said lower level display commands to the client computer when offscreen data associated with said offscreen drawing operation are to be copied onscreen.

22. The system of claim 19 wherein the command scheduler schedules delivery of said plurality of lower level display commands to the client computer in accordance with the size of each of said lower level display commands, wherein smaller sized commands are sent before larger sized commands.

23. The system of claim 22 wherein said command scheduler sorts said plurality of lower level display commands into a plurality of queues, each queue associated with a given command size range, and orders said plurality of lower level display commands in each said queue in accordance with arrival time of said command.

24. The system of claim 19 wherein the predetermined set of lower level display commands includes a command to transmit unencoded pixel data to be displayed on a region of a display screen of the client computer.

25. The system of claim 19 wherein the predetermined set of lower level display commands includes a command to instruct the client computer to copy a region of a screen from its local frame buffer to another location.

26. The system of claim 19 wherein the predetermined set of lower level display commands includes a command to instruct the client computer to fill a significant region on the screen with a single color.

27. The system of claim 19 wherein the predetermined set of lower level display commands includes a command to instruct the client computer to replicate a tile over a screen region.

28. The system of claim 19 wherein the predetermined set of lower level display commands includes a command to instruct the client computer to perform a fill using a bitmap of ones and zeros as a stipple to apply a foreground or background color.

29. The system of claim 19 wherein the predetermined set of lower level display commands support graphics compositing operations.

30. The system of claim 19 wherein the network is a local area network or a wide area network.

31. The system of claim 19 wherein said display commands comprise video commands, and wherein the client computer includes hardware to perform video playback, wherein the server computer determines video hardware capabilities of the client computer, and translates the display primitives into lower level display commands in accordance with the video hardware capabilities of the client computer.

32. The system of claim 19 wherein said display updates are sized by the server to fit the size of the client computer display.

33. The system of claim 19 wherein the server computer uses a server-push architecture to transmit said lower level display commands to the client computer when generated.

34. A method of executing an application remotely on a server computer for a client computer in a computer network, the method comprising:

(a) receiving at the server computer user input from the client computer associated with the application;

(b) determining video hardware capabilities of said client computer;

(c) executing the application at the server computer, including encoding video commands of the application in accordance with the video hardware capabilities of said client computer; and

(d) transmitting the encoded video commands over the network to the client computer to be decoded by the client computer and processed using the video hardware capabilities of the client computer into one or more hardware calls for causing the client computer to display video associated with said video commands.

35. The method of claim 34 wherein said hardware capabilities of said client computer include color space conversion and/or scaling.

36. The method of claim 34 wherein said video commands are encoded using a YUV pixel format.

37. A server computer for use in a network computing system for executing an application remotely for a client computer in a computer network, the server computer comprising:

a server input for receiving user input from one of the one or more client computers over the computer network, said user input being associated with the application;

a window system for processing display commands of the application to generate display primitives;

a virtual display driver for translating the display primitives into lower level display commands defining display updates using semantic information of the display primitives, the lower level display commands being selected from a predetermined set of lower level display commands;

a command scheduler for aggregating a plurality of lower level display commands translated by the virtual display driver and ordering said plurality of lower level display commands into one or more command queues, each command queue associated with a given display region; and

a server output for transmitting the lower level display commands in the one or more command queues over the network to the one of the one or more client computers, the client computer being capable of translating the lower level display commands into hardware calls for causing the client computer to display the display updates.

38. The server computer of claim 37 wherein at least some of said command queues are associated with offscreen regions, and wherein the server computer identifies display commands associated with an offscreen drawing operation, translates said display commands associated with said offscreen drawing operation into lower level display commands, orders said lower level display commands in the command queues associated with said offscreen regions, and transmits said lower level display commands to the client computer when offscreen data associated with said offscreen drawing operation are to be copied onscreen.

39. The server computer of claim 37 wherein said display commands comprise video commands, and wherein the client computer includes hardware to perform video playback, wherein the server computer determines video hardware capabilities of the client computer, and translates the display primitives into lower level display commands in accordance with the video hardware capabilities of the client computer.