Method and apparatus for reducing the amount of information that must be transmitted to slower viewers over a remote viewing session

• US 8,386,628 B1
• Filed: 02/16/2006
• Issued: 02/26/2013
• Est. Priority Date: 05/23/2005
• Status: Active Grant

• Current Assignee: Glance Networks, Inc.
• Original Assignee: Glance Networks, Inc.
• Inventors: Hardebeck, Ed, Topolcic, Claudio, Baker, Richard L.
• Primary Examiner(s): KIM, TAE K
• Application Number:
  US11/355,740
• Time in Patent Office: 2,567 Days
• Field of Search: 709/203, 709/217, 709/224, 709204-207, 709230-232, 345/1.1, 345/1.2, 345/1.3, 345/2.1, 345/2.2, 345/619, 345629-641
• US Class Current: 709/231
• CPC Class Codes: G06F 3/1415 with means for detecting differences between the image stored in the host and the images displayed on the displays
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G09G 2310/04 Partial updating of the display screen
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

To accommodate different viewer capabilities, update commands may be maintained in an update queue at the server and distributed to viewers according to the viewers'"'"' abilities to receive them. As update commands arrive that supersede older update commands, the new update commands may cause the old update commands to be removed from the queue so that the older update commands may be pruned from the update queue. Where the pruned update messages have not yet been sent to the slower viewers, pruning the older update commands from the update queue enables information that will quickly be replaced via another update command to be selectively not transmitted to slower viewers. Panes may be used to keep track of which updates are required to be maintained in the queue and which may be pruned from the queue.

Claims

1. A method of facilitating a remote viewing session over which host display information is exchanged between a host computing device and a plurality of viewer computing devices, the method comprising the steps of:

storing, by a first computing device, a set of update commands related to the host display information in an update queue for transmission to the plurality of viewer computing devices, the update commands in the update queue comprise paint commands and copy commands, said paint commands containing pixel information derived from the host display information and from which pixel data of the host display may be obtained to recreate an area of the host display information by the at least one viewer computing device and the copy commands containing instructions to the viewer computing device to move a source area of host display information to a target area;

maintaining, by the first computing device, positions of each of the plurality of viewer computing devices in the update queue;

upon receipt of a new update command, determining, by the first computing device, whether the new update command renders one or a small subset of the paint commands or copy commands in the set of update commands in the update queue unnecessary;

pruning, by the first computing device, the one or small subset of the paint commands or copy commands from the update queue if the new update command renders the one or small subset of the update commands contained in the update queue unnecessary, to implement selective pruning of individual paint and copy commands from the update queue while maintaining other paint and copy commands not affected by the new update command within the update queue for transmission to the plurality of viewer computing devices;

transmitting commands, by the first computing device, from the update queue to each of the plurality of viewers, the commands to be transmitted to a particular viewer being selected starting at the position of that viewer in the update queue; and

creating, by the first computing device, a plurality of non-overlapping panes referencing update commands in the update queue and used to keep track of which commands in the update queue are required to recreate the host display information, at least some of the plurality of non-overlapping panes containing pointers to a paint command that contains data to be displayed in that pane and one or more pointers to copy commands that describe where data originates that is needed to reconstruct the pane;

wherein if the one or small subset of the paint commands or copy commands is located within the update queue ahead of a first position of a first of the viewing computing devices, the step of pruning the one or small subset of the paint commands or copy commands from the update queue causes the one or small subset of the paint commands or copy commands to not be transmitted to the first of the viewing computing devices while maintaining all of the other update commands within the update queue ahead of the first position and while maintaining the first position of the first viewing computing device within the update queue, to thereby reduce the number of update commands within the update queue to be transmitted to the first viewing computing device; and

wherein the panes are created to not overlap a grid comprising a plurality of grid lines, and wherein when a copy command causes a pane to translate to straddle a grid line, the translated pane will be fractured into a plurality of smaller panes coincident on at least one edge with the grid lines and not coincident on at least one other edge with any of the grid lines.

2. The method of claim 1, further comprising the steps of receiving an update message containing a plurality of update commands, by the first computing device, one of said update commands being the new update command, extracting the plurality of update commands from the update message by the first computing device, and storing, by the first computing device, the update commands in the update queue.

3. The method of claim 1, wherein the new update command is a paint command.

4. The method of claim 1, wherein the new update command is a copy command.

5. The method of claim 1, wherein the step of determining whether the new update command renders the one or small subset of the paint commands or copy commands unnecessary comprises determining, by the first computing device, whether the one or small subset of the paint commands or copy commands is referenced by any of the panes.

6. The method of claim 1, wherein the paint commands are constrained to align with grid lines and the copy commands are not constrained to align with the grid lines.

7. The method of claim 1, wherein each pane contains a reference to at most one paint command and may also contain a reference to one or more copy commands.

8. The method of claim 1, wherein receipt of a copy command by the viewer computing device will cause the viewer computing device to cause panes associated with the source area to be moved to the target area, and will cause the panes affected by the copy command to include a reference to the copy command.

9. The method of claim 1, wherein when a copy command causes a first pane to partially overlap a second pane, the second pane will be fractured into a plurality of smaller panes surrounding the second pane.

10. The method of claim 1, wherein the panes contain references to update commands that are sufficient and necessary to recreate the area represented by the pane.

11. The method of claim 1, wherein the panes contain references to update commands that are sufficient but maybe not necessary to recreate the area represented by the pane.

19. The method of claim 1, wherein the plurality of non-overlapping panes, referencing update commands in the update queue that are required to recreate the host display information, are used to keep track of how data needs to be moved to recreate the host display information.

12. A method of selectively reducing the number of update commands for transmission from an update queue, the method comprising the steps of:

storing update commands received from a host in an update queue by a first computing device, the update commands comprising paint commands containing pixel information created from tiles of host display information, the tiles of host display information being bounded regions defined on the host display, and the update commands further comprising copy commands containing instructions to move portions of the host display information;

maintaining, by the first computing device, a plurality of non-overlapping panes referencing update commands in the update queue and used to keep track of which commands in the update queue are required to recreate the host display information, at least some of the plurality of non-overlapping panes containing pointers to a paint command that contains data to be displayed in that pane and one or more pointers to copy commands that describe where data originates that is needed to reconstruct the pane;

upon receipt of a new paint command for storage in the update queue by the first computing device, creating, by the first computing device, a new pane referencing the new paint command, said pane being coincident with a bounded region associated with a tile of host display information of the new paint command;

deleting, by the first computing device, any pane in the plurality of non-overlapping panes contained wholly within the bounded region associated with the tile of host display information of the new paint command; and

after deleting any pane in the plurality of non-overlapping panes contained wholly within the bounded region, selectively pruning, by the first computing device, any update commands from the update queue that are not referenced by any of the panes of the plurality of non-overlapping panes while maintaining other commands referenced by at least one of the remaining panes in the plurality of non-overlapping panes;

wherein the panes maintain references to update commands that are required to recreate the host display information on a viewing computing device such that if an update command is not referenced by at least one of the panes in the plurality of non-overlapping panes, the update command is not required to recreate the host display information, and may be pruned from the update queue;

wherein the panes are created to not overlap a grid comprising a plurality of grid lines, and wherein when a copy command causes a pane to translate to straddle a grid line, the translated pane will be fractured into a plurality of smaller panes coincident on at least one edge with the grid lines and not coincident on at least one other edge with any of the grid lines.

13. The method of claim 12, further comprising the step of:

upon receipt of a new copy command by the first computing device, translating, by the first computing device, panes from a source area to a target area associated with the copy command, to cause first panes from the source area to overlap with second panes from the target area, said first panes from the source area containing one or more references to update commands associated with the first panes; and

adding, by the first computing device, the update command references from one of the first panes to each of the second panes overlapped by that first pane.

14. The method of claim 13, further comprising performing, by the first computing device, the step of adding, for each of said first panes.

15. The method of claim 12, further comprising the step of performing, by the first computing device, a second pass processing procedure by analyzing positional information associated with update commands referenced by one of the panes to determine whether any of the references may be deleted from the one of the panes.

16. The method of claim 15, wherein the step of performing the second pass processing procedure is performed preferentially on panes containing a larger number of references when compared with numbers of references contained by other panes.

17. The method of claim 15, wherein the step of analyzing positional information associated with update commands referenced by one of the panes comprises creating temporary smaller panes for the update commands referenced by the one of the panes, and using the smaller panes to determine which of the references may be deleted from the pane.

18. A method of reducing the size of updates in an update queue maintained by a server, the method comprising the steps of:

receiving, by a server, updates from a host computing device, said updates comprising update messages containing update commands, the update commands comprising paint commands and copy commands, said paint commands containing pixel information derived from the host display information and from which pixel data of the host display may be obtained to recreate an area of the host display information by the at least one viewer computing device and the copy commands containing instructions to the viewer computing device to move a source area of host display information to a target area;

decomposing, by the server, the updates to extract the update commands and store the update commands in an update queue, said update commands retaining an association with the update while stored in the queue;

selectively pruning, by the first computing device, individual update commands from the update queue as the individual update commands are superseded by subsequently received update commands associated with subsequent updates while maintaining other commands within the queue which have been received prior to the subsequently received update commands and which are not superseded by the subsequently received update commands;

maintaining, by the server, a separate pointer for each viewer computing device that is to receive information stored within the update queue, the pointer indicating a position of the viewer within the update queue where transmission to the viewer should commence when transmission from the update queue is requested by the viewer;

selecting a group of update commands to be transmitted to a viewer starting at the pointer for the viewer and including only commands within the update queue that have not been selectively pruned by the first computing device; and

transmitting, by the server, the update to the viewer;

wherein the step of selectively pruning includes the step of creating, by the first computing device, a plurality of non-overlapping panes referencing update commands in the update queue and used to keep track of which commands in the update queue are required to recreate the host display information, at least some of the plurality of non-overlapping panes containing pointers to a paint command that contains data to be displayed in that pane and one or more pointers to copy commands that describe where data originates that is needed to reconstruct the pane;

wherein the panes are created to not overlap a grid comprising a plurality of grid lines, and wherein when a copy command causes a pane to translate to straddle a grid line, the translated pane will be fractured into a plurality of smaller panes coincident on at least one edge with the grid lines and not coincident on at least one other edge with any of the grid lines.