Application sharing

• US 8,407,605 B2
• Filed: 04/03/2009
• Issued: 03/26/2013
• Est. Priority Date: 04/03/2009
• Status: Active Grant

• Current Assignee: Sococo, Inc.
• Original Assignee: Social Communications Company
• Inventors: Go, Alex Say, Petter, Vladimir
• Primary Examiner(s): Vu, Kieu
• Assistant Examiner(s): Bycer, Eric J
• Application Number:
  US12/418,270
• Publication Number:
  US 20100257450A1
• Time in Patent Office: 1,453 Days
• Field of Search: 715/733, 715/788, 715/790, 715/751, 715/753, 715/757, 715/759, 715/765, 715/766, 709/204, 709/205
• US Class Current: 715/753
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04L 12/1813 for computer conferences, e.g. chat rooms instant messaging H04L51/04; protocols for multimedia communication H04L65/1101; arrangements for multi-party communication H04L65/403; telephonic conference arrangements H04M3/56; television conference systems H04N7/15
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

Ones of the windows associated with a software process are identified in a screen layout on a local display of a sharer network node. On the sharer network node, an image of the identified windows as they are arranged in the screen layout and free of obscuration by any other windows in the screen layout is generated. The image is transmitted from the sharer network node to a viewer network node. Also, commands derived from input device events on sharer and viewer network nodes are received. The received commands are processed into a command sequence, which is passed to a shared process executing on the sharer network node. Windows associated with the shared process are presented on the sharer network node in accordance with the received command sequence. An image of the presented windows is generated and transmitted from the sharer network node to the viewer network node.

Claims

1. A method, comprising, by a sharer network node:

on a local display of the sharer network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, a representation of a second viewscreen object in the virtual area, a representation of a user who is present in the virtual area and operating the sharer network node, and a respective representation of each of one or more other communicants who are present in the virtual area and respectively operating one or more other network nodes;

in a screen layout of windows on the local display, selecting multiple windows that are associated with a first software process based on user input in connection with the first viewscreen object, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;

generating a first sequence of composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout;

transmitting the first sequence of composite images from the sharer network node to a particular one of the one or more other network nodes in connection with the first viewscreen object; and

contemporaneously with the transmitting, communicating a second sequence of composite images of windows associated with a second software process between the sharer network node and the particular one of the one or more other network nodes in connection with the second viewscreen object.

2. The method of claim 1, wherein the selecting comprises selecting all of the windows in the screen layout that are associated with the first software process.

3. The method of claim 1, wherein the selecting comprises selecting ones of the windows in the screen layout that match a handle that is assigned to the first software process.

4. The method of claim 3, wherein the selecting comprises selecting a parent window and at least one associated child window that are created by the first software process.

5. The method of claim 1, wherein the generating comprises for each of the composite images in the first sequence:

determining a layering order of the selected windows in relation to each other that corresponds to the relative layering order of the selected windows in the screen layout;

for each of the selected windows, retrieving a respective image of the window; and

compositing the retrieved images into the composite image in accordance with the determined layering order.

6. The method of claim 5, wherein the determining comprises:

generating a z-order list that associates a respective z-order value with a respective window handle for each of the windows in the screen layout; and

deriving the layering order of the selected windows from the z-order list.

7. The method of claim 6, wherein the deriving comprises:

for each of the z-order values in the z-order list, matching the associated window handle with the window handle of a respective one of the selected windows; and

ordering the selected windows in the layering order in accordance with the respective z-order values in the z-order list associated with the ones of the z-order values determined to match the window handles of the selected windows.

8. The method of claim 5, wherein the generating further comprises determining two-dimensional locations of the selected windows in the screen layout for each of the composite images in the first sequence.

9. The method of claim 1, wherein:

the selecting comprises selecting ones of the windows that are associated with a designated group of software processes; and

the generating comprises, for each of the composite images in the first sequence;

determining a layering order of the selected windows in relation to each other that corresponds to the relative layering order of the selected windows in the screen layout;

for each of the selected windows, retrieving a respective image of the window; and

compositing the retrieved images into the composite image in accordance with the determined layering order.

10. The method of claim 1, wherein a respective image of each of the selected windows is stored in a respective memory buffer, and the generating comprises retrieving each of the images from the respective memory buffers and compositing the retrieved images into the composite image for each of the composite images in the first sequence.

11. The method of claim 1, further comprising by the sharer network node setting the windows that are associated with the first software process to a layered windows style such that a respective image of each of the selected windows is stored in a respective memory buffer.

12. The method of claim 1, wherein the transmitting is performed based on a request from the viewer network node to view screen data associated with the first viewscreen object.

13. The method of claim 12, further comprising transmitting the first sequence of composite images from the sharer network node to multiple of the other network nodes based on a respective request from each of the multiple other network nodes to view screen data associated with the first viewscreen object, wherein the first sequence of composite images is transmitted to the multiple other network nodes contemporaneously.

14. The method of claim 1, wherein the virtual area comprises a plurality of zones, each zone of the plurality of zones supporting establishment of communicant presence in the zone;

and further comprising;

on the local display, displaying a visualization of the virtual area, wherein the visualization comprises a spatial layout of the zones;

on the local display, presenting navigation controls and interaction controls, wherein the navigation controls enable the user to specify where to establish a presence in the virtual area and the interaction controls enable the user to manage interactions with the one or more other communicants in the virtual area;

establishing a respective presence of the user in one or more of the zones of the virtual area in response to input received via the navigation controls; and

on the local display, depicting the representations of the user and the one or more other communicants in the zones in which the user and the one or more other communicants are present.

15. The method of claim 1, further comprising presenting the first sequence of composite images on the local display in connection with the first viewscreen object.

16. The method of claim 1, further comprising by the sharer network node:

receiving locally-generated commands that are derived from local input device events on the sharer network node;

receiving remotely-generated commands that are derived from remote input device events on a remote viewer network node;

processing the received locally-generated commands and the received remotely-generated commands into a command sequence;

passing the command sequence to the first software process;

presenting the windows in the screen layout on the local display in accordance with the received command sequence.

17. The method of claim 16, wherein the processing comprises remapping screen location parameter values in the remotely-generated commands from a coordinate system of a shared application image received from the sharer network node to a coordinate system of a main window on the local display.

18. The method of claim 17, wherein the processing comprises arranging into the command sequence (i) the remotely-generated commands with the remapped screen location parameter values and (ii) the locally-generated commands.

19. The method of claim 18, wherein the arranging comprises ordering (i) the remotely-generated commands with the remapped screen location parameter values and (ii) the locally-generated commands based on respective times that the locally-generated commands are derived from the local input device events and respective times that the remotely-generated commands are received.

20. The method of claim 17, further comprising remapping screen location parameter values in the commands derived from the remote input device events from a coordinate system of a main window on a remote display of the viewer network node to a coordinate system of the shared application image.

21. The method of claim 20, wherein the remapping is performed on the viewer network node, and further comprising transmitting the remotely-generated commands with the remapped screen location parameter values to the sharer network node.

22. The method of claim 16, wherein the command sequence controls operation of the shared process.

23. The method of claim 1, wherein:

in the screen layout, at least one of the selected windows is obscured by a particular unselected one of the windows in the screen layout; and

the generating comprises retrieving a respective image of each of the selected windows from a respective memory buffer for each of respective ones of the composite images in the first sequence, and, from the retrieved images, generating the composite image of the selected windows as they are arranged in the screen layout but free of obscuration in any area obscured by the particular unselected window in the screen layout.

24. The method of claim 1, wherein the transmitting is based on a subscription of the particular one of the other network nodes to content associated with the first viewscreen object.

25. The method of claim 24, wherein

multiple of the communicant operating respective ones of the other network nodes are contemporaneously subscribed to content associated with the first viewscreen object;

and

the transmitting comprises contemporaneously transmitting the sequence of composite images to the network nodes being operated by the communicants who are subscribed to the content associated with the first viewscreen object.

26. The method of claim 24, wherein the virtual area is associated with a governance rule that controls access to the sequence of composite images.

27. The method of claim 26, wherein the governance rule defines one or more conditions on abilities of communicants in the virtual area to subscribe to the transmitted sequence of composite images.

28. The method of claim 26, wherein the governance rule defines one or more conditions on one or more types of access to the transmitted sequence of composite images.

29. The method of claim 28, wherein the governance rule defines a condition on abilities of communicants in the virtual area to view the transmitted sequence of composite images.

30. The method of claim 28, wherein the governance rule defines a condition on abilities of communicants in the virtual area to control the transmitted sequence of composite images.

31. The method of claim 1, further comprising selecting one or more windows that are associated with a second software process, and generating the first sequence of composite image from the selected windows of the first and second software processes as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any unselected ones of the windows in the screen layout.

32. The method of claim 1, wherein the communicating comprises by the sharer network node receiving the second sequence of composite images from the particular one of the one or more other network nodes.

33. The method of claim 1, wherein the communicating comprises by the sharer network node transmitting the second sequence of composite images to the particular one of the one or more other network nodes.

34. A method, comprising:

in a screen layout of windows on a local display of a sharer network node, selecting ones of the windows that are associated with a software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;

on the sharer network node, generating a composite image of the selected windows as they are arranged in the screen layout but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout, wherein the generating comprises generating the composite image of the selected windows that includes any contents of the selected windows that are located outside a visible desktop window that contains the screen layout; and

transmitting the composite image from the sharer network node to a viewer network node.

35. A method, comprising:

on a local display of a sharer network node, displaying a visualization of a virtual area comprising a plurality of zones, wherein the visualization comprises a spatial layout of the plurality of zones, each zone of the plurality of zones supports establishment of communication sessions between network nodes associated with the zone, a sharer at the sharer network node and a viewer at the viewer network node both have presence in a particular one of the zones of the plurality that comprises viewscreen objects, and the displaying comprises displaying respective representations of the viewer, the sharer, and the viewscreen objects in the virtual area;

on the local display, presenting navigation controls and interaction controls, wherein the navigation controls enable the sharer to specify where to establish a presence in the virtual area and the interaction controls enable the sharer to manage interactions with the viewer in the virtual area;

in response to input received via the navigation controls, establishing a presence of the sharer in the particular zone of the virtual area;

in a screen layout of windows on the local display, selecting ones of the windows that are associated with a software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;

on the sharer network node, generating a composite image of the selected windows as they are arranged in the screen layout but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout;

transmitting the composite image from the sharer network node to a viewer network node;

performing the selecting, the generating, and the transmitting with respect to one or more other software processes executing on one or more other network nodes; and

on the local display, presenting the respective composite images transmitted from the one or more other network nodes in connection with different respective ones of the viewscreen objects in the particular zone of the virtual area at the same time.

36. The method of claim 35, further comprising for each of the viewscreen objects in connection with which a respective one of the composite images is presented, on the local display showing the graphical depiction of the respective sharer who is operating the network node from which the respective composite image is transmitted.

37. A method, comprising:

in a screen layout of windows on a local display of a sharer network node, selecting ones of the windows that are associated with a software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;

on the sharer network node, generating a composite image of the selected windows as they are arranged in the screen layout but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout;

transmitting the composite image from the sharer network node to a viewer network node;

sending a request to share screen data associated with the software process executing on the viewer network node; and

at the sharer network node, receiving a composite image of shared screen data from the viewer network node, wherein the receiving and the transmitting are performed contemporaneously.

38. At least one non-transitory, tangible computer-readable medium having computer-readable program code embodied therein, the computer-readable program code adapted to be executed by a computer to perform operations comprising:

on a local display of the sharer network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, a representation of a second viewscreen object in the virtual area, a representation of a user who is present in the virtual area and operating the sharer network node, and a respective representation of each of one or more other communicants who are present in the virtual area and respectively operating one or more other network nodes;

in a screen layout of windows on the local display, selecting multiple windows that are associated with a first software process based on user input in connection with the first viewscreen object, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;

generating a first sequence of composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout;

transmitting the first sequence of composite images from the sharer network node to a particular one of the one or more other network nodes in connection with the first viewscreen object; and

contemporaneously with the transmitting, communicating a second sequence of composite images of windows associated with a second software process between the sharer network node and the particular one of the one or more other network nodes in connection with the second viewscreen object.

39. An apparatus, comprising:

a memory storing computer-readable instructions; and

a data processing unit coupled to the memory, operable to execute the instructions, and based at least in part on the execution of the instructions operable to perform operations comprising

on a local display of the sharer network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, a representation of a second viewscreen object in the virtual area, a representation of a user who is present in the virtual area and operating the sharer network node, and a respective representation of each of one or more other communicants who are present in the virtual area and respectively operating one or more other network nodes;

in a screen layout of windows on the local display selecting multiple windows that are associated with a first software process based on user input in connection with the first viewscreen object, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;

generating a first sequence of composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout;

transmitting the first sequence of composite images from the sharer network node to a particular one of the one or more other network nodes in connection with the first viewscreen object; and

contemporaneously with the transmitting, communicating a second sequence of composite images of windows associated with a second software process between the sharer network node and the particular one of the one or more other network nodes in connection with the second viewscreen object.

40. A method, comprising:

on a local display of a sharer network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, and a representation of a second viewscreen object in the virtual area;

in a screen layout of windows on the local display of the sharer network node, selecting ones of the windows that are associated with a software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;

based on user input, associating the software process with the first viewscreen object;

on the sharer network node, generating a first sequence of successive composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration by any other windows in the screen layout;

transmitting the first sequence of composite images from the sharer network node to a first viewer network node based on a subscription of the first viewer network node to content associated with the first viewscreen object;

selecting a second set of one or more of the windows in the screen layout;

generating a second sequence of successive composite images of the selected windows in the second set as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any of the windows in the screen layout outside the second set; and

based on a subscription of a second viewer network node to content associated with the second viewscreen object, transmitting the second sequence of composite images from the sharer network node to the second viewer network node, wherein the first sequence of composite images and the second sequence of composite images are respectively transmitted to the first and second viewer network nodes contemporaneously.

41. A method, comprising:

on a local display of a sharer network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, and a representation of a second viewscreen object in the virtual area;

in a screen layout of windows on the local display of the sharer network node, selecting ones of the windows that are associated with a first software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;

based on user input, associating the first software process with the first viewscreen object;

on the sharer network node, generating a first sequence of successive composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration by any other windows in the screen layout;

transmitting the first sequence of composite images from the sharer network node to a first viewer network node based on a subscription of the first viewer network node to content associated with the first viewscreen object;

based on user input, associating a second software process with the second viewscreen object;

selecting a second set of one or more of the windows that associated with the second software process;

generating a second sequence of successive composite images of the selected windows in the second set as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any of the windows in the screen layout outside the second set; and

based on a subscription of a second viewer network node to content associated with the second viewscreen object, transmitting the second sequence of composite images from the sharer network node to the second viewer network node, wherein the first sequence of composite images and the second sequence of composite images are respectively transmitted to the first and second viewer network nodes contemporaneously.

42. A method, comprising:

on a local display of a first network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, and a representation of a second viewscreen object in the virtual area, wherein a first communicant is operating the first network node and a second communicant is operating a second network node, the first and second communicants are present in a particular zone of the virtual area comprising the first viewscreen object and the second viewscreen object, and the displaying comprises displaying in the virtual area a representation of the first communicant and a representation of the second communicant;

in a screen layout of windows on the local display of the first network node, selecting ones of the windows that are associated with a software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;

based on input from the first communicant, the first network node associating the software process with the first viewscreen object;

the first network node generating a first sequence of successive composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration by any other windows in the screen layout;

transmitting the first sequence of composite images from the first network node to the second network node based on a subscription of the second network node to content associated with the first viewscreen object;

based on input from the first communicant, the first network node subscribing to content associated with the second viewscreen object;

the first network node receiving from the second network node a second sequence of composite images of one or more windows associated with a software process executing on the second network node, wherein the transmitting of the first sequence of composite images from the first network node to the second network node is contemporaneous with the receiving of the second sequence of composite images by the first network node from the second network node; and

presenting the received second sequence of composite images on the local display in association with the second viewscreen object.