Interactive virtual display system for ubiquitous devices

• US 8,594,467 B2
• Filed: 04/26/2011
• Issued: 11/26/2013
• Est. Priority Date: 12/19/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lu, Yan, Sun, Wen, Li, Shipeng
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US13/094,725
• Publication Number:
  US 20110199389A1
• Time in Patent Office: 945 Days
• Field of Search: 382/302, 382/305
• US Class Current: 382/305
• CPC Class Codes: G06F 2203/0331 Finger worn pointing device
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/60 using transform coding

Abstract

An “Interactive Virtual Display,” as described herein, provides various systems and techniques that facilitate ubiquitous user interaction with both local and remote heterogeneous computing devices. More specifically, the Interactive Virtual Display uses various combinations of small-size programmable hardware and portable or wearable sensors to enable any display surface (e.g., computer display devices, televisions, projected images/video from projection devices, etc.) to act as a thin client for users to interact with a plurality heterogeneous computing devices regardless of where those devices are located relative to the user. The Interactive Virtual Display provides a flexible system architecture that enables communication and collaboration between a plurality of both local and remote heterogeneous computing devices. This communication and collaboration enables a variety of techniques, such as adaptive screen compression, user interface virtualization, real-time gesture detection to improve system performance and overall user experience, etc.

Claims

1. A system for adapting an arbitrary display surface into a thin client having a plurality of concurrent virtual interactive displays each corresponding to one of a plurality of remote heterogeneous computing devices, comprising:

a display interface module having one or more communication mechanisms for providing a video input to an arbitrary display surface;

wherein the display interface module further includes one or more communication mechanisms for receiving a local user input;

wherein the display interface module further includes one or more communication mechanisms for communicating with a plurality of heterogeneous computing devices across one or more networks;

wherein the display interface module periodically receives a virtual display from each of the remote heterogeneous computing devices, and wherein the display interface module composes each of the received virtual displays into a combined video output that is provided to the arbitrary display surface via the one or more communication mechanisms for providing the video input to the arbitrary display surface;

wherein the display interface module monitors received user input to determine which of the virtual displays comprising the combined video output the user intends to interact with, and sends corresponding control events to the corresponding heterogeneous computing devices via the one or more networks; and

wherein each corresponding heterogeneous computing device receiving control events interprets those control events as a local user input to control the actions of the corresponding heterogeneous computing device.

2. The system of claim 1 wherein each of the heterogeneous computing devices performs adaptive screen compression and transmission for reducing network bandwidth requirements by omitting one or more blocks of pixels from the compressed screen images that are periodically transmitted when those blocks are occluded by any of the virtual displays of other remote heterogeneous computing devices on the local display surface, as reported to the heterogeneous computing devices by the display interface module.

3. The method of claim 1 wherein each heterogeneous computing device performs adaptive screen compression and transmission for reducing network bandwidth requirements by transmitting portions of the compressed screen images corresponding to a region of interest at a higher periodic rate than portions of the compressed screen images that are outside of the region of interest.

4. The system of claim 1 wherein the communication mechanisms for receiving the local user input further include one or more wearable sensors that are used by the display interface module to convert user gestures into the user input for interacting with any of the virtual displays comprising the combined video output.

5. The system of claim 4 wherein the display interface module changes an appearance of the combined video output by emulating visual effects and control actions corresponding to touch-screen type inputs in response to corresponding user input received from the one or more wearable sensors.

6. The system of claim 4 wherein one or more of the wearable sensors includes a multi-axis accelerometer, and wherein the display interface module converts particular user finger motions into particular types of mouse events via predefined acceleration profiles corresponding to user finger motions.

7. A method for transforming an arbitrary display surface into a thin client for concurrently hosting a plurality of virtual displays corresponding to a plurality of remote heterogeneous computing devices, comprising:

a display interface module comprising;

a video output mechanism in communication with a video input mechanism of arbitrary display surface, one or more communication mechanisms for receiving a local user input, and an agent application running on the display interface module in communication with a plurality of heterogeneous computing devices across one or more networks via an instance of a host application running on each of the remote heterogeneous computing devices;

wherein the agent composes a composite display from a separate virtual display corresponding to each of the remote heterogeneous computing devices, said composite display being sent by the display interface module to the arbitrary display surface;

wherein each virtual display is generated by the agent from compressed screen images that are periodically transmitted by each corresponding host to the agent via the one or more networks;

wherein the agent monitors received user input to determine which of the virtual displays comprising the composite display the user intends to interact with, and sends corresponding control events to the corresponding hosts via the one or more networks; and

wherein each corresponding host receiving control events interprets those control events as a local user input to control the actions of the corresponding heterogeneous computing device.

8. The method of claim 7 wherein each of the hosts performs adaptive screen compression and transmission for reducing network bandwidth requirements by omitting one or more blocks of pixels from the compressed screen images that are periodically transmitted when those blocks are occluded by any of the virtual displays of other remote heterogeneous computing devices on the arbitrary display surface, as reported to each host by the agent.

9. The method of claim 7 wherein each host performs adaptive screen compression and transmission for reducing network bandwidth requirements by transmitting portions of the compressed screen images corresponding to a region of interest at a higher periodic rate than portions of the compressed screen images that are outside of the region of interest as reported to each host by the agent.

10. The method of claim 7 wherein the communication mechanisms for receiving the local user input further include one or more wearable sensors that are used by the display interface module to convert user gestures into the user input for interacting with any of the virtual displays comprising the composite display.

11. The method of claim 10 wherein the display interface module changes an appearance of the composite display by emulating visual effects and control actions corresponding to touch-screen type inputs in response to corresponding user input received from the one or more wearable sensors.