Interactive virtual display system

• US 9,323,429 B2
• Filed: 10/25/2013
• Issued: 04/26/2016
• Est. Priority Date: 12/19/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lu, Yan, Sun, Wen, Li, Shipeng
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US14/063,106
• Publication Number:
  US 20140053090A1
• Time in Patent Office: 914 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0331 Finger worn pointing device
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/60 using transform coding

Abstract

An “Interactive Virtual Display,” as described herein, provides various systems and techniques that facilitate ubiquitous user interaction with both local and remote heterogeneous computing devices. More specifically, the Interactive Virtual Display uses various combinations of small-size programmable hardware and portable or wearable sensors to enable any display surface (e.g., computer display devices, televisions, projected images/video from projection devices, etc.) to act as a thin client for users to interact with a plurality heterogeneous computing devices regardless of where those devices are located relative to the user. The Interactive Virtual Display provides a flexible system architecture that enables communication and collaboration between a plurality of both local and remote heterogeneous computing devices. This communication and collaboration enables a variety of techniques, such as adaptive screen compression, user interface virtualization, real-time gesture detection to improve system performance and overall user experience, etc.

Claims

1. A method for providing virtual displays of a plurality of remote heterogeneous computing devices on a local display surface, comprising steps for:

providing a locally constructed composite display comprising a separate virtual display corresponding to each of a plurality of remote computing devices on a local display surface;

each virtual display being locally generated from screen images that are periodically captured from a display device of each remote computing device and transmitted by each corresponding remote computing device to the local display surface;

providing a local user input mechanism for interacting with any of the virtual displays corresponding to any of the remote computing devices;

monitoring interaction with any of the virtual displays and sending corresponding control events to the corresponding remote computing device; and

interpreting the control events to provide input to the corresponding remote computing device in order to control the actions of the corresponding remote computing device.

2. The method of claim 1 wherein one or more of the screen images comprising any of the virtual displays are adaptively compressed.

3. The method of claim 2 wherein the adaptively compressed screen images are generated by omitting one or more blocks of pixels from screen images prior to transmission when those blocks are occluded by any of the other virtual displays on the local display surface.

4. The method of claim 2 wherein one or more of the remote computing devices transmits portions of the adaptively compressed screen images corresponding to a region of interest at a higher periodic rate than portions of the adaptively compressed screen images that are outside of the region of interest.

5. The method of claim 1 further comprising using the local user input mechanism for evaluating input received from one or more wearable sensors to convert user gestures into corresponding control events.

6. The method of claim 5 further comprising changing an appearance of the composite display by emulating visual effects and control actions corresponding to touch-screen type inputs in response to corresponding user gestures received from one or more of the wearable sensors.

7. The method of claim 5 wherein one or more of the wearable sensors includes a multi-axis accelerometer for receiving inputs corresponding to user finger motions, and wherein those finger motions are converted into control events corresponding to particular types of mouse events based on a plurality of finger motion acceleration profiles.

8. The method of claim 5 wherein two or more of the wearable sensors include multi-axis accelerometers for receiving inputs corresponding to user finger motions, and wherein those finger motions are converted into control events corresponding to particular types of pointer events based on a plurality of finger motion acceleration profiles to simulate a multi-touch interface for the local display surface.

9. The method of claim 8 wherein one or more of the wearable sensors including multi-axis accelerometers are worn on at least one finger of each of the users hands.

10. The method of claim 5 wherein one or more of the wearable sensors include wireless communications capabilities.

11. A computer-implemented process for adapting an arbitrary display surface into a thin client having a plurality of virtual interactive displays, comprising using one or more computing devices to perform process actions for:

periodically receiving transmissions of separate screen images corresponding to each of a plurality of applications running on a plurality of remote heterogeneous computing devices;

wherein each separate screen image is captured from a display device of a corresponding remote heterogeneous computing device prior to transmission of that screen image;

locally generating separate virtual interactive displays from the received screen images of each corresponding application;

locally constructing a composite interactive display from two or more of the separate virtual interactive displays and rendering the composite interactive display on a local display surface;

providing a local user interface for interacting with any of the virtual interactive displays comprising the composite interactive display;

monitoring inputs received via the local user interface and converting one or more of those inputs to corresponding control events; and

providing the control events as inputs to the corresponding applications running on the corresponding remote heterogeneous computing devices.

12. The computer-implemented process of claim 11 wherein one or more of the separate screen images are adaptively compressed prior to transmission by omitting one or more blocks of pixels from those screen images when those blocks are occluded by any of the other virtual interactive displays.

13. The computer-implemented process of claim 11 wherein one or more regions of interest of one or more of the transmitted screen images are transmitted at a higher periodic rate than portions of those screen images that are outside of the region of interest.

14. The computer-implemented process of claim 11 wherein a plurality of the inputs converted to control events are provided to the local user interface via one or more wearable sensors.

15. The computer-implemented process of claim 14 further comprising changing an appearance of the composite interactive display by emulating visual effects and control actions corresponding to touch-screen type inputs in response to corresponding gesture-type inputs received from one or more of the wearable sensors.

16. A computer-readable storage device having computer executable instructions stored therein for providing virtual displays of a plurality of remote heterogeneous computing devices on a local display surface, said instructions causing a processor to execute operations comprising:

running one or more applications on each of a plurality of remote heterogeneous computing devices;

periodically capturing and compressing separate screen images from a display device of each of the remote heterogeneous computing devices;

periodically receiving transmissions of the compressed screen images from each of the corresponding remote heterogeneous computing devices;

applying the received compressed screen images for locally generating a composite display comprising a separate virtual display corresponding to each of the applications running on the remote heterogeneous computing devices;

rendering the composite display on a local display surface;

receiving user inputs for interacting with any of the virtual displays comprising the composite display; and

converting the user inputs to corresponding control events and transmitting those control events as inputs to the corresponding applications running on corresponding remote heterogeneous computing devices.

17. The computer-readable storage device of claim 16 wherein the compressed screen images are adaptively generated prior to transmission by omitting one or more blocks of pixels from the compressed screen images when those blocks are occluded by any of the other virtual displays of the composite display.

18. The computer-readable storage device of claim 16 wherein portions of the compressed screen images corresponding to a region of interest are transmitted at a higher periodic rate than portions of the compressed screen images that are outside of the region of interest.

19. The computer-readable storage device of claim 16 generating visual effects on the rendered composite display to emulate corresponding touch-screen type inputs in response to user inputs received from one or more wearable sensors.

20. The computer-readable storage device of claim 16 wherein one or more of the wearable sensors includes a multi-axis accelerometer for receiving inputs corresponding to user finger motions.