Surface Computer User Interaction

• US 20100315413A1
• Filed: 06/16/2009
• Published: 12/16/2010
• Est. Priority Date: 06/16/2009
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Garcia-Mendoza, Armando, Wilson, Andrew David, Izadi, Shahram, Hodges, Stephen E., Villar, Nicolas, Hilliges, Otmar
• Application Number:
  US12/485,499
• Publication Number:
  US 20100315413A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04109 FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location
• CPC Class Codes: G06F 2203/04801 Cursor retrieval aid, i.e. visual aspect modification, blinking, colour changes, enlargement or other visual cues, for helping user do find the cursor in graphical user interfaces
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

Surface computer user interaction is described. In an embodiment, an image of a user'"'"'s hand interacting with a user interface displayed on a surface layer of a surface computing device is captured. The image is used to render a corresponding representation of the hand. The representation is displayed in the user interface such that the representation is geometrically aligned with the user'"'"'s hand. In embodiments, the representation is a representation of a shadow or a reflection. The process is performed in real-time, such that movement of the hand causes the representation to correspondingly move. In some embodiments, a separation distance between the hand and the surface is determined and used to control the display of an object rendered in a 3D environment on the surface layer. In some embodiments, at least one parameter relating to the appearance of the object is modified in dependence on the separation distance.

Claims

1. A method of controlling a surface computing device, comprising:

capturing an image of a hand of a user interacting with a user interface displayed on a surface layer of the surface computing device;

using the image to render a corresponding representation of the hand; and

displaying the representation in the user interface on the surface layer, such that the representation is geometrically aligned with the hand.

2. A method according to claim 1, wherein the representation is a representation of a shadow of the hand on the surface layer.

3. A method according to claim 1, wherein the representation is a representation of a reflection of the hand on the surface layer.

4. A method according to claim 1, wherein the steps of capturing an image, using the image, and displaying the representation are performed in real-time, such that movement of the hand causes the representation to correspondingly move on the user interface.

5. A method according to claim 1, further comprising the step of determining a separation distance between the hand and the surface layer.

6. A method according to claim 5, wherein the representation is rendered such that the representation has a transparency related to the separation distance.

7. A method according to claim 5, wherein the step of determining a separation distance between the hand and the surface layer comprises analyzing an average pixel intensity of the image of the hand.

8. A method according to claim 5, further comprising the steps of:

displaying a representation of a 3D environment in the user interface;

detecting selection by a user of an object rendered in the 3D environment;

determining a planar location of the hand relative to the surface layer; and

controlling the display of the object such that the object'"'"'s position in the 3D environment is related to the separation distance and planar location of the hand.

9. A method according to claim 8, wherein the step of controlling the display of the object further comprises modifying at least one parameter relating to the object'"'"'s appearance in dependence on the separation distance.

10. A method according to claim 9, wherein the parameter relating to the object'"'"'s appearance comprises at least one of:

a color value for the object; and

a transparency value for the object.

11. A method according to claim 9, wherein the step of modifying comprises modifying the at least one parameter if the separation distance is greater than a predetermined threshold.

12. A method according to claim 8, further comprising the steps of:

calculating a shadow cast by the object in accordance with the object'"'"'s position in the 3D environment; and

rendering the shadow cast by the object in the 3D environment.

13. A surface computing device, comprising:

a processor;

a surface layer;

a display device arranged to display a user interface on the surface layer;

an image capture device arranged to capture an image of a hand of a user interacting with the surface layer; and

a memory arranged to store executable instructions to cause the processor to render a corresponding representation of the hand from the image and add the representation to the user interface, such that, when displayed by the display device the representation is geometrically aligned with the hand.

14. A surface computing device according to claim 13, wherein the display device comprises one of:

a projector; and

an LCD panel.

15. A surface computing device according to claim 13, wherein the image capture device comprises one of:

a video camera;

a stereo camera; and

a 3D camera.

16. A surface computing device according to claim 13, wherein the surface layer comprises a switchable diffuser having a first mode of operation in which the switchable diffuser is substantially diffuse and a second mode of operation in which the switchable diffuser is substantially transparent.

17. A surface computing device according to claim 13, wherein the surface layer comprises one of:

a rear projection screen;

a holoscreen; and

a touch sensitive layer.

18. A surface computing device according to claim 13, further comprising a light source arranged to illuminate the hand of the user.

19. A surface computing device according to claim 18, wherein the light source comprises at least one of:

an infra-red light source; and

a structured light pattern source.

20. A method of controlling a surface computing device, comprising:

displaying a representation of a 3D environment in a user interface on a surface layer of the surface computing device;

detecting selection by a user of an object rendered in the 3D environment;

capturing an image of a hand of the user;

determining a separation distance between the hand and the surface layer, and a planar location of the hand relative to the surface layer;

using the image to render a corresponding representation of the hand;

displaying the corresponding representation in the 3D environment, such that the corresponding representation is geometrically aligned with the planar location of the hand; and

controlling the display of the object such that the object'"'"'s position in the 3D environment is related to the separation distance and planar location of the hand.