KEYBOARD-BASED MULTI-TOUCH INPUT SYSTEM USING A DISPLAYED REPRESENTATION OF A USERS HAND

• US 20130082928A1
• Filed: 09/30/2011
• Published: 04/04/2013
• Est. Priority Date: 09/30/2011
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kim, Seung Wook, Liu, Eric, Marti, Stefan J.
• Application Number:
  US13/249,421
• Publication Number:
  US 20130082928A1
• Time in Patent Office: Days
• US Class Current: 345/168
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance

Abstract

Example embodiments relate to a keyboard-based multi-touch input system using a displayed representation of a user'"'"'s hand. In example embodiments, a sensor detects movement of a user'"'"'s hand in a direction parallel to a top surface of a physical keyboard. A computing device may then receive information describing the movement of the user'"'"'s hand from the sensor and output a real-time visualization of the user'"'"'s hand on the display. This visualization may be overlaid on a mufti-touch enabled user interface, such that the user may perform actions on objects within the user interface by performing multi-touch gestures.

Claims

1. An apparatus comprising:

a keyboard;

a sensor for detecting movement of a user'"'"'s digits in a direction parallel to a top surface of the keyboard;

a display device; and

a processor to;

receive information describing the movement of the user'"'"'s digits from the sensor,

output a real-time visualization of the user'"'"'s digits on the display device, the visualization overlaid on a multi-touch user interface including a displayed object, and

perform a multi-touch command on the displayed object based on the movement of the user'"'"'s digits detected by the sensor.

2. The apparatus of claim 1, wherein the sensor comprises one or more of a camera placed above the top surface of the keyboard, a capacitive touch sensor, an infrared sensor, and an electric field sensor.

3. The apparatus of claim 2, wherein, when the sensor includes a camera, the processor outputs the real-time visualization of the user'"'"'s digits as a video representation of the user'"'"'s hand obtained by the camera.

4. The apparatus of claim 1, wherein the sensor is a camera mounted to a top surface of the keyboard.

5. The apparatus of claim 4, wherein the camera is mounted to a boom movable between an extended position and a retracted position.

6. The apparatus of claim 1, wherein the processor is additionally to:

detect a predetermined input for switching between a multi-touch mode and a keyboard-only mode, and

toggle display of the real-time visualization of the user'"'"'s digits in response to switching between the multi-touch mode and the keyboard-only mode.

7. The apparatus of claim 1, wherein:

the multi-touch user interface includes windows in a plurality of stacked layers, and

the processor is additionally to update the multi-touch user interface to display a window of a currently-selected layer in a foreground of the interface based on a position of the real-time visualization within the plurality of stacked layers.

8. The apparatus of claim 7, wherein the processor is additionally to:

modify the currently-selected layer of the plurality of stacked layers based on user selection of one or more predetermined keys on the keyboard.

9. The apparatus of claim 7, wherein the processor is additionally to:

modify the currently-selected layer of the plurality of stacked layers based on a distance of the user'"'"'s digits from the top surface of the keyboard.

10. The apparatus of claim 7, wherein the processor is additionally to:

modify the currently-selected layer of the plurality of stacked layers based on a speed of the movement of the user'"'"'s digits in the direction parallel to the top surface of the keyboard.

11. The apparatus of claim 7, wherein, in outputting the real-time visualization, the processor is additionally to:

identify a plurality of portions of the real-time visualization of the user'"'"'s digits, each portion intersecting a respective layer of the plurality of stacked layers, and

apply a unique visualization to each identified portion of the plurality of portions of the real-time visualization.

12. The apparatus of claim 7, wherein, the processor is additionally to:

display the currently-selected layer in the foreground of the interface within the boundaries of the real-time visualization of the user'"'"'s digits, and

display a top layer of the plurality of stacked layers in the foreground of the interface outside of the boundaries of the real-time visualization of the user'"'"'s digits.

13. The apparatus of claim 1, wherein the processor is additionally to:

simulate physical interaction between the displayed object and the user'"'"'s digits by applying a physics effect to the displayed object based on a collision between the displayed object and the real-time visualization of the user'"'"'s digits.

14. The apparatus of claim 13, wherein the physics effect comprises one or more of flicking the displayed object, swiping the displayed object, pushing the displayed object, dragging the displayed object, bouncing the displayed object, and deforming the displayed object.

15. A machine-readable storage medium encoded with instructions executable by a processor of a computing device for enabling multi-touch user interaction with a keyboard, the machine-readable storage medium comprising:

instructions for receiving data from a sensor that detects movement of a user'"'"'s hand on or above a top surface of the keyboard;

instructions for displaying a representation of the user'"'"'s hand on a display of the computing device overlaid on an existing touch interface, the representation updating in real-time as the user moves the hand;

instructions for identifying, in response to a multi-touch gesture of the user'"'"'s digits, an object displayed in the touch interface with which the user has interacted; and

instructions for performing an action corresponding to the multi-touch gesture on the object with which the user has interacted.

16. The machine-readable storage medium of claim 15, wherein:

the touch interface includes windows in a plurality of stacked layers, and

the instructions for displaying are configured to update the touch interface to display a window of a currently-selected layer in a foreground of the interface based on a position of the representation of the user'"'"'s hand within the plurality of stacked layers.

17. The machine-readable storage medium of claim 16, further comprising:

instructions for receiving a user selection of a current layer to be displayed in the foreground, the instructions receiving the user selection based on one or more of;

user activation of one or more predetermined keys on the keyboard,

a distance of the user'"'"'s hand from the top surface of the keyboard, and

a speed of the movement of the user'"'"'s hand parallel to the top surface of the keyboard.

18. A method for enabling indirect manipulation of objects displayed in a multi-touch user interface using a keyboard in communication with a computing device, the method comprising:

using a sensor to detect movement of a user'"'"'s hand in a direction parallel to a top surface of the keyboard;

displaying a representation of the user'"'"'s hand on a display of the computing device overlaid on a touch interface including a plurality of layers, the representation of the user'"'"'s hand updating in real-time as the user moves the hand;

displaying a current layer of the plurality of layers in a foreground of the touch interface in response to a user selection of the current layer; and

performing a mufti-touch gesture on an object in the current layer with which the user has interacted using the representation of the user'"'"'s hand.

19. The method of claim 18, wherein:

the sensor is a wide-angle camera placed above the top surface of the keyboard, and

displaying the representation of the user'"'"'s hand comprises;

normalizing a video representation of the user'"'"'s hand captured by the wide-angle camera to reverse a wide angle effect of the camera,

shifting a perspective of the normalized video representation to a perspective directly above the user'"'"'s hand, and

displaying the normalized and shifted video representation of the user'"'"'s hand.

20. The method of claim 18, wherein the representation of the user'"'"'s hand is a transparent image overlaid on the touch interface.