USE OF A SENSOR TO ENABLE TOUCH AND TYPE MODES FOR HANDS OF A USER VIA A KEYBOARD

• US 20130257734A1
• Filed: 03/30/2012
• Published: 10/03/2013
• Est. Priority Date: 03/30/2012
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Marti, Stefan J., Kim, Seung Wook, Di Censo, Davide, Liu, Eric
• Application Number:
  US13/435,734
• Publication Number:
  US 20130257734A1
• Time in Patent Office: Days
• US Class Current: 345/168
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0213 Arrangements providing an integrated pointing device in a keyboard, e.g. trackball, mini-joystick for pointing devices integrated in a laptop computer G06F1/169; joysticks G05G9/047; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886

Abstract

Example embodiments relate to a keyboard-based system that enables a user to provide touch input via the keyboard with one hand and typed input via the keyboard with the other hand. In example embodiments, a sensor detects the user'"'"'s hands on the top surface of the keyboard. In response, a computing device identifies a first hand and a second hand by analyzing information provided by the sensor. The computing device then assigns the first hand to a touch mode and the second hand to a typing mode. The user may then provide touch input using a visualization of the first hand overlaid on a user interface, while providing typed input with the second hand via the keyboard.

Claims

1. An apparatus comprising:

a keyboard;

a sensor for detecting a user'"'"'s hands on or above a top surface of the keyboard;

a display device; and

a processor to;

identify a first hand of the user and a second hand of the user by analyzing information provided by the sensor,

assign the first hand to a touch mode and the second hand to a typing mode,

output a real-time visualization of the user'"'"'s first hand overlaid on a touch user interface on the display device to enable the user to perform touch commands by moving the first hand on or above the top surface of the keyboard, and

receive typed input provided to the keyboard with the user'"'"'s second hand.

2. The apparatus of claim 1, wherein the sensor comprises one or more of a camera placed above the top surface of the keyboard, a capacitive touch sensor, an infrared sensor, and an electric field sensor.

3. The apparatus of claim 1, wherein:

the sensor includes a camera, and

the processor is additionally to output the real-time visualization of the user'"'"'s first hand as a video representation of the user'"'"'s first hand obtained by the camera.

4. The apparatus of claim 1, wherein, to identify the first hand and the second hand, the processor is additionally to:

analyze the information provided by the sensor to obtain an isolated image of each of the user'"'"'s hands,

generate an approximation of a shape of each of the user'"'"'s hands using each isolated image, and

detect a direction of the user'"'"'s thumb on each hand in each approximation to identify a left hand and a right hand of the user.

5. The apparatus of claim 1, wherein, to identify the first hand and the second hand, the processor is additionally to:

analyze the information provided by the sensor to obtain an isolated image of each of the user'"'"'s hands,

compare each of the isolated images to known images of left hands and right hands, and

identify a left hand of the user and a right hand of the user based on a similarity of each isolated image to the known images.

6. The apparatus of claim 1, wherein the processor is additionally to:

toggle the user'"'"'s first hand between the touch mode and the typing mode in response to a first predetermined input, and

toggle the user'"'"'s second hand between the touch mode and the typing mode in response to a second predetermined input.

7. The apparatus of claim 6, wherein the first predetermined input is a first key on the keyboard and the second predetermined input is a second key on the keyboard.

8. The apparatus of claim 6, wherein the first predetermined input is a gesture using the first hand and the second predetermined input is a gesture using the second hand.

9. The apparatus of claim 1, wherein:

the touch user interface includes windows in a plurality of stacked layers, and

the processor is additionally to update the touch user interface to display a window of a currently-selected layer in a foreground of the interface based on a position of the real-time visualization within the plurality of stacked layers.

10. The apparatus of claim 9, wherein the processor is additionally to:

modify the currently-selected layer of the plurality of stacked layers based on at least one of;

a user selection of a predetermined key on the keyboard,

a distance of the user'"'"'s first hand from the top surface of the keyboard, and

a speed of movement of the user'"'"'s first hand in a direction parallel to the top surface of the keyboard.

11. The apparatus of claim 9, wherein, in outputting the real-time visualization, the processor is additionally to:

identify a plurality of portions of the real-time visualization of the user'"'"'s first hand, each portion intersecting a respective layer of the plurality of stacked layers, and

apply a unique visualization to each identified portion of the plurality of portions of the real-time visualization.

12. The apparatus of claim 9, wherein, the processor is additionally to:

display the currently-selected layer in the foreground of the interface within the boundaries of the real-time visualization of the user'"'"'s first hand, and

display a top layer of the plurality of stacked layers in the foreground of the interface outside of the boundaries of the real-time visualization of the user'"'"'s first hand.

13. The apparatus of claim 1, wherein the processor is additionally to:

simulate physical interaction between a displayed object and the user'"'"'s first hand by applying a physics effect to the displayed object based on a collision between the displayed object and the real-time visualization of the user'"'"'s first hand.

14. A machine-readable storage medium encoded with instructions executable by a processor of a computing device for enabling touch interaction via a keyboard, the machine-readable storage medium comprising:

instructions for receiving data from a sensor that detects movement and a position of a user'"'"'s hands on or above a top surface of the keyboard;

instructions for identifying a left hand of the user and a right hand of the user by analyzing the data received from the sensor;

instructions for assigning a first hand of the user'"'"'s hands to a touch mode and a second hand of the user'"'"'s hands to a typing mode;

instructions for displaying a visualization of the user'"'"'s first hand on a display of the computing device overlaid on an existing touch interface to enable touch input via movement of the first hand; and

instructions for receiving typed input provided to the keyboard with the user'"'"'s second hand.

15. The machine-readable storage medium of claim 14, wherein the instructions for identifying are configured to:

analyze the data received from the sensor to obtain an isolated image of each of the user'"'"'s hands,

generate an approximation of a shape of each of the user'"'"'s hands using each isolated image, and

detect a direction of the user'"'"'s thumb on each hand in each approximation to identify the left hand and the right hand.

16. The machine-readable storage medium of claim 14, wherein the instructions for identifying are configured to:

analyze the data received from the sensor to obtain an isolated image of each of the user'"'"'s hands,

compare each of the isolated images to known images of left hands and right hands, and

identify the left hand and the right hand based on a similarity of each isolated image to the known images.

17. The machine-readable storage medium of claim 14, wherein the instructions for assigning are configured to assign the user'"'"'s hands to the touch mode and the typing mode based on a user specification of a mode for the left hand and a mode for the right hand.

18. A method for enabling touch and text input using a keyboard in communication with a computing device, the method comprising:

using a sensor to obtain sensor data regarding a user'"'"'s hands on the keyboard;

identifying a first hand of the user and a second hand of the user by analyzing the sensor data;

assigning the first hand to a touch mode and the second hand to a typing mode;

outputting a visualization of the user'"'"'s first hand on the display device, wherein the visualization of the user'"'"'s first hand is overlaid on a touch interface of an application; and

forwarding, to the application, typed input provided to the keyboard with the user'"'"'s second hand.

19. The method of claim 18, wherein the assigning comprises:

assigning the user'"'"'s hands to the touch mode and the typing mode based on a user specification of a mode for each hand.

20. The method of claim 18, wherein identifying the first hand and the second hand comprises at least one of:

identifying a left hand and a right hand of the user by analyzing the sensor data to obtain an image of each hand and to detect a direction of the user'"'"'s thumb in each image, and

identifying the left hand and the right hand of the user by analyzing the sensor data with reference to known images of left hands and right hands.