User interface control using a keyboard

• US 8,471,814 B2
• Filed: 02/26/2010
• Issued: 06/25/2013
• Est. Priority Date: 02/26/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: LaFave, Harper, Hodges, Stephen, Scott, James, Izadi, Shahram, Molyneaux, David, Villar, Nicolas, Butler, David Alexander, Hazas, Mike
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US12/714,295
• Publication Number:
  US 20110210917A1
• Time in Patent Office: 1,215 Days
• Field of Search: 345156-178, 178 1801- 1807, 178 2001- 2004
• US Class Current: 345/160
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/0489 using dedicated keyboard keys or combinations thereof

Abstract

User interface control using a keyboard is described. In an embodiment, a user interface displayed on a display device is controlled using a computer connected to a keyboard. The keyboard has a plurality of alphanumeric keys that can be used for text entry. The computer receives data comprising a sequence of key-presses from the keyboard, and generates for each key-press a physical location on the keyboard. The relative physical locations of the key-presses are compared to calculate a movement path over the keyboard. The movement path describes the path of a user'"'"'s digit over the keyboard. The movement path is mapped to a sequence of coordinates in the user interface, and the movement of an object displayed in the user interface is controlled in accordance with the sequence of coordinates.

Claims

1. A method of controlling a graphical user interface displayed on a display device using a computer connected to a keyboard having a plurality of alphanumeric keys for typing, the method comprising:

receiving, at the computer, data comprising a first sequence of key-presses from a first set of adjacent keys on the keyboard and a second sequence of key-presses from a second set of adjacent keys on the keyboard, wherein the first and second set do not intersect;

generating for each key-press in the first and second sequence a physical location on the keyboard;

comparing the relative physical locations of the first sequence of key-presses to calculate a first movement path of a first touch-point of a user'"'"'s digit over the keyboard;

comparing the relative physical locations of the second sequence of key-presses to calculate a second movement path of a second touch-point of a user'"'"'s digit over the keyboard;

mapping the first movement path to a first sequence of coordinates in the user interface and the second movement path to a second sequence of coordinates in the user interface;

comparing the first and second sequence of coordinates to detect a multi-touch gesture;

controlling movement of an object displayed in the user interface in accordance with the multi-touch gesture.

2. A method according to claim 1, wherein the step of comparing the first and second sequence of coordinates to detect a multi-touch gesture comprises comparing at least one of the distance and angle between the first and second sequence of coordinates.

4. A method according to claim 1, wherein the movement path comprises a touch-point location, and each key-press updates the touch-point location.

5. A method according to claim 4, wherein the data further comprises a sequence of key-release events.

6. A method according to claim 5, wherein each key-release updates the touch-point location.

7. A method according to claim 6, further comprising the step of using the sequence of key-presses and key-release events to detect that a plurality of adjacent keys are pressed concurrently.

8. A method according to claim 7, wherein the step of comparing the relative physical locations of the key-presses to calculate a movement path over the keyboard comprises, in the case that adjacent keys are pressed concurrently, interpolating between the physical locations of the plurality of adjacent keys to create an interpolated location, and updating the touch-point location to the interpolated location.

9. A method according to claim 7, wherein the step of comparing the relative physical locations of the key-presses to calculate a movement path over the keyboard comprises, in the case that adjacent keys are pressed concurrently, interpolating between the physical location of the key most recently pressed and the touch-point location to create an interpolated location, and updating the touch-point location to the interpolated location.

10. A method according to claim 1, further comprising the step of comparing the sequence of coordinates to a further independent sequence of coordinates to detect a multi-touch gesture, and the step of controlling movement of the object comprises controlling movement of the object displayed in the user interface in accordance with the multi-touch gesture.

11. A method according to claim 10, wherein the data further comprises a further sequence of key-presses on the keyboard, and the further independent sequence of coordinates is generated from the further sequence of key-presses.

12. A method according to claim 11, wherein the further sequence of key-presses and sequence of key-presses in the data do not overlap.

13. A method according to claim 1, further comprising the step of detecting that all key-presses have corresponding key-release events, and resetting the movement path by deleting the touch-point location.

14. A method according to claim 1, further comprising the step of detecting that neither a key-press event or key-release event has occurred for a predetermined time period, and resetting the movement path by deleting the touch-point location.

3. A method of controlling a user interface displayed on a display device using a computer connected to a keyboard having a plurality of alphanumeric keys for text entry, the method comprising:

receiving, at the computer, data comprising a sequence of key-presses from the keyboard;

generating for each key-press in the sequence a physical location on the keyboard;

comparing the relative physical locations of the key-presses to calculate a movement path over the keyboard;

mapping the movement path to a sequence of coordinates in the user interface; and

controlling movement of an object displayed in the user interface in accordance with the sequence of coordinates.

15. A computer system, comprising:

a display device;

a keyboard having a plurality of alphanumeric keys for text entry; and

a processor arranged to display a user interface on the display device, receive data from the keyboard comprising a sequence of key-presses from a user, generate for each key-press in the sequence a physical location on the keyboard, compare the relative physical locations of the key-presses to calculate a movement path of the user'"'"'s digit over the keyboard, map the movement path to a sequence of coordinates in the user interface, and control movement of an object displayed in the user interface in accordance with the sequence of coordinates.

16. A computer system according to claim 15, wherein the processor is further arranged to display a representation of the keyboard on the display device overlaid on the user interface, and indicate the movement path on the representation.

17. A computer system according to claim 16, wherein the computer system further comprises a pointing device, and the processor is arranged to move the representation responsive to input from the pointing device, such that the representation can be moved over a selected object displayed in the user interface to be controlled by the keyboard.

18. A computer system according to claim 15, wherein the computer system further comprises a data store, and the processor is arranged generate for each key-press in the sequence a physical location on the keyboard by determining the keyboard layout and accessing a table stored on the data store mapping each key to a location for the layout.

19. A computer system according to claim 15, wherein the object displayed in the user interface is at least one of:

a cursor;

a window;

an image;

a web-page;

a document; and

a video.

20. A computer system according to claim 15, wherein the keyboard is a QWERTY keyboard.