Indirect multi-touch interaction

• US 9,513,798 B2
• Filed: 10/01/2009
• Issued: 12/06/2016
• Est. Priority Date: 10/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Villar, Nicolas, Helmes, John, Benko, Hrvoje, Izadi, Shahram, Rosenfeld, Daniel, Hodges, Stephen, Butler, David Alexander, Cao, Xiang, Banks, Richard
• Primary Examiner(s): ABDIN, SHAHEDA A
• Application Number:
  US12/571,649
• Publication Number:
  US 20110080341A1
• Time in Patent Office: 2,623 Days
• Field of Search: 345/156, 345/157, 345/174, 345/173, 715/702, 715/800
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Indirect multi-touch interaction is described. In an embodiment, a user interface is controlled using a cursor and a touch region comprising a representation of one or more digits of a user. The cursor and the touch region are moved together in the user interface in accordance with data received from a cursor control device, such that the relative location of the touch region and the cursor is maintained. The representations of the digits of the user are moved in the touch region in accordance with data describing movement of the user'"'"'s digits. In another embodiment, a user interface is controlled in a first mode of operation using an aggregate cursor, and switched to a second mode of operation in which the aggregate cursor is divided into separate portions, each of which can be independently controlled by the user.

Claims

1. A computer-implemented method of controlling a user interface displayed on a display device, comprising:

receiving a first data sequence describing movement of a cursor control device relative to a non-display surface, the cursor control device being separate from the display device and operable by a user;

receiving a second data sequence describing movement of a digit of the user;

receiving an indication of a selection of a fixed mode or a relative mode;

displaying in the user interface a cursor and a touch region comprising a representation of the digit of the user;

processing the first data sequence such that, when the fixed mode is selected, movement of the cursor control device moves the cursor independent of the touch region in the user interface and, when the relative mode is selected, movement of the cursor control device moves the cursor and the touch region in the user interface, whilst maintaining the location of the touch region relative to the cursor; and

processing the second data sequence such that movement of the digit by the user moves the representation of the digit in the touch region.

2. A method according to claim 1, further comprising the step of:

analyzing the movement of the representation of the digit relative to the cursor to detect a user gesture.

3. A method according to claim 2, further comprising the step of:

responsive to detecting the user gesture, selecting an action in dependence on the user gesture and executing the action.

4. A method according to claim 3, further comprising the step of:

applying the action to an object selected in the user interface by the cursor.

5. A method according to claim 3, wherein the action comprises at least one of:

rotating an object;

scaling an object;

selecting at least one object; and

displaying a menu.

6. A method according to claim 1, wherein the second data sequence further describes movement of at least one further digit of the user, the touch region further comprises a representation of the or each further digit, and the method further comprises the step of processing the second data sequence such that movement of the further digit by the user moves the respective representation in the touch region relative to the cursor and other representations in the touch region.

7. A method according to claim 6, further comprising the step of:

analyzing the relative movement of each representation in the touch region to detect a user gesture.

8. A method according to claim 1, wherein the cursor control device is a multi-touch, non-virtual mouse device arranged to sense movement of a base portion of the multi-touch, non-virtual mouse device over a supporting surface and sense movement of one or more digits of the user of the multi-touch, non-virtual mouse device relative to the base portion, and wherein the first data sequence describes the movement of the base portion, and the second data sequence describes movement of the digit of the user relative to the base portion.

9. A method according to claim 1, wherein the cursor control device is a physical mouse device, and the first data sequence describes the movement of the physical mouse device over a supporting surface.

10. A method according to claim 1, wherein the cursor control device is a touch pad, and the first data sequence describes the movement of a contact point of the user on the touch pad.

11. A method according to claim 1, wherein the second data sequence is provided by a touch pad arranged to sense movement of at least one digit of the user over the touch pad.

12. A method according to claim 1, wherein the representation is displayed in proximity to a control point of the cursor.

13. A method according to claim 1, wherein the touch region surrounds a control point of the cursor.

14. A computer-implemented method of controlling a user interface displayed on a display device, comprising:

receiving data from at least one pointing device, the data describing movement of a plurality digits of a user; and

switching operation of the user interface from a first mode of operation in which the data is used to control movement of an aggregate cursor displayed in the user interface to a second mode of operation in which the aggregate cursor is divided into a plurality of separate portions and the data is used to control movement of each portion independently, switching between the first and second mode based at least in part on a multi-touch input mode being available to the user.

15. A method according to claim 14, wherein switching between the first and second mode comprises changing at least one color of at least one portion of the aggregate cursor.

16. A method according to claim 14, wherein switching between the first and second mode comprises analyzing the relative movement of the plurality of digits to determine an appropriate mode of operation.

17. A method according to claim 14, wherein switching between the first and second mode comprises receiving and processing a control signal from the user indicating a selected mode of operation.

18. A method according to claim 14, wherein the pointing device is at least one of:

a multi-touch touch pad; and

a multi-touch mouse device.

19. A method according to claim 14, further comprising the steps of:

analyzing the movement of portions relative to each other to detect a user gesture;

responsive to detecting the user gesture, selecting an action in dependence on the user gesture; and

executing the action.

20. A computer system, comprising:

a display device;

a communication interface arranged to receive a first and second data sequence from a multi-touch mouse device operable by a user, the first data sequence describing movement of a base portion of the multi-touch mouse device, and the second data sequence describing movement of a plurality of digits of the user of the multi-touch mouse device relative to the base portion; and

a processor arranged to display a user interface on the display device, display in the user interface a cursor and a touch region comprising a representation for each of the digits of the user, process the first data sequence such that movement of the base portion moves the cursor and the touch region in the user interface, whilst maintaining the location of the touch region relative to the cursor, and process the second data sequence such that movement of one of the digits moves the respective representation in the touch region relative to the cursor, movement of the respective representation of the digit comprising leaving a trail on the display device indicating at least a portion of the movement of the respective representation.