Method for gesture control

• US 20140320434A1
• Filed: 04/24/2014
• Published: 10/30/2014
• Est. Priority Date: 04/26/2013
• Status: Active Grant

• Current Assignee: Klaus Pantel
• Original Assignee: Inodyn Newmedia GmbH
• Inventors: Pantel, Lothar
• Granted Patent: US 9,323,340 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A method for gesture-based control of a portable, electronic device (1) with a touchscreen (2). Novel gestures are introduced which are compatible with conventional touchscreen gestures and can be used simultaneously. Contrary to custom, the user holds the finger (101) or any other input object in a steady manner and moves, displaces, or rotates the electronic device (1) below the unmoved finger (101) or input object. In one embodiment, the user moves a smartphone (1) or a smartwatch toward or away from the unmoved finger (101) to establish or terminate contact with the touchscreen (2). In another embodiment, the user drags or rotates the smartphone (1) below the unmoved finger (101). An acceleration sensor and/or a gyroscope are used for capturing the movements of the electronic device (1).

Claims

1. A method for gesture-based control of a portable electronic device (1, 26), comprising:

a) providing a sensor device (3, 4, 5) which is able to measure the acceleration (16), the velocity (17), the position shift (18), and/or the rotation (19, 50) of said portable electronic device (1, 26), and a touch-sensitive surface (2), which is able to detect at least one finger (101) or input object (111),

b) maintaining said finger (101) or input object (111) steady,

c) moving said portable electronic device (1, 26) toward or away from the substantially unmoved finger (101) or input object (111) to establish or terminate contact with said touch-sensitive surface (2) or shifting or rotating said portable electronic device (1, 26) below said substantially unmoved finger (101) or input object (111), said substantially unmoved finger (101) or input object (111) sliding on said touch-sensitive surface (2),

d) processing the values of said acceleration (16), velocity (17), position shift (18), and/or rotation (19, 50) captured by said sensor device (3, 4, 5), as soon as said touch-sensitive surface (2) detects a relevant input by said finger (101) or input object (111),

e) triggering a corresponding gesture event (21) or action in case a gesture sequence is recognized.

2. The method according to claim 1, further comprising continual buffering of the values captured by said sensor device (3, 4, 5) in a buffer (11), so that said buffer (11) always contains the values of the most recent past, accessing the values stored in said buffer (11) as soon as said touch-sensitive surface (2) detects a relevant input by said finger (101) or input object (111).

3. The method according to claim 1, wherein the values captured by said sensor device (3, 4, 5) are evaluated to distinguish by means of a decision criterion whether said portable electronic device (1, 26) is moving or essentially steady in the period of time or at a time point in the temporal proximity of the beginning of contact (27) and/or the end of contact (28) of said finger (101) or input object (111) with said touch-sensitive surface.

4. The method according to claim 1, further comprising the steps of:

a) touching said touch-sensitive surface (2) with said finger (101) or input object (111) while holding said portable electronic device (1, 26) substantially steady in place, the beginning of contact (27) being detected by said touch-sensitive surface (2);

b) keeping the current position of said finger (101) or input object (111) in a steady manner;

c) moving said portable electronic device (1, 26) approximately perpendicularly to said touch-sensitive surface (2), in the direction (106) of the rear side of said touch-sensitive surface (2), away from the unmoved finger (101) or input object (111), the end of contact (28) being detected by said touch-sensitive surface (2);

d) stopping the movement of said portable electronic device (1, 26) after reaching a desired distance (7);

e) triggering a corresponding gesture event (21) or action, at the latest, after stopping said portable electronic device (1, 26).

5. The method according to claim 1, further comprising the steps of:

a) holding said finger (101) or input object (111) substantially steady in place at a desired distance (8) apart from said touch-sensitive surface (2);

b) moving said portable electronic device (1, 26) approximately perpendicularly to said touch-sensitive surface (2), in the direction (105) of the front side of said touch-sensitive surface (2), toward the unmoved finger (101) or input object (111) until said touch-sensitive surface (2) touches said unmoved finger (101) or input object (111), the beginning of contact (27) being detected by said touch-sensitive surface (2);

c) removing said finger (101) or input object (111) from said touch-sensitive surface (2) while holding said portable electronic device (1, 26) substantially steady in place, the end of contact (28) being detected by said touch-sensitive surface (2);

d) triggering a corresponding gesture event (21) or action.

6. The method according to claim 1, further comprising the steps of:

a) holding said finger (101) or input object (111) substantially steady in place at a desired distance (8) apart from said touch-sensitive surface (2);

b) moving said portable electronic device (1, 26) approximately perpendicularly to said touch-sensitive surface (2), in the direction (105) of the front side of said touch-sensitive surface (2), toward the unmoved finger (101) or input object (111) until said touch-sensitive surface (2) touches said unmoved finger (101) or input object (111), the beginning of contact (27) being detected by said touch-sensitive surface (2);

c) keeping the current position of said finger (101) or input object (111) in a steady manner;

d) moving said portable electronic device (1, 26) approximately perpendicularly to said touch-sensitive surface (2), in the direction (106) of the rear side of said touch-sensitive surface (2), away from said unmoved finger (101) or input object (111), the end of contact (28) being detected by said touch-sensitive surface (2);

e) stopping the movement of said portable electronic device (1, 26) after reaching a desired distance (7);

f) triggering a corresponding gesture event (21) or action, at the latest, after stopping said portable electronic device (1, 26).

7. The method according to claim 1, wherein a setting value is adjustable in both directions, as follows:

a) holding said finger (101) or input object (111) substantially steady in place at a desired distance (8) apart from said touch-sensitive surface (2) and moving said portable electronic device (1, 26) toward the unmoved finger (101) or input object (111) until said touch-sensitive surface (2) touches said unmoved finger (101) or input object (111) and, at the latest when said touch-sensitive surface (2) detects the beginning of contact (27), changing said setting value in correlation to the determined distance (8) in the first direction;

or

b) touching said touch-sensitive surface (2) with said finger (101) or input object (111) while holding said portable electronic device (1, 26) substantially steady in place, and subsequently moving said portable electronic device (1, 26) away from the unmoved finger (101) or input object (111), stopping the movement after reaching a desired distance (7) and, at the latest, when the stop is recognized, changing said setting value in correlation to the determined distance (7) in the second direction.

8. The method according to claim 1, wherein an onscreen keyboard (32) is displayed on said touch-sensitive surface (2) and wherein, proceeding from a pressed key from said onscreen keyboard (32), a lowercase letter is entered, in case said portable electronic device (1, 26) is kept substantially steady while removing said finger (101) or input object (111) from said pressed key and otherwise, proceeding from said pressed key, an uppercase letter is entered, in case said portable electronic device (1, 26) is moved away from the unmoved finger (101) or input object (111), essentially in the direction (106) of the rear side of said touch-sensitive surface (2), terminating contact with the key.

9. The method according to claim 1, further comprising a decision criterion which evaluates whether the acceleration (16), velocity (17), and/or position shift (18) of said portable electronic device (1, 26) takes place in a direction approximately perpendicular to said touch-sensitive surface (2).

10. The method according to claim 1, further comprising determining the velocity (17) of said portable electronic device (1, 26) by integrating the acceleration (16) measured by said sensor device (3) and evaluating the magnitude and/or direction of said velocity (17) from a time point shortly before the beginning of contact (27) detected by said touch-sensitive surface (2), and/or from a time point shortly after the end of contact (28) detected by said touch-sensitive surface (2).

11. The method according to claim 1, further comprising calculating the distance (8) from the position shift (18) between the beginning of the gesture sequence and the beginning of contact (27) detected by said touch-sensitive surface (2), and/or calculating the distance (7) from the position shift (18) between the end of contact (28) detected by said touch-sensitive surface (2) and the end of the gesture sequence, the respective distance (7, 8) being handed over as an input parameter.

12. The method according to claim 1, wherein said portable electronic device (1, 26) is shifted below the substantially unmoved finger (101) or input object (111) essentially in the plane (109) of said touch-sensitive surface (2), while, in the course of this displacement, said unmoved finger (101) or input object (111) remains in contact with said touch-sensitive surface (2) and is held in place so that, as a result of the relative displacement, a dragging movement (41, 42, 43) of said finger (101) or input object (111) over said touch-sensitive surface (2) is detected by said touch-sensitive surface (2).

13. The method according to claim 1, further comprising the steps of:

a) providing contact of said finger (101) or input object (111) with said touch-sensitive surface, said portable electronic device (1, 26) and said finger (101) or input object (111) substantially being unmoved;

b) shifting said portable electronic device (1, 26) below said unmoved finger (101) or input object (111) essentially in the plane (109) of said touch-sensitive surface (2) while maintaining continual contact of said finger (101) or input object (111) with said touch-sensitive surface (2);

c) removing said finger (101) or input object (111) from said touch-sensitive surface (2) while said portable electronic device (1, 26) is in full motion or alternatively, shifting said portable electronic device (1, 26) below said unmoved finger (101) or input object (111) beyond the edge of said touch-sensitive surface (2) such that said finger (101) or input object (111) loses contact with said touch-sensitive surface (2) at the edge;

d) stopping the movement of said portable electronic device (1, 26) as soon as said finger (101) or input object (111) has left said touch-sensitive surface (2).

14. The method according to claim 1, further comprising a decision criterion which evaluates whether the acceleration (16), velocity (17), and/or position shift (18) of said portable electronic device (1, 26) takes place approximately in the plane (109) of said touch-sensitive surface (2).

15. The method according to claim 1, further comprising a decision criterion which evaluates to what extent the direction (49, 52) of the dragging movement of said finger (101) or input object (111) over said touch-sensitive surface (2) and the direction of the acceleration (16), velocity (17), and/or position shift (18) of said portable electronic device (1, 26) are approximately parallel and pointing in opposite directions.

16. The method according to claim 1, further comprising determining the velocity (17) of said portable electronic device (1, 26) by integrating the acceleration (16) measured by said sensor device (3) and evaluating the magnitude and/or direction of said velocity (17) at the time point at which a dragging movement (41, 43) of said finger (101) or input object (111) is detected and/or at the time point at which the end of contact (28, 43, 45) is detected by said touch-sensitive surface (2).

17. The method according to claim 1, wherein said touch-sensitive surface (2) is curved around a forearm (102), said portable electronic device (26) being operated by the steps of:

a) touching said touch-sensitive surface (2) with said finger (101) or input object (111), the beginning of contact (27) being detected by said touch-sensitive surface (2);

b) maintaining said finger (101) or input object (111) steady;

c) twisting said forearm (102) below said unmoved finger (101) or input object (111) while maintaining continual contact, so that said unmoved finger (101) or input object (111) slides over said touch-sensitive surface (2);

d) measuring the rotation (50) of said forearm (102) by means of said sensor device (4, 5);

e) stopping the rotation of said forearm (102);

f) removing said finger (101) or input object (111) from said touch-sensitive surface (2), the end of contact (28) being detected by said touch-sensitive surface (2);

g) triggering a corresponding gesture event (21) or action in case the angle of rotation (50) between said beginning of contact (27) and said end of contact (28) and/or the speed of the rotation meets a corresponding decision criterion.

18. The method according to claim 1, further comprising a decision criterion which evaluates to what extent the speed of the sliding finger (101) or input object (111) detected by said touch-sensitive surface (2) and the velocity (17) or rotational speed of said portable electronic device (1, 26) suit each other, a tolerance being permitted.

19. The method according to claim 1, further comprising the steps of:

a) touching said touch-sensitive surface (2) with said finger (101) or input object (111), the beginning of contact (27) being detected by said touch-sensitive surface (2);

b) rotating said portable electronic device (1) below the substantially unmoved finger (101) or input object (111) around a rotation axis (57) approximately perpendicular to said touch-sensitive surface (2) while maintaining continual contact, the contact point of said finger (101) or input object (111) on said touch-sensitive surface (2) constituting the pivot point (53);

c) increasing, decreasing, or changing an adjustable value (54) or property in correlation to the rotation (19) around said rotation axis (57), measured by said sensor device (4, 5), as long as said contact point of said finger (101) or input object (111) is detected by said touch-sensitive surface (2);

d) retaining the last adjusted value (54) or the last configured property as soon as said finger (101) or input object (111) is removed from said touch-sensitive surface (2), the end of contact (28) being detected by said touch-sensitive surface (2).

20. The method according to claim 19, wherein an additional contact point of an additional finger (103) or input object (111) detected by said touch-sensitive surface (2) results in a reduction of the conversion ratio between the rotation of said portable electronic device (1) around said rotation axis (57) and said adjustable value (54) or property.

21. The method according to claim 19, wherein the portion of said touch-sensitive surface (2) which allows the modification of said adjustable value (54) or property is visualized by means of a display as a control dial (64, 68, 69), a circle (56) surrounding said pivot point (53), a color-selection element (61, 67), and/or a numeric value (54, 63), the rotation of said portable electronic device (1) around said rotation axis (57) resulting in changes to the pointer position of said control dial (64, 68, 69), changes to the size of said circle (56), changes to the selected color (62) of said color-selection element (61, 67), and/or changes to the numeric value (54, 63), these changes being visualized respectively.

22. The method according to claim 19, further comprising an overturn protection, which in case an upper or lower limit of said adjustable value (54) or property is exceeded as the result of the rotation of said portable electronic device (1) around said rotation axis (57), rotates the entire screen content (65), visualized on said touch-sensitive surface (2) by means of a display, by an angle opposite to the over-rotation determined by said sensor device (4, 5), such that the opposite rotation angles offset each other.

23. The method according to claim 19, wherein, as soon as the beginning of contact (27) of said finger (101) or input object (111) is detected by said touch-sensitive surface, the rotation of said portable electronic device (1) around said rotation axis (57) is continuously monitored by means of said sensor device (4, 5), and as soon as the angle of rotation (50) since the beginning of contact (27) exceeds a threshold value, said adjustable value (54) or property is increased, decreased, or changed, and at the end of contact (28), a gesture event (21) is triggered which indicates the change of said adjustable value (54) or property, and otherwise in case the angle of rotation (50) between said beginning of contact (27) until the end of contact (28) is below said threshold value, a gesture event (21) is triggered which indicates a different gesture.

24. The method according to claim 1, wherein, depending on the recognized gesture sequence, the gesture event (21) or action executes one of the following tasks:

a) aborting or suspending a running process or operation,

b) undoing an action,

c) deleting an element or object,

d) changing or sorting the order of elements, objects, pages, cards, or tabs,

e) locking or unlocking the scroll capability of a page, card, or tab,

f) terminating a running application program or switching to another application program,

g) minimizing or maximizing a running application program,

h) moving a running application program to the background or foreground,

i) returning to a main menu or home screen,

j) executing a search function or displaying an input screen for a search function,

k) entering an uppercase letter,

l) accepting a word or text suggested by an auto-complete function,

m) taking a telephone call or terminating a telephone call,

n) enabling or disabling a lock screen,

o) jumping to the next or previous paragraph, page, or chapter of a text or document,

p) jumping to the beginning or end of a text or document,

q) leaping forward or backward during the playback of an audio or video recording,

r) skipping the current, the next, or the previous recording in a playlist,

s) setting a volume or a brightness.