Method for gesture control

• US 9,323,340 B2
• Filed: 04/24/2014
• Issued: 04/26/2016
• Est. Priority Date: 04/26/2013
• Status: Active Grant

• Current Assignee: Klaus Pantel
• Original Assignee: Inodyn Newmedia GmbH
• Inventors: Pantel, Lothar
• Primary Examiner(s): Sasinowski, Andrew
• Application Number:
  US14/261,385
• Publication Number:
  US 20140320434A1
• Time in Patent Office: 733 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A method for gesture-based control of a portable, electronic device with a touch-sensitive surface. Novel gestures are introduced that may be compatible with conventional touchscreen gestures. Contrary to custom, the user may hold a finger or any other input object in a steady manner and may move, shift, and/or rotate the electronic device below the unmoved finger or input object. In one embodiment, the user moves a smartphone or a smartwatch toward or away from the unmoved finger to establish or terminate contact with the touchscreen. In another embodiment, the user drags or rotates the smartphone below the unmoved finger such that the unmoved finger slides on the touchscreen. An acceleration sensor and/or a gyroscope may be used for capturing the movements of the electronic device.

Claims

1. A method for the recognition of gesture sequences consisting of at least one gesture part, comprising:

a) providing a portable electronic device with at least one sensor device comprising an accelerometer, a gyroscope, a magnetometer, or a combination thereof, and with at least one touch-sensitive surface that is able to detect at least one finger,

b) processing values captured at least in part by said sensor device, at least as soon as said touch-sensitive surface detects a relevant input by said at least one finger,

c) discerning

gesture parts based on moving said portable electronic device while said at least one finger remains substantially unmoved

from

gesture parts based on changing position of said at least one finger while said portable electronic device remains substantially unmoved,

and

d) invoking an assigned function in case a gesture sequence is recognized.

2. The method of claim 1, further comprising continually buffering of at least a part of the values captured by said sensor device in a buffer, such that said buffer always contains the values of a period of time from the recent past up until to the present, accessing the values stored in said buffer as soon as said touch-sensitive surface detects a relevant input by said finger.

3. The method of claim 1, wherein at least a part of the values captured by said sensor device are evaluated to distinguish by means of a decision criterion whether said portable electronic device is moving or substantially unmoved at least at a time point in the temporal proximity of a relevant input by said finger, the input being detected by said touch-sensitive surface.

4. The method of claim 3, further comprising:

a) determining the velocity of said portable electronic device by integrating the acceleration measured by said accelerometer,

b) evaluating the velocity of said portable electronic device taken at least from a time point shortly before the beginning of contact or shortly after the end of contact, said contact being detected by said touch-sensitive surface.

5. The method of claim 1, further comprising a decision criterion that evaluates whether the movement of said portable electronic device takes place in a direction approximately perpendicular to said touch-sensitive surface.

6. The method of claim 1, further comprising the steps of:

a) touching said touch-sensitive surface with said finger while holding said portable electronic device substantially steady in place, the beginning of contact being detected by said touch-sensitive surface;

b) keeping the current position of said finger in a steady manner;

c) moving said portable electronic device approximately perpendicularly to said touch-sensitive surface, in the direction of the rear side of said touch-sensitive surface, away from said substantially unmoved finger, the end of contact being detected by said touch-sensitive surface;

d) stopping the movement of said portable electronic device after reaching a distance;

e) invoking an assigned function, at the latest, after stopping said portable electronic device.

7. The method of claim 6, further comprising:

a) determining, at least in part by means of said sensor device, a distance from the change in position of said portable electronic device between said end of contact detected by said touch-sensitive surface and the end of the movement of said portable electronic device,

b) handing over an input parameter derived at least in part from said distance.

8. The method of claim 1, further comprising the steps of:

a) holding said finger substantially steady in place at a distance apart from said touch-sensitive surface;

b) moving said portable electronic device approximately perpendicularly to said touch-sensitive surface, in the direction of the front side of said touch-sensitive surface, toward said substantially unmoved finger until said touch-sensitive surface touches said substantially unmoved finger, the beginning of contact being detected by said touch-sensitive surface;

c) removing said finger from said touch-sensitive surface while holding said portable electronic device substantially steady in place, the end of contact being detected by said touch-sensitive surface;

d) invoking an assigned function.

9. The method of claim 8, further comprising:

a) determining, at least in part by means of said sensor device, a distance from the change in position of said portable electronic device between the beginning of the movement of said portable electronic device and said beginning of contact detected by said touch-sensitive surface,

b) handing over an input parameter derived at least in part from said distance.

10. The method of claim 1, further comprising the steps of:

a) holding said finger substantially steady in place at a distance apart from said touch-sensitive surface;

b) moving said portable electronic device approximately perpendicularly to said touch-sensitive surface, in the direction of the front side of said touch-sensitive surface, toward said substantially unmoved finger until said touch-sensitive surface touches said substantially unmoved finger, the beginning of contact being detected by said touch-sensitive surface;

c) keeping the current position of said finger in a steady manner;

d) moving said portable electronic device approximately perpendicularly to said touch-sensitive surface, in the direction of the rear side of said touch-sensitive surface, away from said substantially unmoved finger, the end of contact being detected by said touch-sensitive surface;

e) stopping the movement of said portable electronic device after reaching a distance;

f) invoking an assigned function, at the latest, after stopping said portable electronic device.

11. The method of claim 10, further comprising:

a) determining, at least in part by means of said sensor device, at least

i) a first distance from the change in position of said portable electronic device between the beginning of the movement of said portable electronic device and said beginning of contact detected by said touch-sensitive surface, or

ii) a second distance from the change in position of said portable electronic device between said end of contact detected by said touch-sensitive surface and the end of the movement of said portable electronic device,

b) handing over at least one input parameter derived at least in part from said first distance, said second distance, or a combination thereof.

12. The method of claim 1, wherein at least one setting value is adjustable in a first and second direction, comprising:

a) changing said setting value in the first direction by keeping said finger substantially unmoved at a distance apart from said touch-sensitive surface, and subsequently moving said portable electronic device toward said substantially unmoved finger until said touch-sensitive surface touches said substantially unmoved finger, or

b) changing said setting value in the second direction by touching said touch-sensitive surface with said finger while keeping said portable electronic device substantially unmoved, and subsequently moving said portable electronic device away from said finger while keeping said finger substantially unmoved such that said substantially unmoved finger loses contact with said touch-sensitive surface.

13. The method of claim 12, further comprising changing said setting value in dependence on a distance derived from the change in position of said portable electronic device, said distance being determined at least in part by means of said sensor device.

14. The method of claim 12, wherein said setting value controls at least in part:

a) the zoom-level of content displayed on a display,

b) the size of content displayed on a display,

c) the scale of content displayed on a display,

d) the brightness of content displayed on a display, or

e) a combination thereof.

15. The method of claim 1, wherein an onscreen keyboard is displayed on said touch-sensitive surface and wherein, proceeding from a pressed key on said onscreen keyboard, a lowercase letter is entered, in case said portable electronic device is kept substantially unmoved while removing said finger from said pressed key and otherwise, proceeding from said pressed key, an uppercase letter is entered, in case said portable electronic device is moved away from the substantially unmoved finger, essentially in the direction of the rear side of said touch-sensitive surface, terminating contact with said pressed key.

16. The method of claim 1, further comprising a decision criterion that evaluates whether the movement of said portable electronic device takes place approximately in the plane of said touch-sensitive surface.

17. The method of claim 1, further comprising:

a) providing contact of said finger with said touch-sensitive surface,

b) keeping said finger substantially unmoved,

c) shifting said portable electronic device below said substantially unmoved finger essentially in the plane of said touch-sensitive surface, while maintaining contact with said substantially unmoved finger, such that said substantially unmoved finger slides over said touch-sensitive surface.

18. The method of claim 17, further comprising:

a) determining the velocity of said portable electronic device by integrating the acceleration measured by said accelerometer,

b) evaluating the velocity of said portable electronic device taken at least from a time point at which a moving contact point is detected by said touch-sensitive surface.

19. The method of claim 17, further comprising a decision criterion that evaluates whether the direction of a moving contact point, detected by said touch-sensitive surface, and the direction of the movement of said portable electronic device, detected at least in part by said sensor device, are approximately pointing in opposite directions.

20. The method of claim 17, further comprising a decision criterion that evaluates to what extent the speed of a moving contact point, measured by said touch-sensitive surface, and the velocity of said portable electronic device, measured at least in part by said sensor device, suit each other, a tolerance being permitted.

21. The method of claim 1, further comprising the steps of:

a) touching said touch-sensitive surface with said finger, the beginning of contact being detected by said touch-sensitive surface;

b) keeping said finger substantially unmoved;

c) shifting said portable electronic device below said substantially unmoved finger essentially in the plane of said touch-sensitive surface while maintaining contact between said substantially unmoved finger and said touch-sensitive surface;

d) removing said finger from said touch-sensitive surface while said portable electronic device is in full motion, the end of contact in full motion being detected by said touch-sensitive surface;

e) invoking an assigned function.

22. The method of claim 1, further comprising the steps of:

a) touching said touch-sensitive surface with said finger, the beginning of contact being detected by said touch-sensitive surface;

b) keeping said finger substantially unmoved;

c) shifting said portable electronic device below said substantially unmoved finger essentially in the plane of said touch-sensitive surface beyond the edge of said touch-sensitive surface such that said substantially unmoved finger loses contact with said touch-sensitive surface at the edge, the end of contact in full motion being detected by said touch-sensitive surface;

d) invoking an assigned function.

23. The method of claim 1, wherein said portable electronic device is a wrist-worn electronic device, and wherein said touch-sensitive surface is curved at least in part around a forearm, said wrist-worn electronic device being operated by the steps of:

a) touching said touch-sensitive surface with said finger, the beginning of contact being detected by said touch-sensitive surface;

b) keeping said finger substantially unmoved;

c) twisting said forearm below said substantially unmoved finger while maintaining contact, such that said substantially unmoved finger slides over said touch-sensitive surface;

d) measuring the rotation of said forearm at least in part by means of said sensor device;

e) stopping the rotation of said forearm;

f) removing said finger from said touch-sensitive surface, the end of contact being detected by said touch-sensitive surface;

g) invoking an assigned function at least in case the angle of rotation between said beginning of contact and said end of contact meets a decision criterion, or invoking an assigned function at least in case the speed of the rotation meets a decision criterion.

24. The method of claim 23, further comprising a decision criterion that evaluates to what extent the speed of a moving contact point, measured by said touch-sensitive surface, and the rotational speed of said portable electronic device, measured at least in part by said sensor device, suit each other, a tolerance being permitted.

25. The method of claim 1, further comprising the steps of:

a) touching said touch-sensitive surface with said finger, the beginning of contact being detected by said touch-sensitive surface;

b) rotating said portable electronic device below the substantially unmoved finger around a rotation axis approximately perpendicular to said touch-sensitive surface while maintaining contact, the contact point of said finger on said touch-sensitive surface constituting the pivot point;

c) increasing or decreasing an adjustable value or changing a property in dependence on the rotation around said rotation axis, the rotation being measured at least in part by said sensor device;

d) retaining the last adjusted value or the new configuration of said property at the latest when said finger is removed from said touch-sensitive surface, the end of contact being detected by said touch-sensitive surface.

26. The method of claim 25, wherein an additional contact point of an additional finger, detected by said touch-sensitive surface, results in a modification of the conversion ratio between the rotation of said portable electronic device around said rotation axis and said adjustable value or property.

27. The method of claim 25, wherein the area of said touch-sensitive surface that allows the modification of said adjustable value or property is visualized by means of a display at least as:

a) a control dial, the rotation of said portable electronic device around said rotation axis resulting in changes to the pointer position of said control dial,

b) a circle surrounding said pivot point, the rotation of said portable electronic device around said rotation axis resulting in changes to the size of said circle,

c) a color-selection element, the rotation of said portable electronic device around said rotation axis resulting in changes to the selected color, or

d) a numeric value, the rotation of said portable electronic device around said rotation axis resulting in changes to the value.

28. The method of claim 25, further comprising an overwind protection that, in case an upper or lower limit of said adjustable value or property is exceeded as the result of the rotation of said portable electronic device around said rotation axis, rotates at least a part of the screen content, visualized on said touch-sensitive surface by means of a display, by an angle opposite to the over-rotation determined at least in part by said sensor device, such that the opposite rotation angles offset each other.

29. The method of claim 1, wherein, as soon as the beginning of contact of said finger is detected by said touch-sensitive surface, the rotation of said portable electronic device around a rotation axis approximately perpendicular to said touch-sensitive surface is monitored at least in part by means of said sensor device, and if the angle of rotation since the beginning of contact exceeds a threshold value, a first assigned function is invoked or a value or property is changed, and otherwise, if till the end of contact the angle of rotation remains below said threshold value, a second assigned function is invoked.

30. The method of claim 1, wherein a first assigned function is invoked in case said touch-sensitive surface detects a single finger and wherein at least a second assigned function is invoked in case said touch-sensitive surface detects two fingers.

31. The method of claim 1, wherein said at least one finger is a thumb or a stylus, said thumb or said stylus being detected as an input by said touch-sensitive surface.

32. The method of claim 1, wherein:

a) said sensor device comprises at least two different sensors that are measuring different physical properties, the measurements of the sensors being combined such that the accuracy is increased,

b) said touch-sensitive surface is a touchscreen,

c) said assigned function is a gesture event, causes an action, changes a value, changes a property, or a combination thereof.

33. The method of claim 1, wherein said assigned function involves at least one of the following tasks:

a) aborting or suspending a running process or operation,

b) undoing an action,

c) deleting an element or object,

d) changing or sorting the order of elements, objects, pages, cards, or tabs,

e) locking or unlocking the scroll capability of a page, card, or tab,

f) terminating a running application program,

g) switching between application programs,

h) minimizing or maximizing a running application program,

i) moving a running application program to the background or foreground,

j) returning to a main menu or home screen,

k) executing a search function,

l) displaying an input screen for a search function,

m) entering an uppercase letter,

n) accepting a word or text suggested by an auto-complete function,

o) taking a telephone call,

p) terminating a telephone call,

q) enabling or disabling a lock screen,

r) jumping to the next or previous paragraph, page, or chapter of a text or document,

s) jumping to the beginning or end of a text or document,

t) leaping forward or backward during the playback of an audio or video recording,

u) skipping the current, the next, or the previous recording in a playlist,

v) setting a volume or a brightness,

w) rotating the screen content or an object displayed on a display.

34. A portable electronic device for recognizing gesture sequences consisting of at least one gesture part, comprising:

a) at least one sensor device selected from the group consisting of accelerometers, gyroscopes, magnetometers, and combinations thereof,

b) at least one touch-sensitive surface that is able to detect at least one input object, and

c) a control unit configured or programmed to;

i) determine an input caused by at least one input object, said input being detected by said touch-sensitive surface,

ii) process values captured at least in part by said sensor device

to discern

gesture parts that are based on the motion of said portable electronic device while said at least one input object remains substantially unmoved

from

gesture parts that are based on the change in position of said at least one input object while said portable electronic device remains substantially unmoved,

and

iii) invoke an assigned function in case a gesture sequence is recognized.

35. The portable electronic device of claim 34, wherein the motion of said portable electronic device involves:

a) to move said portable electronic device toward or away from said substantially unmoved input object,

b) to shift said portable electronic device below said substantially unmoved input object,

c) to rotate said portable electronic device below said substantially unmoved input object, or

d) a combination thereof.

36. The portable electronic device of claim 34, wherein the motion of said portable electronic device involves to shift said portable electronic device below said substantially unmoved input object essentially in the plane of said touch-sensitive surface beyond the edge of said touch-sensitive surface such that said substantially unmoved input object, which slides over said touch-sensitive surface, loses contact with said touch-sensitive surface at the edge.

37. The portable electronic device of claim 34, wherein said portable electronic device is a wrist-worn electronic device, said touch-sensitive surface being curved at least in part around a forearm, and wherein the motion of said wrist-worn electronic device involves to twist said forearm below said substantially unmoved input object while maintaining contact between said input object and said touch-sensitive surface, such that said substantially unmoved input object slides over said touch-sensitive surface.

38. The portable electronic device of claim 34, wherein the motion of said portable electronic device involves to rotate said portable electronic device below said substantially unmoved input object around a rotation axis approximately perpendicular to said touch-sensitive surface, the pivot point of said rotation axis on said touch-sensitive surface being located in the region in which said substantially unmoved input object is sliding on said touch-sensitive surface.

39. The portable electronic device of claim 34, wherein said control unit further is configured or programmed to recognize a gesture sequence comprising at least the following successive steps:

a) to touch said touch-sensitive surface with said input object while said portable electronic device remains substantially unmoved;

b) to move said portable electronic device while said input object remains substantially unmoved.

40. The portable electronic device of claim 34, wherein:

a) said touch-sensitive surface is a touchscreen,

b) said control unit further is configured or programmed to determine that at least one input object is touching said touchscreen while said portable electronic device is rotated below said substantially unmoved input object around a rotation axis approximately perpendicular to said touchscreen, the pivot point of said rotation axis on said touchscreen being located in the region in which said substantially unmoved input object is sliding on said touchscreen,

c) said assigned function involves to change at least a property or the orientation of the screen content displayed on said touchscreen.

41. The portable electronic device of claim 34, wherein said assigned function has a different behavior depending on the number of input objects detected by said touch-sensitive surface.

42. The portable electronic device of claim 34, wherein:

a) said sensor device is a interconnection of at least two different sensors that are measuring different physical properties, the measurements of the sensors being combined such that the accuracy is increased,

b) said touch-sensitive surface is a touchscreen,

c) said at least one input object is selected from the group consisting of fingers, thumbs, styluses, and combinations thereof,

d) said assigned function is a gesture event, causes an action, changes a value, changes a property, or a combination thereof.

43. The portable electronic device of claim 34, wherein said assigned function involves at least one of the following tasks:

a) to abort or suspend a running process or operation,

b) to undo an action,

c) to delete an element or object,

d) to change or sort the order of elements, objects, pages, cards, or tabs,

e) to lock or unlock the scroll capability of a page, card, or tab,

f) to terminate a running application program,

g) to switch between application programs,

h) to minimize or maximize a running application program,

i) to move a running application program to the background or foreground,

j) to return to a main menu or home screen,

k) to execute a search function,

l) to display an input screen for a search function,

m) to enter an uppercase letter,

n) to accept a word or text suggested by an auto-complete function,

o) to take a telephone call,

p) to terminate a telephone call,

q) to enable or disable a lock screen,

r) to jump to the next or previous paragraph, page, or chapter of a text or document,

s) to jump to the beginning or end of a text or document,

t) to leap forward or backward during the playback of an audio or video recording,

u) to skip the current, the next, or the previous recording in a playlist,

v) to set a volume or a brightness,

w) to rotate the screen content or an object displayed on a display.