Wink Gesture Based Control System

• US 20160203359A1
• Filed: 01/12/2016
• Published: 07/14/2016
• Est. Priority Date: 01/12/2015
• Status: Active Grant

• Current Assignee: BMT Business meets Technology Holding AG
• Original Assignee: BMT Business meets Technology Holding AG
• Inventors: von und zu Liechtenstein, Maximilian Ralph Peter
• Granted Patent: US 10,121,063 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0233 Character input methods
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A wink gesture based control technique for selecting, controlling and manipulating virtual objects and smart applications on a display using facial gestures of the user, in particular winking, blinking and squinting movements of the eyes. The wink gesture based control technique utilizes wink gestures of a user to select, control, and manipulate virtual objects on a display. A head mounted device is adapted to allow detection and classification of specific wink gestures. The head mounted device in some embodiments may also be adapted to recognize certain characteristics of the wink gestures, such as duration or amplitude, to allow enhanced navigation of a computer interface.

Claims

1. A method comprising:

detecting at least one wink gesture;

determining parameters intrinsic to the at least one wink gesture; and

controlling a user interface of a computer system in response to detecting the at least one wink gesture.

2. The method of claim 1, wherein the at least one wink gesture comprises an indication of magnitude, further comprising the step of detecting the indication of magnitude.

3. The method of claim 2, wherein the indication of magnitude comprises the degree of facial distortion associated with the at least one wink gesture.

4. The method of claim 2, wherein the indication of magnitude comprises the duration of the facial distortion associated with the at least one wink gesture.

5. The method of claim 2, wherein the at least one wink gesture comprises an indication of direction and wherein the indication of direction is associated with a side of a face on which the wink gesture is made.

6. The method of claim 1, wherein the controlling a user interface step comprises translating parameters intrinsic to the at least one wink gesture to a corresponding scalar-type or vector-type user interface control parameter.

7. The method of claim 1, wherein the plurality of wink types relating to the at least one wink gesture is selected from the group consisting of a single-sided single-wink;

a single-sided double-wink;

a duration-based single-sided wink;

a duration-based double-sided wink; and

a amplitude-based single-sided wink.

8. The method of claim 1, wherein detecting the at least one wink gesture comprises bringing a plurality of wearable surface dry electrodes into contact with a user'"'"'s skin.

9. The method of claim 8, wherein the wearable surface dry electrodes are comprised of silver nano wires.

10. The method of claim 8, wherein the wearable surface dry electrodes are brought into contact with the user'"'"'s skin on at least one of a plurality of sites, wherein the plurality of sites is selected from the group consisting of a nose bridge area, a nose side, a temple area, and an ear area.

11. The method of claim 10, wherein at least one reference electrode contacts the ear area, wherein a plurality of signal electrodes is placed at sites which are selected from the list consisting of a nose bridge area, a nose side and a temple area.

12. The method of claim 1, wherein detecting the at least one wink gesture comprises the use of a plurality of distance sensors and wherein the plurality of distance sensors is used to detect a bulge of at least one facial muscle group when the at least one facial muscle group is flexed.

13. The method of claim 12, wherein the at least one muscle group comprises the Zygomaticus muscle group.

14. The method of claim 1, wherein detecting the at least one wink gesture comprises the use of a plurality of motion sensors, wherein the plurality of motion sensors is configured to detect the motion associated with the facial distortion associated with a wink gesture.

15. The method of claim 1, wherein detecting the at least one wink gesture comprises the use of a plurality of user-facing cameras;

and wherein detecting the at least one wink gesture comprises visual recognition of a plurality of reference points on the user'"'"'s face which are selected from the group consisting of eyebrows, eye lids, eyes; and

wherein detecting the at least one wink gesture comprises utilizing any changes of position of the plurality of reference points on the user'"'"'s face as parameters for detecting the at least one wink gesture.

16. The method of claim 1, wherein controlling the user interface of a computer system in response to detecting the at least one wink gesture further comprises displaying list items as a plurality of virtual symbol cards attached to at least one virtual object, wherein movement of the at least one virtual object is controlled in response to the at least one wink gesture.

17. The method of claim 16, wherein the direction of the movement of the at least one virtual object is determined by the side of the face on which the at least one wink gesture is made.

18. The method of claim 17, wherein the movement of the at least one virtual object further comprises simulated torque being applied;

wherein the magnitude of the simulated torque is derived from a function of the magnitude of the at least one wink gesture; and

wherein the function of the magnitude of the at least one wink gesture comprises both linear and exponential functions.

19. The method of claim 16, further comprising:

selecting one of the at least two symbol cards when the movement of the at least one virtual object ceases; and

configuring the movement of the at least one virtual object so that the selecting of one of the at least two symbol cards always yields an unambiguous selection.

20. The method of claim 16, further comprising:

designating one of the plurality of virtual objects as the active object;

selecting one other of the plurality of virtual objects with the at least one wink gesture; and

designating the selected virtual object as the new active virtual object.

21. The method of claim 20, further comprising:

entering alphanumeric input into the user interface of the computer system, wherein entering the alphanumeric input comprises;

controlling both a simulated typewriter carriage and the at least one virtual object carrying the at least two symbol cards with the at least one wink gesture; and

transposing the symbol of the selected symbol card to the simulated print position on the simulated typewriter carriage;

wherein the simulated typewriter carriage comprises a virtual object capable of attachment of a plurality symbol cards;

wherein the simulated typewriter carriage is capable of movements from the list consisting of moving one print position to the left, moving one print position to the right, and returning n print position to the initial start position.

22. The method of claim 1, wherein controlling the user interface of a computer system in response to detecting the at least one wink gesture further comprises a method for selecting an augmented reality object comprising:

defining a region of interest based on the at least one wink gesture;

displaying to the user a shape, wherein the shape outlines the region of interest;

displaying to the user a plurality of augmented reality objects, each of the plurality of augmented reality objects being associated with a target inside the region of interest; and

reducing or enlarging the size of the regions of interest based on the magnitude of the at least one wink gesture to form a reduced/enlarged-sized region of interest, wherein the reduced/enlarged-sized region of interest is used to select a specific augmented reality object from the plurality of augmented reality objects.

23. The method of claim 22, wherein multiple augmentations are associated with the specific AR object, the method further comprising:

displaying to the user a corresponding augmentation from the multiple augmentations associated with the specific augmented reality object commanded by a second wink gesture.

24. The method of claim 22, further comprising:

capturing text inside the reduced-sized region of interest and initializing translation based on the captured text commanded by a second wink gesture.

25. The method of claim 22, further comprising:

performing a visual recognition search inside the reduced-sized region of interest commanded by a second wink gesture.

26. The method of claim 7, further comprising:

iterating through a plurality of virtual objects, comprising the steps of;

deriving a vector from the at least one wink gesture, wherein the side of the face on which the at least one wink gesture takes place determines the direction of the vector and wherein the magnitude of the wink gesture determines the magnitude of the vector; and

controlling the direction of the iteration by the direction of the vector and; and

controlling the speed of the iteration by the magnitude of the vector.

27. A non-transitory computer-readable medium having stored therein instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

detecting a wink gesture at a head mounted device, wherein detecting a wink gesture comprises;

determining intrinsic parameters of the wink gesture wherein the intrinsic parameters are selected from the list comprising;

duration of the wink gesture, magnitude of the wink gesture, side of the wink gesture; and

translating scalar values of the intrinsic parameters of the plurality of wink gestures to corresponding magnitudes of control inputs relating to input parameters of the user interface of the computing device; and

translating the directional component of the intrinsic parameters of the plurality of wink gestures to corresponding direction of control inputs relating to input parameters of the user interface of the computing device.

28. The non-transitory computer-readable medium of claim 27, further comprising the step of translating the magnitude of a wink gesture to a corresponding control input relating to input parameters of the computing device using an exponential function.

29. A head mounted device system, comprising:

a head mounted device;

a wink detection system that is connected to the head mounted device, wherein the wink detection system is adapted to detect a wink gesture at the head mounted device, wherein the wink detection system comprises a plurality of sensors selected from the group consisting of electromyography sensors, distance sensors, motion sensors, infrared cameras, visual-range cameras and any combination thereof, wherein the wink gesture detected is selected from the group consisting of amplitude-dependent single-sided wink, duration-dependent single-sided wink, single-sided single-wink, single-sided double wink and double-sided single wink;

a display; and

a non-transitory computer-readable medium having stored therein program instructions that, upon execution by a computing device, cause the computing device to perform functions comprising;

allowing the user to control smart applications through control inputs comprising the wink gesture and wherein the wink gesture can be used to achieve tasks from the following group consisting of resizing a virtual object, iterating between different virtual objects, shifting the active focus to a different virtual object, controlling movable virtual object, and making or confirming selections;

setting intrinsic HMD parameters comprising adjusting the speaker volume and adjusting the brightness of the display; and

wherein the response to a wink gesture input may take into account parameters selected from the group consisting of magnitude of the wink gesture, duration of the wink gesture, number of repetitions of the wink gesture, the side of the face on which the wink gesture was performed, and any combination of the said parameters.

30. The system of claim 29, further comprising a simulated mouse-operated user interface system comprising:

an eye movement detection system that is connected to the head mounted device, wherein when the head mounted device is worn the eye movement detection system is adapted to detect the movement of the user'"'"'s eyes, wherein user interface of the head mounted device is adapted to display a mouse pointer, wherein the mouse pointer is configured to be controlled by the movement of the user'"'"'s eyes; and

wherein the functions of the simulated mouse-operated user interface system comprise;

simulating a mouse-click when the wink gesture is performed, wherein the wink gesture is selected from the group consisting of a single left wink, a single right wink, a double left wink, and a double right wink.