System and method for detecting facial gestures for control of an electronic device

• US 8,160,311 B1
• Filed: 09/26/2008
• Issued: 04/17/2012
• Est. Priority Date: 09/26/2008
• Status: Active Grant

• Current Assignee: Vortant Technologies, LLC
• Original Assignee: Philip Raymond Schaefer
• Inventors: Schaefer, Philip Raymond
• Primary Examiner(s): Hindi, Nabil
• Application Number:
  US12/287,087
• Time in Patent Office: 1,299 Days
• Field of Search: 382/115, 382/128, 382/165, 382/170, 382/181, 382/103, 382/190, 382/118, 345/156, 345/157, 345/158, 345/118, 345/684, 345/172, 340/5.8, 348/155, 348/154, 348/77, 348/78, 348/169, 704/271, 704/275, 715/785, 715/786, 351/210
• US Class Current: 382/118
• CPC Class Codes: G02C 11/10 Electronic devices other than hearing aids
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A control system enables a user to control an electronic device by moving parts of the face, including the eyes. Optical emitters and optical detectors are used to reflect light from the relevant parts of the user'"'"'s face. Pattern matching or approximation of a pointing location are used to define the control input to the electronic device based on the user'"'"'s motions. Visual feedback is optionally provided to the user during the controller'"'"'s actions. Additional sensor elements are optionally provided to improve performance of the system. Hardware and software elements are combined to provide for the specified control actions.

Claims

1. A control system for enabling a user to interface with an electronic device comprising:

A light-emitting means that causes light to shine on the user'"'"'s face;

A light-detection means that detects light from the light-emitting means reflected from the user'"'"'s face and outputs a light-intensity signal;

Context sensing means that senses input about the current activity of the user and outputs a user-activity signal;

Pattern-matching means that inputs the light-intensity signal and the user-activity signal and outputs a category signal; and

Controller means that inputs the category signal and outputs a control signal to control the electronic device,

wherein the light-transmitting means causes light to shine on the vicinity of the user'"'"'s mouth, and wherein the context sensing means comprises means to determine if the user is speaking and

wherein the pattern-matching means outputs a category indicating that no control signal is to be issued, as a result of the Context Input means determining that the user is speaking.

2. The control system of claim 1, wherein the light emitting means comprises a plurality of light emitting devices or the light detecting means comprises a plurality of light detecting devices, and wherein the pattern matching means outputs a category based on the relative values of a plurality of outputs from the light detecting means.

3. A control system for enabling a user to interface with an electronic device comprising:

A light-emitting means that causes light to shine on an eye of the user;

A light-detection means that detects light from the light-emitting means that is reflected from the eye of the user and outputs a light-intensity signal;

Pattern-matching means that inputs the light-intensity signal and outputs a category signal, wherein the pattern-matching means comprises means to distinguish between at least two directions of motion of the eye of the user; and

Controller means that inputs the category signal and outputs a control signal to control the electronic device,

further comprising Context Input means that determines when the contents of the display of the electronic device have recently changed, whereby changes in the contents of the display of the electronic device will indicate that the user will likely visually scan the contents of the display of the electronic device, and that at that time, no control motion is likely from the user.

4. The control system of claim 3, wherein the Context Input means comprises means to measure the light intensity emitted by the display of the electronic device.

5. The control system of claim 3, wherein the Context Input means comprises means to extract the user options contained on the display of the electronic device and to detect changes in the user options contained on the display of the electronic device.

6. The control system of claim 3, wherein the pattern-matching means outputs a category indicating that no control signal is to be issued, as a result of the Context Input means determining that the contents of the display of the electronic device have recently changed.

9. The control system of claim 3, wherein the Context Input means additionally comprises motion sensing means that outputs an indication of when the head of the user has moved and wherein the indication of when the head of the user has moved is used by the pattern-matching means to modify the category signal.

7. A control system for enabling a user to interface with an electronic device comprising:

A light-transmitting means that causes light to shine on the user'"'"'s face,

A light-detection means that detects light reflected from the user'"'"'s face and outputs a light-intensity signal;

Conversion means that inputs the light-intensity signal and outputs a conversion signal;

Extraction means in communication with the electronic device that outputs a plurality of user options available on the interface of the electronic device;

Selection means that converts the conversion signal into a selection of a user option from the users options outputted by the extraction means; and

Controller means that inputs the selection of a user option from the selection means and outputs a control signal to control the electronic device,

wherein the conversion means is a Pattern Matching means and the conversion signal is a category, and

wherein the light-transmitting means causes light to shine on a portion of an eye of the user, and

wherein the pattern-matching means includes means to distinguish between at least two directions of motion of the eye of the user and wherein a category outputted by the pattern-matching means corresponds to one of the at least two directions of motions of the eye of the user and wherein the selection means converts the category outputted by the pattern-matching means into a user option that, compared to the previous user option outputted by the selection means, is in the direction corresponding to the category outputted by the pattern-matching means.

8. A control system for enabling a user to interface with an electronic device comprising:

A light-transmitting means that causes light to shine on the user'"'"'s face;

A light-detection means that detects light reflected from the user'"'"'s face and outputs a light-intensity signal;

Conversion means that inputs the light-intensity signal and outputs a conversion signal;

Extraction means in communication with the electronic device that outputs a plurality of user options available on the interface of the electronic device;

Selection means that converts the conversion signal into a selection of a user option from the user options outputted by the extraction means; and

Controller means that inputs the selection of a user option from the selection means and outputs a control signal to control the electronic device,

wherein the conversion means is a position approximating means that converts the light intensity signal into a coordinate in the coordinate space of the electronic device, and wherein the Selection means selects a user option by comparing the coordinates of the user option with the coordinate outputted by the conversion means.