Automated E-tran application

• US 10,353,475 B2
• Filed: 10/03/2016
• Issued: 07/16/2019
• Est. Priority Date: 10/03/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhang, Xiaoyi, Kulkarni, Harish Sripad, Morris, Meredith
• Primary Examiner(s): Mercado, Ariel
• Application Number:
  US15/284,124
• Publication Number:
  US 20180095539A1
• Time in Patent Office: 1,016 Days
• Field of Search: None
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/32 Normalisation of the pattern dimensions
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

Techniques for text entry using gestures are disclosed. As disclosed, a camera may capture a frame and the face of the user can be detected therein. Landmarks can be aligned to the face in the captured frame. A left eye image and a right eye image may be extracted from the captured frame. The left eye image and the right image each may be resized and compared to a calibration template. A direction of eye gaze may be determined based upon the comparison. A character or word may be predicted based upon the determination of the direction of eye gaze and a known configuration of an eye gaze board (e.g., an E-tran board). The predicted character or word can be included as a part of a text-based message.

Claims

1. A system, comprising:

A rear camera of a mobile device configured to capture at least one frame that includes a face of a user;

a database configured to store the at least one frame;

a processor communicatively coupled to the rear camera and the database, the processor configured to;

detect the face in the captured at least one frame;

extract a left eye image and a right eye image of the face from the at least one frame;

resize the left eye image to a size matching one of a calibrated set of left eye images for the user and the right eye image to a size matching one of a calibrated set of right eye images for the user;

compare the resized left eye image to the calibrated set of left eye images for the user, wherein each image in the calibrated set of left eye images corresponds to a known eye gaze direction for a left eye of the user;

compare the resized right eye image to the calibrated set of right eye images for the user, wherein each image in the calibrated set of right eye images corresponds to a known eye gaze direction for a right eye of the user;

determine a direction of eye gaze based upon the comparison of the resized left eye image to the calibrated set of images for the left eye and the comparison of the resized right eye image to the calibrated set of images for the right eye;

identify, based upon the determination of the direction of eye gaze and a known configuration of an eye gaze board having a plurality of quadrants, a quadrant of the eye gaze board corresponding to the direction of eye gaze, wherein each of the plurality of quadrants of the eye gaze board comprise at least one character; and

display the at least one character from the quadrant as a part of a text-based message.

2. The system of claim 1, the processor is further configured to generate the calibration template comprising the calibrated set of left eye images and the calibrated set of right eye images, comprising:

receive an indication of eye gaze of the user in a direction;

receiving a second at least one frame of the user that is captured from the rear camera;

detect the face of the user in the second at least one frame;

extract a calibrated left eye image and a calibrated right eye image of the face from the second at least one frame; and

store the calibrated left eye image and the calibrated right eye image to the database as part of the calibration template for the direction of eye gaze.

3. The system of claim 2, the processor further configured to:

determine lighting condition based on an amount of light detected by the rear camera; and

associate the lighting condition with the calibration template.

4. The system of claim 1, the processor further configured to:

based upon the at least one character in the quadrant, determine at least one word associated with the at least one character using a predictive text algorithm that determines words based upon frequency of usage.

5. The system of claim 4, the processor further configured to:

align landmarks to the face in the captured at least one frame.

6. The system of claim 1, the processor further configured to:

cause an audible or visible indication of the at least one character.

7. The system of claim 1, the processor further configured to:

receive a gesture from the user; and

perform an action based upon the gesture.

8. The system of claim 7, wherein the action is to delete the at least one character from the text-based message.

9. The system of claim 1, the processor further configured to:

convert the left eye image to hue, saturation, and value (HSV) color space and retain only the value portion; and

convert the right eye image to HSV color space and retain only the value portion.

10. The system of claim 1, the processor further configured to:

receiving a plurality of eye gazes from the user to form a sequence of eye gazes; and

predicting a word based upon the sequence of eye gazes using a predictive text algorithm.

11. A computer-implemented method, comprising:

capturing, by a rear camera of a mobile device, at least one frame that includes a face of a user;

detecting the face in the captured at least one frame;

extracting a left eye image and a right eye image of the face from the at least one frame;

resizing the left eye image to a size matching one of a calibrated set of left eye images for the user and the right eye image to a size matching one of a calibrated set of right eye images for the user;

comparing the resized left eye image to the calibrated set of left eye images for the user, wherein each image in the calibrated set of left eye images corresponds to a known eye gaze direction for a left eye of the user;

comparing the resized right eye image to the calibrated set of right eye images for the user, wherein each image in the calibrated set of right eye images corresponds to a known eye gaze direction for a right eye of the user;

determining a direction of eye gaze based upon the comparison of the resized left eye image to the calibrated set of images for the left eye and the comparison of the resized right eye image to the calibrated set of images for the right eye;

identifying, based upon the determination of the direction of eye gaze and a known configuration of an eye gaze board having a plurality of quadrants, a quadrant of eye gaze board corresponding to the direction of eye gaze, wherein each of the plurality of quadrants of the eye gaze board comprise at least one character; and

displaying the at least one character from the quadrant as a part of a text-based message.

12. The method of claim 11, further comprising generating the calibration template comprising the calibrated set of left eye images and the calibrated set of right eye images, comprising:

receiving an indication of eye gaze of the user in a direction;

capturing a second at least one frame of the user,

detecting the face of the user in the second at least one frame;

extracting a calibrated left eye image and a calibrated right eye image of the face from the second at least one frame; and

storing the calibrated left eye image and the calibration right eye image as part of the calibration template for the direction of eye gaze.

13. The method of claim 12, further comprising:

determining lighting condition based on an amount of light detected by the rear camera; and

associating the lighting condition with the calibration template.

14. The method of claim 13, further comprising:

based upon the at least one character in the quadrant, determining at least one word associated with the at least one character using predictive text algorithm that determines words based upon frequency of usage.

15. The method of claim 14, further comprising:

receiving a selection of at least one word associated with the at least one character.

16. The method of claim 11, further comprising:

generating an audible or visible indication of the at least one character.

17. The method of claim 11, further comprising:

receiving a gesture from the user; and

performing an action based upon the gesture.

18. The method of claim 11, further comprising:

converting the left eye image to hue, saturation, and value (HSV) color space and retain only the value portion; and

converting the right eye image to HSV color space and retain only the value portion.

19. The method of claim 11, further comprising:

receiving a plurality of eye gazes from the user to form a sequence of eye gazes; and

predicting a word based upon the sequence of eye gazes using a predictive text algorithm.

20. A non-transitory computer readable medium having stored thereon computer readable instructions that are executable to cause one or more processors to perform operations, comprising:

capturing, by a rear camera of a mobile device, at least one frame that includes a face of a user;

detecting the face in the captured at least one frame;

extract a left eye image and a right eye image of the face from the at least one frame;

resizing the left eye image to a size matching one of a calibrated set of left eye images for the user and the right eye image to a size matching one of a calibrated set of right eye images for the user;

comparing the resized left eye image to the calibrated set of left eye images for the user, wherein each image in the calibrated set of left eye images corresponds to a known eye gaze direction for a left eye of the user;

comparing the resized right eye image to the calibrated set of right eye images for the user, wherein each image in the calibrated set of right eye images corresponds to a known eye gaze direction for a right eye of the user;

determining a direction of eye gaze based upon the comparison of the resized left eye image to the calibrated set of images for the left eye and the comparison of the resized right eye image to the calibrated set of images for the right eye;

identifying, based upon the determination of the direction of eye gaze and a known configuration of an eye gaze board having a plurality of quadrants, a quadrant of the eye gaze board corresponding to the direction of eye gaze, wherein each of the plurality of quadrants of the eye gaze board comprise at least one character; and

displaying the at least one character from the quadrant as a part of a text-based message.