Eye gaze correction

• US 9,749,581 B2
• Filed: 07/06/2015
• Issued: 08/29/2017
• Est. Priority Date: 04/28/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Nilsson, Mattias, Johnson, Sam
• Primary Examiner(s): Nguyen, Joseph J
• Application Number:
  US14/792,327
• Publication Number:
  US 20160323541A1
• Time in Patent Office: 785 Days
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A user'"'"'s eye gaze is corrected in a video of the user'"'"'s face. Each of a plurality of templates comprises a different image of an eye of the user looking directly at the camera. Every frame of at least one continuous interval of the video is modified to replace each of the user'"'"'s eyes with that of a respective template selected for that frame, whereby the user is perceived to be looking directly at the camera in the modified frames. Different templates are selected for different frames of the continuous interval so that the user'"'"'s eyes exhibit animation throughout the continuous interval.

Claims

1. A user device for correcting an eye gaze of a user comprising:

an input configured to receive from a camera video of the user'"'"'s face;

computer storage holding a plurality of templates, each comprising a different image of an eye of the user looking directly at the camera;

an eye gaze correction module configured to modify every frame of at least one continuous interval of the video to replace each of the user'"'"'s eyes with that of a respective template selected for that frame, whereby the user is perceived to be looking directly at the camera in the modified frames; and

a template selection module configured to select the templates for the continuous interval, wherein different templates are selected for different frames of the continuous interval so that the user'"'"'s eyes exhibit animation throughout the continuous interval.

2. A user device according to claim 1 wherein each of the plurality of templates is at least a part of a frame of a template video.

3. A user device according to claim 1 wherein the template selection module is configured to select the templates using a randomized selection procedure.

4. A user device according to claim 2 wherein the template selection module is configured to select the templates using a randomized selection procedure comprising, after an initial template has been selected for use by the eye gaze correction module, selecting a template at random, to be used next by the eye gaze correction module, that is one of the following:

at least a part of the next frame in the template video or at least part of a frame in the template video which matches the initial template and which is not the next frame in the template video.

5. A user device according to claim 1 comprising a blink detection module configured to detect a blinking action performed by the user, wherein the modification by the eye gaze correction module is halted for frames of the received video in which the user is detected to be blinking.

6. A user device according to claim 5 wherein following a detected blink by the user at least some of the templates are compared to a current frame of the received video to select an initial template that matches the current frame of the received video.

7. A user device according to claim 6 wherein templates are selected according to a randomized selection procedure thereafter until the user blinks again;

wherein each of the plurality of templates is at least a part of a frame of a template video, and wherein the randomized selection procedure comprises, after the initial template has been selected for use by the eye gaze correction module, selecting a template at random, to be used next by the eye gaze correction module, that is one of the following;

at least a part of the next frame in the template video or at least part of a frame in the template video which matches the initial template and which is not the next frame in the template video.

8. A user device according to claim 1 wherein the template selection module is configured to perform an individual template selection for every frame or every two frames of the at least one continuous interval.

9. A user device according to claim 1 wherein the template selection module is configured to cause a change of template every frame or every two frames.

10. A user device according to claim 1 comprising a template capture module configured to output to the user a notification that the user should look directly at the camera, and to capture the templates in response the user looking directly at the camera.

11. A user device according to claim 1 comprising a template capture module configured to automatically detect the that user is looking directly at the camera and to capture the templates in response.

12. A user device according to claim 1 comprising the camera or an external interface configured to receive the video from the camera.

13. A user device according to claim 12 wherein the external interface is a network interface via which the video is received from a network.

14. A user device according to claim 1 comprising a template modification module configured to modify the templates so as to modify a visual appearance of the eyes.

15. A user device according to claim 14 wherein the template modification module is configured to modify the templates to:

change an iris colour, correct an asymmetry of the eyes, and/or whiten the eyes.

16. A user device according to claim 1 comprising a network interface configured to transmit the modified frames in an outgoing video stream to another user device via a network.

17. A user device according to claim 1 wherein each of the templates comprises an image of an eye of the user looking directly at the camera and at least a portion of the user'"'"'s face surrounding that eye, wherein the eye gaze correction module is configured, responsive to a template being selected for a frame, to blend a portion of the template with a corresponding portion of the frame.

18. A user device according to claim 1 comprising a facial tracking module configured, in an active tracking mode, to track at least one angular coordinate of the user'"'"'s face and to output a current value of the at least one angular coordinated that is within a range of possible values;

wherein the received video is modified responsive to the facial tracking module being in the active tracking mode and the current value is within a restricted subrange of the range of possible values for which the user'"'"'s face is oriented towards the camera.

19. A method of correcting an eye gaze of a user comprising:

receiving from a camera video of the user'"'"'s face;

accessing a plurality of stored templates, each comprising a different image of an eye of the use looking directly at the camera; and

modifying every frame of at least one continuous interval of the video to replace each of the user'"'"'s eyes with that of a respective template selected for that frame, whereby the user is perceived to be looking directly at the camera in the modified frames, wherein different templates are selected for different frames of the continuous interval so that the user'"'"'s eyes exhibit animation throughout the continuous interval.

20. A computer program product for correcting an eye gaze of a user comprising code stored on a computer readable storage medium device and configured to:

receive from a camera video of the user'"'"'s face;

access a plurality of stored templates, each comprising a different image of an eye of the use looking directly at the camera; and

modify every frame of at least one continuous interval of the video to replace each of the user'"'"'s eyes with that of a respective template selected for that frame, whereby the user is perceived to be looking directly at the camera in the modified frames, wherein different templates are selected for different frames of the continuous interval so that the user'"'"'s eyes exhibit animation throughout the continuous interval.