Eye gaze correction

• US 9,740,938 B2
• Filed: 07/06/2015
• Issued: 08/22/2017
• Est. Priority Date: 04/28/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Nilsson, Mattias, Johnson, Sam
• Primary Examiner(s): Nguyen, Joseph J
• Application Number:
  US14/792,324
• Publication Number:
  US 20160323540A1
• Time in Patent Office: 778 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A user'"'"'s eye gaze is corrected in a video of the user'"'"'s face. A facial tracking module is configured, in an active tracking mode, to track at least one angular coordinate of the user'"'"'s face and to output a current value of the at least one angular coordinated that is within a range of possible values. Frames of the video are modified to correct the eye gaze of the user, whereby the user is perceived to be looking directly at the camera in the modified frames, only when the tracking module is in the active tracking mode and the current value is within a restricted subrange of the range of possible values for which the user'"'"'s face is oriented towards the camera.

Claims

1. A user device for correcting an eye gaze of a user comprising:

an input configured to receive from a camera video of the user'"'"'s face;

a facial tracking module configured, in an active tracking mode, to track at least one angular coordinate of the user'"'"'s face and to output a current value of the at least one angular coordinated that is within a range of possible values; and

an eye gaze correction module configured to modify frames of the video to correct the eye gaze of the user, whereby the user is perceived to be looking directly at the camera in the modified frames responsive to the facial tracking module being in the active tracking mode and the current value is within a restricted subrange of the range of possible values for which the user'"'"'s face is oriented towards the camera.

2. A user device according to claim 1, wherein the facial tracking module is also configured to track at least one spatial coordinate of the user'"'"'s face and to output current values of the tracked coordinates that are each within a respective range of possible values;

and

wherein the frames are modified responsive to the facial tracking module being in the active tracking mode and the current values are each within a respective restricted subrange of the respective range of possible values for which the user'"'"'s face is oriented towards the camera and within a restricted spatial region.

3. A user device according to claim 2 wherein the at least one spatial coordinate comprises at least two or at least three spatial coordinates of the user'"'"'s face.

4. A user device according to claim 1 wherein the facial tracking module is configured to track at least two angular coordinates of the user'"'"'s face and to output current values of the tracked at least two coordinates that are each within a respective range of possible values;

and

wherein the frames are modified responsive to the tracking module being in the active tracking mode and the current values are each within a respective restricted subrange of the respective range of possible values for which the user'"'"'s face is oriented towards the camera.

5. A user device according to claim 4 wherein the at least two angular coordinates comprise at least three angular coordinates of the user'"'"'s face.

6. A user device according to claim 1 wherein the facial tracking module is configured to track at least one spatial coordinate of the user'"'"'s face, and the user device comprises a limit setting module configured to vary the restricted subrange for the at least one angular coordinate based on the tracking of the at least one spatial coordinate.

7. A user device according to claim 1 wherein the user device comprises a display and a limit setting module configured to vary the restricted subrange for the at least one angular coordinate based on display data indicating a current state of the display.

8. A user device according to claim 7 comprising a network interface configured to receive far-end video of another user which is displayed on the display, and the restricted subrange for the at least one angular coordinate is varied based on a current display parameter of the displaying of the far-end video.

9. A user device according to claim 8 wherein the restricted subrange for the at least one angular coordinate is varied based on a current location of and/or a current area occupied by the far-end video on the display.

10. A user device according to claim 1 comprising computer storage holding one or more templates, each comprising an image of an eye of the user looking directly at the camera, wherein the eye gaze is corrected by replacing each of the user'"'"'s eyes with a respective template.

11. A user device according claim 10 wherein each of the one or more templates comprises an image of the user'"'"'s eyes looking directly at the camera and at least portions of the user'"'"'s face surrounding the eyes, wherein the eye gaze correction module is configured to blend those portions with corresponding portions of the frames.

12. A user device according to claim 10 comprising a template modification module configured to modify the templates so as to modify a visual appearance of the eyes.

13. A user device according to claim 12 wherein the template modification module is configured to modify the templates to:

change an iris colour, correct an asymmetry of the eyes, and/or whiten the eyes.

14. A user device according to claim 1 comprising a network interface configured to transmit the modified frames in an outgoing video stream to another user device via a network.

15. A user device according to claim 10 wherein every frame of at least one continuous interval of the video is modified to replace each of the user'"'"'s eyes with a respective template selected for that frame;

wherein the user device comprises a template selection module configured to select the templates for the continuous interval, different templates being selected for different frames of the continuous interval so that the user'"'"'s eyes exhibit animation throughout the continuous interval.

16. A method of correcting an eye gaze of a user comprising:

receiving from a camera video of the user'"'"'s face;

responsive to a facial tracking module being in an active tracking mode, receiving from the facial tracking module a current value of at least one angular coordinate of the user'"'"'s face that the facial tracking module is tracking; and

modifying frames of the video to correct the eye gaze of the user, whereby the user is perceived to be looking directly at the camera in the modified frames, responsive to the facial tracking module being in the active tracking mode and the current value is within a restricted subrange of the range of possible values for which the user'"'"'s face is oriented towards the camera.

17. A method according to claim 16 wherein the facial tracking module is also configured to track at least one spatial coordinate of the user'"'"'s face and to output current values of the tracked coordinates that are each within a respective range of possible values;

and

wherein the frames are modified responsive to the facial tracking module being in the active tracking mode and the current values are each within a respective restricted subrange of the respective range of possible values for which the user'"'"'s face is oriented towards the camera and within a restricted spatial region.

18. A method according to claim 16 wherein the facial tracking module is configured to track at least two angular coordinates of the user'"'"'s face and to output current values of the tracked at least two coordinates that are each within a respective range of possible values;

and

wherein the frames are modified responsive to the tracking module being in the active tracking mode and the current values are each within a respective restricted subrange of the respective range of possible values for which the user'"'"'s face is oriented towards the camera.

19. A method according to claim 16 wherein the restricted subrange for the at least one angular coordinate is varied based on display data indicating a current state of a display available to the user.

20. A computer readable storage medium comprising code stored on a computer for correcting an eye gaze of a user configured to:

receive from a camera video of the user'"'"'s face;

responsive to a facial tracking module being in an active tracking mode, receiving from the facial tracking module a current value of at least one angular coordinate of the user'"'"'s face that the facial tracking module is tracking; and

modify frames of the video to correct the eye gaze of the user, whereby the user is perceived to be looking directly at the camera in the modified frames, responsive to the facial tracking module being in the active tracking mode and the current value is within a restricted subrange of the range of possible values for which the user'"'"'s face is oriented towards the camera.