Method and portable terminal for correcting gaze direction of user in image

• US 9,288,388 B2
• Filed: 01/15/2013
• Issued: 03/15/2016
• Est. Priority Date: 02/29/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Son, Byung-Jun, Cho, Sung-Dae
• Primary Examiner(s): PRABHAKHER, PRITHAM DAVID
• Application Number:
  US13/741,991
• Publication Number:
  US 20130222644A1
• Time in Patent Office: 1,155 Days
• Field of Search: 348/208.12, 348345-357, 348/239
• US Class Current: 1/1
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact

Abstract

A method and portable terminal are provided for correcting a gaze direction of a user in an image. A face of the user is detected in the image. Contour points of an eye area are detected in the detected face. The contour points of the detected eye area in the image are changed to contour points that correspond to a gaze direction toward a reference camera. The gaze direction of the user in the image is corrected by transforming the eye area according to the changed contour points.

Claims

1. A method for correcting a gaze direction of a user in an image captured by a camera, the method comprising the steps of:

detecting a face of the user in the captured image;

detecting contour points of an eye area in the detected face, the eye area being defined by an eye contour connecting the contour points;

changing the contour points of the detected eye area in the captured image to contour points corresponding to a gaze direction toward the camera by moving at least some of the detected contour points; and

correcting the gaze direction of the user in the captured image by transforming the eye area according to a changed eye contour which connects the changed contour points,

wherein detecting the contour points of the eye area in the detected face comprises setting a point having a largest change in a brightness value of a pixel in a direction of a vector passing through a contour point of a previously-set eye model, as a contour point of the eye area.

2. The method as claimed in claim 1, wherein detecting the face of the user in the captured image comprises:

comparing a preset threshold with a difference between coordinates of each eye in a previous frame and coordinates of each eye in a current frame; and

adjusting the coordinates of each eye in the current frame when the difference is less than the threshold.

3. The method as claimed in claim 2, wherein adjusting the coordinates of each eye comprises changing the coordinates of each eye in the current frame to coordinates located between the coordinates of each eye in the previous frame and the coordinates of each eye in the current frame.

4. The method as claimed in claim 1, wherein changing the contour points comprises changing at least some of the detected contour points to contour points of a previously-stored eye area that gazes in the direction toward the camera.

5. The method as claimed in claim 1, wherein transforming the eye area comprises performing texture mapping on the eye area by using a forward warping technology or a backward warping technology.

6. The method as claimed in claim 5, wherein the texture mapping is performed within an oval mask including the changed contour points of the eye area.

7. The method as claimed in claim 1, further comprising:

determining whether it is appropriate to correct the gaze direction by performing at least one of;

determining whether a distance between both eyes in the captured image is less than or equal to a previously-set threshold;

determining whether both eyes are located in a previously-set guide area; and

determining whether a slope of the detected face, which is measured by using coordinates of both eyes, is less than or equal to a previously-set slope.

8. An article of manufacture for correcting a gaze direction of a user in an image captured by a camera, comprising a non-transitory machine-readable storage medium for recording one or more programs, which when executed implement the steps of:

detecting a face of the user in the captured image;

detecting contour points of an eye area in the detected face, the eye area being defined by an eye contour connecting the contour points;

changing the contour points of the detected eye area in the captured image to contour points corresponding to a gaze direction toward the camera by moving at least some of the detected contour points; and

correcting the gaze direction of the user in the captured image by transforming the eye area according to a changed eye contour which connects the changed contour points,

wherein detecting the contour points of the eye area in the detected face comprises setting a point having a largest change in a brightness value of a pixel in a direction of a vector passing through a contour point of a previously-set eye model, as a contour point of the eye area.

9. A portable terminal for correcting a gaze direction of a user in an image, the portable terminal comprising:

a camera for capturing an image;

a processor for detecting a face of the user in the captured image, detecting contour points of an eye area in the detected face, the eye area being defined by an eye contour connecting the contour points, changing the contour points of the detected eye area in the captured image to contour points corresponding to a gaze direction toward the camera by moving at least some of the detected contour points, and correcting the gaze direction of the user in the captured image by transforming the eye area according to a changed eye contour which connects the changed contour points; and

a display for displaying the captured image in which the eye area is transformed by the processor,

wherein the processor sets a point having a largest change in a brightness value of a pixel in a direction of a vector passing through a contour point of a previously-set eye model, as a contour point of the eye area.

10. The portable terminal as claimed in claim 9, wherein the processor compares a preset threshold with a difference between coordinates of each eye in a previous frame and coordinates of each eye in a current frame, and adjusts the coordinates of each eye in the current frame when the difference is less than the threshold.

11. The portable terminal as claimed in claim 10, wherein the processor changes the coordinates of each eye in the current frame to coordinates located between the coordinates of each eye in the previous frame and the coordinates of each eye in the current frame.

12. The portable terminal as claimed in claim 9, wherein the gaze direction toward the reference camera corresponds to a straight-ahead gaze direction.

13. The portable terminal as claimed in claim 9, wherein the processor changes at least some of the detected contour points to contour points of a previously-stored eye area that gazes in the direction toward the camera.

14. The portable terminal as claimed in claim 9, wherein the processor performs texture mapping on the eye area by using a forward warping technology or a backward warping technology.

15. The portable terminal as claimed in claim 14, wherein the texture mapping is performed within an oval mask including the changed contour points of the eye area.

16. The portable terminal as claimed in claim 9, wherein the processor determines whether it is appropriate to correct the gaze direction by performing at least one of:

determining whether a distance between both eyes in the captured image is less than or equal to a previously-set threshold;

determining whether both eyes are located in a previously-set guide area; and

determining whether a slope of the detected face, which is measured by using coordinates of both eyes, is less than or equal to a previously-set slope.

17. The portable terminal as claimed in claim 9, wherein the processor changes the contour points based on a difference between a gaze direction of the face of the user in the captured image and the gaze direction toward the camera, which is previously set as a reference.

18. The portable terminal as claimed in claim 9, wherein the image is captured while a self-image capturing application, a video phone call application or a mirror application is executed.