METHOD AND PORTABLE TERMINAL FOR CORRECTING GAZE DIRECTION OF USER IN IMAGE

• US 20130222644A1
• Filed: 01/15/2013
• Published: 08/29/2013
• Est. Priority Date: 02/29/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: SON, Byung-Jun, Cho, Sung-Dae
• Granted Patent: US 9,288,388 B2
• Time in Patent Office: Days
• US Class Current: 348/239
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact

Abstract

A method and portable terminal are provided for correcting a gaze direction of a user in an image. A face of the user is detected in the image. Contour points of an eye area are detected in the detected face. The contour points of the detected eye area in the image are changed to contour points that correspond to a gaze direction toward a reference camera. The gaze direction of the user in the image is corrected by transforming the eye area according to the changed contour points.

Claims

1. A method for correcting a gaze direction of a user in an image, the method comprising the steps of:

detecting a face of the user in the image;

detecting contour points of an eye area in the detected face;

changing the contour points of the detected eye area in the image to contour points that correspond to a gaze direction toward a reference camera; and

correcting the gaze direction of the user in the image by transforming the eye area according to the changed contour points.

2. The method as claimed in claim 1, wherein detecting the face of the user in the image comprises:

comparing a preset threshold with a difference between coordinates of each eye in a previous frame and coordinates of each eye in a current frame; and

adjusting the coordinates of each eye in the current frame when the difference is less than the threshold.

3. The method as claimed in claim 2, wherein adjusting the coordinates of each eye comprises changing the coordinates of each eye in the current frame to coordinates located between the coordinates of each eye in the previous frame and the coordinates of each eye in the current frame.

4. The method as claimed in claim 1, wherein detecting the contour points of the eye area in the detected face comprises setting a point having a largest change in a brightness value of a pixel in a direction of a vector passing through each contour point of a previously-set eye model from center coordinates of the eye area, as a contour point of the eye area.

5. The method as claimed in claim 1, wherein changing the contour points comprises changing at least some of the detected contour points to contour points of a previously-stored eye area that gazes in the direction toward the reference camera.

6. The method as claimed in claim 1, wherein transforming the eye area comprises performing texture mapping on the eye area by using a forward warping technology or a backward warping technology.

7. The method as claimed in claim 6, wherein the texture mapping is performed within an oval mask including the changed contour points of the eye area.

8. The method as claimed in claim 1, further comprising:

determining whether it is appropriate to correct the gaze direction by performing at least one of;

determining whether a distance between both eyes in the image is less than or equal to a previously-set threshold;

determining whether both eyes are located in a previously-set guide area; and

determining whether a slope of the detected face, which is measured by using coordinates of both eyes, is less than or equal to a previously-set slope.

9. An article of manufacture for correcting a gaze direction of a user in an image, comprising a non-transitory machine-readable storage medium for recording one or more programs, which when executed implement the steps of:

detecting a face of the user in the image;

detecting contour points of an eye area in the detected face;

changing the contour points of the detected eye area in the image to contour points that correspond to a gaze direction toward a reference camera; and

correcting the gaze direction of the user in the image by transforming the eye area according to the changed contour points.

10. A portable terminal for correcting a gaze direction of a user in an image, the portable terminal comprising:

a camera for capturing an image;

a controller for detecting a face of the user in the image, detecting contour points of an eye area in the detected face, changing the contour points of the detected eye area in the image to contour points that correspond to a gaze direction toward a reference camera, and correcting the gaze direction of the user in the image by transforming the eye area according to the changed contour points; and

a display unit for displaying the image into which the controller transforms the eye area.

11. The portable terminal as claimed in claim 10, wherein the controller compares a preset threshold with a difference between coordinates of each eye in a previous frame and coordinates of each eye in a current frame, and adjusts the coordinates of each eye in the current frame when the difference is less than the threshold.

12. The portable terminal as claimed in claim 11, wherein the controller changes the coordinates of each eye in the current frame to coordinates located between the coordinates of each eye in the previous frame and the coordinates of each eye in the current frame.

13. The portable terminal as claimed in claim 10, wherein the controller sets a point having a largest change in a brightness value of a pixel in a direction of a vector passing through each contour point of a previously-set eye model from center coordinates of the eye area, as a contour point of the eye area.

14. The portable terminal as claimed in claim 10, wherein the gaze direction toward the reference camera corresponds to a straight-ahead gaze direction.

15. The portable terminal as claimed in claim 10, wherein the controller changes at least some of the detected contour points to contour points of a previously-stored eye area that gazes in the direction toward the reference camera.

16. The portable terminal as claimed in claim 10, wherein the controller performs texture mapping on the eye area by using a forward warping technology or a backward warping technology.

17. The portable terminal as claimed in claim 16, wherein the texture mapping is performed within an oval mask including the changed contour points of the eye area.

18. The portable terminal as claimed in claim 10, wherein the controller determines whether it is appropriate to correct the gaze direction by performing at least one of:

determining whether a distance between both eyes in the image is less than or equal to a previously-set threshold;

determining whether both eyes are located in a previously-set guide area; and

determining whether a slope of the detected face, which is measured by using coordinates of both eyes, is less than or equal to a previously-set slope.

19. The portable terminal as claimed in claim 10, wherein the controller changes the contour points based on a difference between a gaze direction of the face of the user in the image and the gaze direction toward the reference camera, which is previously set as a reference.

20. The portable terminal as claimed in claim 10, wherein the image is captured while a self-image capturing application, a video phone call application or a mirror application is executed.