METHOD FOR CORRECTING USER'S GAZE DIRECTION IN IMAGE, MACHINE-READABLE STORAGE MEDIUM AND COMMUNICATION TERMINAL

• US 20140016871A1
• Filed: 02/01/2013
• Published: 01/16/2014
• Est. Priority Date: 07/13/2012
• Status: Active Grant

• Current Assignee: Postech Academy-Industry Foundation, Samsung Electronics Co. Ltd.
• Original Assignee: Postech Academy-Industry Foundation, Samsung Electronics Co. Ltd.
• Inventors: SON, Byung-Jun, HONG, Tae-Hwa, KIM, Dai-Jin, SHIN, Jong-Ju, CHOI, In-Ho
• Granted Patent: US 9,111,171 B2
• Time in Patent Office: Days
• US Class Current: 382/201
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2207/20124 Active shape model [ASM]
• CPC Class Codes: G06T 2207/30041 Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/149 involving deformable models, e.g. active contour models
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation

Abstract

A method and mobile terminal for correcting a gaze of a user in an image includes setting eye outer points that define an eye region of the user in an original image, transforming the set eye outer points to a predetermined reference camera gaze direction, and transforming the eye region of the original image based on the transformed eye outer points.

Claims

1. A method of correcting a gaze of a user in an image, the method comprising:

setting eye outer points that define an eye region of the user in an original image;

transforming the set eye outer points to a predetermined reference camera gaze direction; and

transforming the eye region of the original image based on the transformed eye outer points.

2. The method of claim 1, wherein setting the eye outer points comprises:

superposing an average shape of an eye model including initial eye outer points on the eye region of the original image; and

detecting a point of having a greatest change in a pixel brightness value in a normal line of each initial eye outer point, and setting the detected point to be a corresponding eye outer point.

3. The method of claim 2, wherein detection ranges for the eye outer points are set to be different from each other based on locations of the initial eye outer points.

4. The method of claim 2, wherein the eye outer points are set by extracting eye model shape parameters of the eye outer points associated with eigen vectors of the eye model, and limiting a range of each of the extracted eye model shape parameters.

5. The method of claim 1, wherein the predetermined reference camera gaze direction corresponds to a direction corresponding to a gaze facing straight on to a camera of a device.

6. The method of claim 1, wherein transforming the eye outer points comprises:

shifting each eye outer point to a location obtained by adding a coordinate of each eye outer point to an average displacement between an eye gazing at a lower side and an eye gazing at a front side.

7. The method of claim 1, wherein transforming the eye region of the original image comprises:

mapping pixels inside a first eye outline that sequentially connects the set eye outer points and pixels inside a second eye outline that sequentially connects the transformed eye outer points.

8. The method of claim 7, wherein each of the first and second eye outlines is divided into a plurality of triangles formed by connecting corresponding eye outer points, and the mapping is performed on corresponding triangles of the first and the second eye outlines.

9. The method of claim 1, wherein transforming the eye region of the original image comprises:

mapping pixels inside a first window including a first eye outline that sequentially connects the set eye outer points and pixels inside a second window including a second eye outline that sequentially connects the transformed eye outer points.

10. The method of claim 1, wherein transforming the eye region of the original image comprises:

mapping pixels of a predetermined virtual eye and pixels inside a window including an eye outline that sequentially connects the transformed eye outer points.

11. The method of claim 10, wherein a brightness of the predetermined virtual eye is adjusted based on a difference between a brightness of the eye region of the original image and the brightness of the predetermined virtual eye, and the mapping is performed based on the virtual eye of which brightness is adjusted.

12. The method of claim 10, wherein the virtual eye corresponds to a three-dimensional (3D) virtual eye.

13. A non-transitory machine-readable storage medium in which a program for executing a method of correcting a gaze of a user in an image as claimed is recorded, the method comprising:

setting eye outer points that define an eye region of the user in an original image;

transforming the set eye outer points to a predetermined reference camera gaze direction; and

transforming the eye region of the original image based on the transformed eye outer points.

14. A mobile terminal for correcting a gaze of a user in an image, the mobile terminal comprising:

a camera for capturing an original image;

a processor for setting eye outer points that define an eye region of the user in an original image, transforming the set eye outer points to a predetermined reference camera gaze direction, and transforming the eye region of the original image based on the transformed eye outer points; and

a display unit for displaying a transformed image including the transformed eye region.

15. The mobile terminal of claim 14, wherein setting the eye outer points includes superposing an average shape of an eye model including initial eye outer points on the eye region of the original image, and detecting a point of having a greatest change in a pixel brightness value in a normal line of each initial eye outer point, and setting the detected point to be a corresponding eye outer point.

16. The mobile terminal of claim 15, wherein detection ranges for the eye outer points are set to be different from each other based on locations of the initial eye outer points.

17. The mobile terminal of claim 15, wherein the eye outer points are set by extracting eye model shape parameters of the eye outer points associated with eigen vectors of the eye model, and limiting a range of each of the extracted eye model shape parameters.

18. The mobile terminal of claim 14, wherein the predetermined reference camera gaze direction corresponds to a direction corresponding to a gaze facing straight on to the camera.

19. The mobile terminal of claim 14, wherein transforming the eye outer points includes shifting each eye outer point to a location obtained by adding a coordinate of each eye outer point to an average displacement between an eye gazing at a lower side and an eye gazing at a front side.

20. The mobile terminal of claim 14, wherein transforming the eye region of the original image includes mapping pixels inside a first eye outline that sequentially connects the set eye outer points and pixels inside a second eye outline that sequentially connects the transformed eye outer points.

21. The mobile terminal of claim 20, wherein each of the first and second eye outlines is divided into a plurality of triangles formed by connecting corresponding eye outer points, and the mapping is performed on corresponding triangles of the first and the second eye outlines.

22. The mobile terminal of claim 14, wherein transforming the eye region of the original image includes mapping pixels inside a first window including a first eye outline that sequentially connects the set eye outer points and pixels inside a second window including a second eye outline that sequentially connects the transformed eye outer points.

23. The mobile terminal of claim 14, wherein transforming the eye region of the original image includes mapping pixels of a predetermined virtual eye and pixels inside a window including an eye outline that sequentially connects the transformed eye outer points.

24. The mobile terminal of claim 23, wherein a brightness of the predetermined virtual eye is adjusted based on a difference between a brightness of the eye region of the original image and the brightness of the predetermined virtual eye, and the mapping is performed based on the virtual eye of which brightness is adjusted.

25. The mobile terminal of claim 23, wherein the virtual eye corresponds to a three-dimensional (3D) virtual eye.