APPARATUS AND METHOD OF CONTROLLING MOBILE TERMINAL BASED ON ANALYSIS OF USER'S FACE

• US 20130293456A1
• Filed: 05/02/2013
• Published: 11/07/2013
• Est. Priority Date: 05/02/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: SON, Byung-Jun, Hong, Tae-Hwa, Kim, Hong-II
• Granted Patent: US 9,239,617 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G09G 2340/0492 Change of orientation of the displayed image, e.g. upside-down, mirrored
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

An apparatus and method of controlling a mobile terminal by detecting a face or an eye in an input image are provided. The method includes performing face recognition on an input image facing and being captured by an image input unit equipped on the front face of the mobile terminal; determining, based on the face recognition, user state information that includes whether a user exists, a direction of the user'"'"'s face, a distance from the mobile terminal, and/or a position of the user'"'"'s face; and performing a predetermined function of the mobile terminal according to the user state information. According to the method, functions of the mobile terminal may be controlled without direct inputs from the user.

Claims

1. An apparatus for controlling a mobile terminal based on analysis of a user'"'"'s face, the apparatus comprising:

an image input unit;

an image processor configured to perform, upon reception of an input image through the image input unit, face detection or eye detection on the input image; and

a controller configured to determine user state information based on face recognition results or eye recognition results from the image processor, and control to perform a predetermined function of the mobile terminal according to the user state information.

2. The apparatus of claim 1, wherein the user state information includes at least one of the user'"'"'s presence or absence, a distance from the mobile terminal, and a position of the user'"'"'s face.

3. The apparatus of claim 1, wherein the controller controls the mobile terminal based on the user'"'"'s face analysis results which are characterized by driving the image input unit when a predetermined event occurs.

4. The apparatus of claim 3, wherein the controller drives the image input unit if a screen lock mode begins, unlocks the screen if it is determined from the recognition results on the input image from the image input unit that the user exists, and maintains the screen to be unlocked even if a predetermined time has elapsed, as long as it is determined that the user exists.

5. The apparatus of claim 1, wherein the image processor determines whether a face is detected in the input image with a face recognition algorithm, and performs eye detection on the input image with an eye recognition algorithm if the face is not detected.

6. The apparatus of claim 5, wherein the image processor detects one or more eye candidate areas with the eye recognition algorithm, and determines a final eye area from among the detected eye candidate areas according to an eye detection scheme that uses skin color.

7. The apparatus of claim 6, wherein the image processor divides the detected eye candidate areas into a predetermined number of areas in horizontal and vertical directions, calculates a proportion of skin color in areas divided in the vertical direction except for a middle area and a proportion of non-skin color in a portion of areas divided in the horizontal direction, and determines the areas to be the final eye area if the calculated proportions of the skin color and the non-skin color are greater than predetermined thresholds, respectively.

8. The apparatus of claim 6, wherein the image processor determines pixel values that correspond to the skin color by using values of Y, Cb, and/or Cr of the detected eye candidate areas.

9. The apparatus of claim 6, wherein the image processor determines, upon determination of the final eye area, top and bottom of the face by normalizing a surrounding area of the final eye area.

10. The apparatus of claim 1, wherein the image processor compensates a part of a face determination area that deviates from an edge of the input image using at least one of a mirroring method and a template matching method while the input image is scanned by moving the face determination area horizontally or vertically.

11. The apparatus of claim 10, wherein the image processor, if the face determination area moves to a position where a part of the face determination area deviates from a left or right edge of the input image, compensates the part by filling the part with pixel information of a part of the face determination area included in the image using the mirroring method and detects a partial face from the face determination area that includes the compensated part.

12. The apparatus of claim 10, wherein the image processor, if the face determination area moves to a position where a part of the face determination area deviates from an upper or lower edge of the input image, compensates the part by using the template matching method to attach an image resulting from the template matching to the part and detects a partial face from the face determination area that includes the compensated part.

13. The apparatus of claim 10, wherein the image processor, if the face determination area moves to a position where a part of the face determination area deviates from a corner of the input image, compensates the part by applying both the mirroring method and the template matching method and detects a partial face from the face determination area that includes the compensated part.

14. A method of controlling a function based on analysis of a user'"'"'s face in a mobile terminal, the method comprising:

driving, upon occurrence of a predetermined event, an image input unit of the mobile terminal;

performing, upon reception of an input image through the image input unit, face detection on the input image;

performing eye detection on the input image if the face detection fails;

determining user state information based on face recognition results or eye recognition results; and

performing a predetermined function of the mobile terminal according to the user state information.

15. The method of claim 14, wherein the user state information includes at least one of the user'"'"'s presence or absence, a distance from the mobile terminal, and a position of the user'"'"'s face.

16. The method of claim 14, wherein driving the image input unit comprises, driving the image input unit if a screen is changed to be locked.

17. The method of claim 16, wherein performing the predetermined function of the mobile terminal comprises

unlocking the screen if it is determined from the recognition results that the user exists, and

maintaining the screen to be unlocked as long as it is determined that the user exists.

18. The method of claim 14, wherein performing the eye detection comprises,

detecting one or more eye candidate areas with an eye recognition algorithm;

and

determining a final eye area from among the eye candidate areas by using an eye detection scheme that uses skin color.

19. The method of claim 18, wherein the final eye area is determined by dividing the detected eye candidate areas into a predetermined number of areas in horizontal and vertical directions, calculating a proportion of skin color in areas divided in the vertical direction except for a middle area and a proportion of non-skin color in a portion of areas divided in the horizontal direction, and determining the areas to be the final eye area if the calculated proportions of the skin color and the non-skin color are greater than predetermined thresholds, respectively.

20. The method of claim 19, wherein pixel values corresponding to the skin color are determined by using values of Y, Cb, and/or Cr of the detected eye candidate areas.

21. The method of claim 14, further comprising:

scanning the input image horizontally or vertically using a face determination area; and

if the face determination area moves to a position where a part of the face determination area deviates from an edge of the input image, compensating a part of the face determination area that deviates from the edge of the input image using at least one of a mirroring method and a template matching method.

22. The method of claim 21, wherein performing face detection comprises:

if the face determination area moves to a position where a part of the face determination area deviates from a left or right edge of the input image, compensating the part by filling the part with pixel information of a part of the face determination area included in the image using the mirroring method; and

detecting a partial face from the face determination area that includes the compensated part.

23. The method of claim 21, wherein performing face detection comprises:

if the face determination area moves to a position where a part of the face determination area deviates from an upper or lower edge of the input image, compensating the part by using the template matching method to attach an image resulting from the template matching to the part; and

detecting a partial face from the face determination area that includes the compensated part.

24. The method of claim 21, wherein performing face detection comprises:

if the face determination area moves to a position where a part of the face determination area deviates from a corner of the input image, compensating the part by applying both the mirroring method and the template matching method; and

detecting a partial face from the face determination area that includes the compensated part.