Method and apparatus for user interface using gaze interaction

• US 10,324,524 B2
• Filed: 01/27/2014
• Issued: 06/18/2019
• Est. Priority Date: 02/21/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Choi, Chang Kyu, Suh, Sung Joo, Park, Joon Ah, Park, Du Sik
• Primary Examiner(s): Azari, Sepehr
• Application Number:
  US14/164,617
• Publication Number:
  US 20140232638A1
• Time in Patent Office: 1,968 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

A method and apparatus for a user interface using a gaze interaction is disclosed. The method for the user interface using the gaze interaction may include obtaining an image including eyes of a user, estimating a gaze position of the user, using the image including the eyes of the user, and determining whether to activate a gaze adjustment function for controlling a device by a gaze of the user, based on the gaze position of the user with respect to at least one toggle area on a display.

Claims

1. A method for a user interface using a gaze interaction, the method comprising:

obtaining, using a camera, an image including eyes of a user using at least two internal light sources included in a display of a device comprising pixels for displaying an image;

changing at least one of a position or a shape of the at least two internal light sources, in response to an occurrence of noise by an external light source;

estimating, using the obtained image, a gaze position of the user with respect to at least one toggle area on the display, based on the at least two changed internal light sources; and

determining, based on the estimated gaze position, whether to activate a gaze adjustment function, wherein the gaze adjustment function controls an interface of the device using a gaze of the user,

wherein the determining of whether to activate the gaze adjustment function comprises

identifying a current mode of the gaze adjustment function;

determining whether the gaze position matches the at least one toggle area;

in response to the current mode being a deactivation mode, changing a mode of the gaze adjustment function to an activation mode, based on a result of the determination; and

in response to the current mode being the activation mode, changing the mode of the gaze adjustment function to the deactivation mode, based on the result of the determination.

2. The method of claim 1, wherein the at least one toggle area comprises:

at least one calibration point.

3. The method of claim 1, wherein the at least one toggle area comprises:

at least one of a plurality of objects displayed on the display.

4. The method of claim 1, wherein the estimating of the gaze position of the user further comprises:

extracting respective positions of at least two glints, being an area in which the at least two internal light sources are displayed on the eyes of the user, respectively, based on the image including the eyes of the user;

calculating a position of a cornea center of the eyes of the user, using respective positions of the at least two internal light sources and the respective positions of the at least two glints;

calculating respective positions of a pupil center of each of the eyes of the user, using the respective positions of the cornea center of the eyes of the user;

calculating respective directions of an optical axis of each of the eyes of the user, using the calculated respective positions of the cornea center and the calculated respective positions of the pupil center of each of the eyes of the user;

calculating respective directions of a visual axis of each of the eyes of the user, based on the calculated respective directions of the optical axis of each of the eyes of the user; and

identifying the gaze position, using an intersection point of the calculated respective visual axes of each of the eyes of the user.

5. The method of claim 1, further comprising:

removing noise by at least one external light source,

wherein the removing of the noise by the at least one external light source comprises;

changing at least one of a position, a color, and a shape of the at least two internal light sources by determining at least one of a position, a color, and a shape of an image associated with the at least two internal light sources;

separating the at least two changed internal light sources and the at least one external light source; and

estimating the gaze position of the user, based on the at least two changed internal light sources.

6. The method of claim 1, wherein the determining of whether to activate the gaze adjustment function comprises:

determining whether the gaze position matches the at least one toggle area; and

setting the gaze adjustment function to be an activation mode or a deactivation mode, based on a result of the determination.

7. The method of claim 6, wherein the determining of whether the gaze position matches the at least one toggle area further comprises:

determining whether the gaze position matches the at least one toggle area within a predetermined period of time.

8. The method of claim 6, wherein the determining of whether the gaze position matches the at least one toggle area further comprises:

determining whether the gaze position matches the at least one toggle area, based on a predetermined pattern.

9. The method of claim 6, wherein the determining of whether to activate the gaze adjustment function further comprises:

setting a default value of the gaze adjustment function to be the deactivation mode;

setting the gaze adjustment function to be the activation mode during a predetermined period of time when the gaze position matches the at least one toggle area, based on the result of the determination.

10. The method of claim 1, further comprising:

setting at least two vertex areas from among a plurality of vertex areas of the display to be the at least two internal light sources.

11. The method of claim 1, wherein the at least one toggle area comprises a predetermined area present on the display for determining whether to activate or deactivate the gaze adjustment function for the user interface using the gaze interaction.

12. The method of claim 1, wherein the determining whether the gaze position matches the at least one toggle area comprises:

determining whether a sequence of gaze positions matches a sequence of toggle areas based on a predetermined pattern.

13. A method of changing content of a mobile device having a display with a plurality of toggle areas, the display further comprising pixels for displaying an image, the method comprising:

obtaining an image including eyes of the user using at least two internal light sources included in the display;

changing at least one of a position or a shape of the at least two internal light sources, in response to an occurrence of noise by an external light source;

monitoring, based on the obtained image, a gaze position of the user, based on the at least two changed internal light sources;

determining, based on the monitored gaze position, whether to activate a gaze adjustment function, wherein the gaze adjustment function controls an interface of the mobile device using a gaze of the user;

determining, based on the monitored gaze position, whether the user has gazed at the plurality of toggle areas according to a predetermined pattern; and

changing the content of the mobile device based on the monitored gaze position of the user and a predetermined rule when an activation mode has been activated,

wherein the determining of whether to activate the gaze adjustment function comprises

identifying a current mode of the gaze adjustment function;

determining whether the gaze position matches the toggle areas;

in response to the current mode being a deactivation mode, changing a mode of the gaze adjustment function to the activation mode, based on a result of the determination; and

in response to the current mode being the activation mode, changing the mode of the gaze adjustment function to the deactivation mode, based on the result of the determination.

14. The method of claim 13, wherein the predetermined rule comprises scrolling the content of the mobile device downward when the gaze position of the user is monitored as directed towards a bottom portion of the display of the mobile device.

15. The method of claim 13, wherein the predetermined rule comprises enlarging the content displayed on a predetermined portion of the display of the mobile device when the gaze position of the user is monitored as directed towards the predetermined portion.

16. A method for controlling content of a mobile device having a display with a toggle area, the method comprising:

obtaining an image including eyes of a user using at least two internal light sources included in the display of the mobile device comprising pixels for displaying an image;

changing at least one of a position or a shape of the at least two internal light sources, in response to an occurrence of noise by an external light source;

monitoring a gaze position of the user based on the at least two changed internal light sources;

determining, based on the monitored gaze position, whether the user has gazed at the toggle area for a threshold amount of time;

upon determining that the user has gazed at the toggle area for the threshold amount of time, setting a mode of a gaze adjustment function of the mobile device to an activation mode,

wherein the gaze adjustment function controls an interface of the mobile device using a gaze of the user; and

controlling the content of the mobile device based on a position of a user gaze while the mode of the gaze adjustment function of the mobile device is in the activation mode.

17. An apparatus for a user interface using a gaze interaction, the apparatus comprising:

a display comprising pixels for displaying an image;

a camera;

a processor configured to

obtain, using the camera, an image including eyes of a user, using at least two internal light sources included in the display;

changing at least one of a position or a shape of the at least two internal light sources, in response to an occurrence of noise by an external light source;

estimate, using the obtained image, the gaze position of the user with respect to at least one toggle area on the display, based on the at least two changed internal light sources; and

determine, based on the estimated gaze position, whether to activate a gaze adjustment function, wherein the gaze adjustment function controls an interface of the apparatus using the gaze of the user,

wherein the processor is configured to determine whether to activate the gaze adjustment function by

identifying a current mode of the gaze adjustment function;

determining whether the gaze position matches the at least one toggle area;

in response to the current mode being a deactivation mode, changing a mode of the gaze adjustment function to an activation mode, based on a result of the determination; and

in response to the current mode being the activation mode, changing the mode of the gaze adjustment function to the deactivation mode, based on the result of the determination.

18. A user interface method in a mobile device having a display comprising pixels for displaying an image, the method comprising:

displaying an icon on the display while a user waits for content to be loaded onto the display;

obtaining an image including eyes of the user using at least two internal light sources included in the display;

changing at least one of a position or a shape of the at least two internal light sources, in response to an occurrence of noise by an external light source;

estimating, using the obtained image, a gaze position of the user by using the icon as a calibration point, based on the at least two changed internal light sources; and

determining, based on the estimated gaze position, whether to activate a gaze adjustment function, wherein the gaze adjustment function controls an interface of the device using a gaze of the user,

wherein the determining of whether to activate the gaze adjustment function comprises

identifying a current mode of the gaze adjustment function;

determining whether the gaze position matches the at least one toggle area;

in response to the current mode being a deactivation mode, changing a mode of the gaze adjustment function to an activation mode, based on a result of the determination; and

in response to the current mode being the activation mode, changing the mode of the gaze adjustment function to the deactivation mode, based on the result of the determination.

19. A method of unlocking content of a mobile device having a display with a plurality of toggle areas, the display further comprising pixels for displaying an image, the method comprising:

obtaining an image including eyes of the user using at least two internal light sources included in the display;

changing at least one of a position or a shape of the at least two internal light sources, in response to an occurrence of noise by an external light source;

monitoring, based on the obtained image, a gaze position of the user based on the at least two changed internal light sources;

determining, based on the monitored gaze position, whether to activate a gaze adjustment function, wherein the gaze adjustment function controls an interface of the mobile device using a gaze of the user;

determining, based on the monitored gaze position, whether the user has gazed at the plurality of toggle areas according to a pattern; and

unlocking the content of the mobile device upon determining that the user has gazed at the plurality of toggle areas according to the pattern,

wherein the determining of whether to activate the gaze adjustment function comprises

identifying a current mode of the gaze adjustment function;

determining whether the gaze position matches the toggle areas;

in response to the current mode being a deactivation mode, changing a mode of the gaze adjustment function to an activation mode, based on a result of the determination; and

in response to the current mode being the activation mode, changing the mode of the gaze adjustment function to the deactivation mode, based on the result of the determination.

20. A method of locking content of a mobile device having a display with a plurality of toggle areas, the display further comprising pixels for displaying an image, the method comprising:

obtaining an image including eyes of a user using at least two internal light sources included in the display;

changing at least one of a position or a shape of the at least two internal light sources, in response to an occurrence of noise by an external light source;

monitoring, based on the obtained image, a gaze position of the user based on the at least two changed internal light sources;

determining, based on the monitored gaze position, whether to activate a gaze adjustment function, wherein the gaze adjustment function controls an interface of the mobile device using a gaze of the user;

determining, based on the monitored gaze position, whether the user has gazed at the toggle areas according to a predetermined pattern; and

locking the content of the mobile device when it has been determined that the user has gazed at the toggle areas according to the predetermined pattern,

wherein the determining of whether to activate the gaze adjustment function comprises

identifying a current mode of the gaze adjustment function;

determining whether the gaze position matches the toggle areas;

in response to the current mode being a deactivation mode, changing a mode of the gaze adjustment function to an activation mode, based on a result of the determination; and

in response to the current mode being the activation mode, changing the mode of the gaze adjustment function to the deactivation mode, based on the result of the determination.