Visibility improvement method based on eye tracking, machine-readable storage medium and electronic device

• US 10,203,754 B2
• Filed: 12/22/2017
• Issued: 02/12/2019
• Est. Priority Date: 06/11/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Myung-Sik, Kim, Chang-Yeong, Han, Ick-Sang, Kim, Joon-Oo, Kim, Jinho, Oh, Yun-Je, Jung, Sun-Tae
• Primary Examiner(s): Faragalla, Michael
• Application Number:
  US15/853,277
• Publication Number:
  US 20180120935A1
• Time in Patent Office: 417 Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

A visibility improvement method using gaze tracking includes detecting a gaze of a user using a camera; determining a focus object at which the user gazes from among at least one object viewed through a display unit of an electronic device; and displaying, on the display unit, an image with high visibility that corresponds to the focus object and has higher visibility than the focus object.

Claims

1. A method of improving visibility using gaze tracking, comprising:

displaying a first image on a display of an electronic device;

while the first image is displayed on the display of the electronic device, obtaining a gaze of a user by using a camera device coupled to the electronic device;

determining a first portion of the first image, based on the gaze of the user obtained by the camera device;

obtaining, from a server, a second image having a second portion with higher visibility than visibility of the first portion of the first image based on the determined first portion;

displaying, on the display of the electronic device, the second image having the second portion with higher visibility obtained from the server,

wherein the second image having the second portion with higher visibility than the visibility of the first portion of the first image has at least one of higher resolution, higher definition, or higher brightness than at least one of a corresponding resolution, definition, or brightness of the first portion of the first image,

wherein the method further comprises;

while the second image is displayed on the display of the electronic device, determining a movement of a virtual viewpoint of the user based on a gaze of the user, and

wherein the second image having the second portion with higher visibility is an actual image acquired from an actual three-dimensional (3D) map obtained by photographing cities or streets.

2. The method of claim 1, further comprising:

detecting an eye of the user using the camera device coupled to the electronic device,

wherein the gaze of the user is detected based on a position of a pupil of the eye of the user.

3. The method of claim 1, further comprising:

determining whether the user focuses on an object.

4. The method of claim 3, wherein determining whether the user focuses on the object comprises determining whether the user focuses on the object based on a facial expression of the user or a change in the pupil of the eye of the user.

5. The method of claim 3, wherein determining whether the user focuses on the object comprises determining whether the user focuses on the object based on a facial expression of the user, and

wherein the second image is enlarged and the degree of enlargement of the second image is based on a degree of a frown of a face of the user or the virtual viewpoint is determined based on a degree of the frown of the face of the user.

6. The method of claim 1, further comprising:

receiving a command of the user,

wherein obtaining the gaze of the user comprises obtaining the gaze of the user based on the command of the user.

7. The method of claim 1, further comprising:

receiving a command of the user,

wherein determining the first portion of the first image comprises determining the first portion of the first image based on the command of the user.

8. The method of claim 1, wherein the second image comprises a focus object.

9. A non-transitory machine-readable storage medium having a program recorded thereon, which when executed, performs a method of improving visibility using gaze tracking, the method comprising:

displaying a first image on a display of an electronic device;

while the first image is displayed on the display of the electronic device, obtaining a gaze of a user by using a camera device coupled to the electronic device;

determining a first portion of the first image, based on the gaze of the user obtained by the camera device;

obtaining, from a server, a second image having a second portion with higher visibility than visibility of the first portion of the first image based on the determined first portion;

displaying, on the display of the electronic device, the second image having the second portion with higher visibility obtained from the server,

wherein the second image having the second portion with higher visibility than the visibility of the first portion of the first image has at least one of higher resolution, higher definition, or higher brightness than at least one of a corresponding resolution, definition, or brightness of the first portion of the first image,

wherein the method further comprises;

while the second image is displayed on the display of the electronic device, determining a movement of a virtual viewpoint of the user based on a gaze of the user, and

wherein the second image having the second portion with higher visibility is an actual image acquired from an actual three-dimensional (3D) map obtained by photographing cities or streets.

10. The non-transitory machine-readable storage medium of claim 9, wherein the method further comprises:

detecting an eye of the user using the camera device coupled to the electronic device, and

wherein the gaze of the user is detected based on a position of a pupil of the eye of the user.

11. The non-transitory machine-readable storage medium of claim 9, wherein the method further comprises:

determining whether the user focuses on an object.

12. The non-transitory machine-readable storage medium of claim 11, wherein determining whether the user focuses on the object comprises determining whether the user focuses on the object based on a facial expression of the user, and

wherein the second image is enlarged and the degree of enlargement of the second image is based on a degree of a frown of a face of the user or the virtual viewpoint is determined based on a degree of the frown of the face of the user.

13. The non-transitory machine-readable storage medium of claim 9, wherein the method further comprises:

receiving a command of the user,

wherein obtaining the gaze of the user comprises obtaining the gaze of the user based on the command of the user.

14. The non-transitory machine-readable storage medium of claim 9, wherein the second image comprises a focus object.

15. An electronic device, comprising:

a camera device that photographs an eye of a user coupled to the electronic device;

a display that displays at least one object; and

a processor configured to;

control the display to display a first image on the display of the electronic device;

while the first image is displayed on the display of the electronic device, obtain a gaze of a user by using the camera device coupled to the electronic device;

determine a first portion of the first image, based on the gaze of the user obtained by the camera device;

obtain, from a server, a second image having a second portion with higher visibility than visibility of the first portion of the first image based on the determined first portion;

control the display of the electronic device to display the second image having the second portion with higher visibility obtained from the server,

wherein the second image having the second portion with higher visibility than the visibility of the first portion of the first image has at least one of higher resolution, higher definition, or higher brightness than at least one of a corresponding resolution, definition, or brightness of the first portion of the first image,

wherein the processor is further configured to;

while the second image is displayed on the display of the electronic device, determining a movement of a virtual viewpoint of the user based on a gaze of the user, and

wherein the second image having the second portion with higher visibility is an actual image acquired from an actual three-dimensional (3D) map obtained by photographing cities or streets.

16. The electronic device of claim 15, wherein the processor is further configured to:

detect an eye of the user using the camera, and

wherein the gaze of the user is detected based on a position of a pupil of the eye of the user.

17. The electronic device of claim 15, wherein the processor is further configured to:

determine whether the user focuses on an object.

18. The electronic device of claim 17, wherein determining whether the user focuses on the object comprises determining whether the user focuses on the object based on a facial expression of the user, and

wherein the second image is enlarged and the degree of enlargement of the second image is based on a degree of a frown of a face of the user or the virtual viewpoint is determined based on a degree of the frown of the face of the user.

19. The electronic device of claim 15, wherein the processor is further configured to:

receive a command of the user, and

wherein obtaining the gaze of the user comprises obtaining the gaze of the user based on the command of the user.

20. The electronic device of claim 15, wherein the second image comprises a focus object.