Visibility improvement method based on eye tracking, machine-readable storage medium and electronic device

• US 10,514,758 B2
• Filed: 02/04/2019
• Issued: 12/24/2019
• Est. Priority Date: 06/11/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Myung-Sik, Kim, Chang-Yeong, Han, Ick-Sang, Kim, Joon-Oo, Kim, Jinho, Oh, Yun-Je, Jung, Sun-Tae
• Primary Examiner(s): Faragalla, Michael A
• Application Number:
  US16/266,978
• Publication Number:
  US 20190179410A1
• Time in Patent Office: 323 Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

A visibility improvement method using gaze tracking includes detecting a gaze of a user using a camera; determining a focus object at which the user gazes from among at least one object viewed through a display unit of an electronic device; and displaying, on the display unit, an image with high visibility that corresponds to the focus object and has higher visibility than the focus object.

Claims

1. A method of improving visibility by using gaze tracking, the method comprising:

displaying a first image on a display of an electronic device, wherein the first image is a first view of a three-dimensional (3D) map including at least one of a front of a building, a rear of the building, or a side of the building;

while the first image is displayed on the display of the electronic device, obtaining a gaze of a user by using a camera device;

determining a first portion of the first image, based on a dwell time of the gaze of the user;

receiving a second image from a server, wherein a second portion of the second image has higher visibility than visibility of the first portion of the first image based on the first portion of the first image, wherein the second image is a second view of the 3D map;

displaying, on the display of the electronic device, the second image received from the server, wherein the higher visibility of the second portion of the second image includes at least one of a higher resolution, a higher definition, or a higher brightness than at least one of a corresponding resolution, a corresponding definition, or a corresponding brightness of the first portion of the first image; and

while the second image is displayed on the display of the electronic device, determining a movement of a virtual viewpoint of the user based on the gaze of the user.

2. The method of claim 1, further comprising:

detecting an eye of the user by using the camera device,

wherein the gaze is detected based on a position of a pupil of the eye of the user.

3. The method of claim 1, further comprising:

determining whether the user focuses on the first portion of the first image displayed on the display.

4. The method of claim 3, wherein determining whether the user focuses on the first portion of the first image comprises determining whether the user focuses on the first portion of the first image based on a facial expression of the user or a change in a pupil of an eye of the user.

5. The method of claim 1, wherein the second portion of the second image is enlarged and a degree of enlargement of the second portion of the second image is based on a degree of a frown on a face of the user;

or

wherein the virtual viewpoint is determined based on the degree of the frown on the face of the user.

6. The method of claim 1, further comprising:

receiving a command of the user,

wherein the obtaining comprises obtaining the gaze based on the command of the user.

7. The method of claim 1, further comprising:

receiving a command of the user,

wherein determining the first portion of the first image comprises determining the first portion of the first image based on the command of the user.

8. The method of claim 1, wherein the first image is acquired by photographing cities or streets.

9. The method of claim 1, wherein the first image and the second image are acquired from a map database of the server.

10. An electronic device for improving visibility using gaze tracking, comprising:

a camera;

a display;

a communication interface;

at least one memory storing instructions; and

at least one processor configured to execute the instructions to;

control the display to display a first image on the display, wherein the first image is a three-dimensional (3D) map including at least one of a front of a building, a rear of the building, or a side of the building,

while the first image is displayed on the display, control the camera to obtain a gaze of a user;

determine a first portion of the first image, based on a dwell time of the gaze of the user,

control the communication interface to receive a second image from a server, wherein a second portion of the second image has higher visibility than visibility of the first portion of the first image, wherein the second image is a second view of the 3D map,

control the display to display the second image received from the server, wherein the higher visibility of the second portion of the second image includes at least one of a higher resolution, a higher definition, or a higher brightness than at least one of a corresponding resolution, a corresponding definition, or a corresponding brightness of the first portion of the first image, and

while the second image is displayed on the display control to determine a movement of a virtual viewpoint of the user based on the gaze of the user.

11. The electronic device of claim 10, wherein the at least one processor is further configured to:

control the camera to detect an eye of the user,

wherein the gaze of the user is detected based on a position of a pupil of the eye of the user.

12. The electronic device of claim 10, wherein the at least one processor is further configured to:

determine whether the user focuses on the first portion of the first image displayed on the display.

13. The electronic device of claim 10, wherein the at least one processor is further configured to:

determine whether the user focuses on the first portion of the first image based on a facial expression of the user or a change in a pupil of an eye of the user.

14. The electronic device of claim 10, wherein the second portion of the second image is enlarged and a degree of enlargement of the second portion of the second image is based on a degree of a frown of a face of the user or the virtual viewpoint is determined based on the degree of the frown of the face of the user.

15. The electronic device of claim 10, wherein the at least one processor is further configured to:

control to receive a command of the user,

wherein the at least one processor controls the camera to obtain the gaze of the user based on the command of the user.

16. The electronic device of claim 10, wherein the at least one processor is further configured to:

control to receive a command of the user,

wherein the at least one processor controls to determine the first portion of the first image based on the command of the user.

17. The electronic device of claim 10, wherein the first image is an image acquired by photographing cities or streets.

18. The electronic device of claim 10, wherein the first image and the second image acquired from a map database of the server.

19. A system for improving visibility using gaze tracking, the system comprising:

a first device, comprising;

a display;

a first communication interface;

at least one first memory storing first instructions; and

at least one first processor configured to execute the first instructions to;

control the display to display a first image on the display, wherein the first image is a three-dimensional (3D) map including at least one of a front of a building, a rear of the building, or a side of the building,

while the first image is displayed on the display, control a camera device coupled to the first device to obtain a gaze of a user,

determine a first portion of the first image, based on a dwell time of the gaze of the user,

control the first communication interface to transmit information regarding the first portion of the first image towards a second device,

control the first communication interface to receive, from the second device, a second image, wherein a second portion of the second image has higher visibility than visibility of the first portion of the first image, wherein the second image is a second view of the 3D map;

control the display to display the second image obtained from the second device,

wherein the higher visibility of the second portion of the second image includes at least one of a higher resolution, a higher definition, or a higher brightness than at least one of a corresponding resolution, corresponding definition, or a corresponding brightness of the first portion of the first image; and

the second device, the second device comprising;

a second communication interface,

at least one second memory storing second instructions, and

at least one second processor configured to execute the second instructions to;

control the second communication interface to receive the information regarding the first portion of the first image, wherein the first portion of the first image being determined, by using the camera device coupled to the first device, based on the dwell time of the gaze of the user,

control the second communication interface to transmit the second image to the first device based on the information regarding the first portion of the first image.

20. The system of claim 19, wherein the at least one first processor is further configured to:

control the camera device to detect an eye of the user,

wherein the gaze of the user is detected based on a position of a pupil of the eye of the user.

21. The system of claim 19, wherein the at least one first processor is further configured to:

determine whether the user focuses on the first portion of the first image displayed on the display.

22. The system of claim 21, wherein the at least one first processor is further configured to:

determine whether the user focuses on the first portion of the first image based on a facial expression of the user or a change in a pupil of an eye of the user.

23. The system of claim 19, wherein the second portion of the second image is enlarged and a degree of enlargement of the second portion of the second image is based on a degree of a frown of a face of the user or the virtual viewpoint is determined based on the degree of the frown of the face of the user.

24. The system of claim 19, wherein the at least one first processor is further configured to:

control to receive a command of the user,

wherein the at least one first processor controls the camera device to obtain the gaze of the user based on the command of the user.

25. The system of claim 19, wherein the at least one first processor is further configured to:

control to receive a command of the user,

wherein the at least one processor controls to determine the first portion of the first image based on the command of the user.

26. The system of claim 19, wherein the first image and the second image are acquired by photographing cities streets.

27. The system of claim 19, wherein the first image and the second image are acquired from a map database of the second device.