Method and apparatus for controlling display of digital content using eye movement

• US 10,409,366 B2
• Filed: 04/28/2014
• Issued: 09/10/2019
• Est. Priority Date: 04/28/2014
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Beri, Tarun, Tyagi, Vaibhav
• Primary Examiner(s): Trapanese, William C
• Application Number:
  US14/263,496
• Publication Number:
  US 20150309565A1
• Time in Patent Office: 1,961 Days
• Field of Search: None
• CPC Class Codes: G06F 2203/04805 Virtual magnifying lens, i.e. window or frame movable on top of displayed information to enlarge it for better reading or selection
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 3/053 Detail-in-context presentations fisheye or wide-angle transformations G06T3/047

Abstract

A computer implemented method and apparatus for controlling display of displayed digital content using eye movement. The method comprises mapping eye movements, tracked by a camera of a display device, to an area of interest of displayed content. A window containing a magnified version of display content from within the area of interest is generated. The magnified display content from the area of interest is displayed together with display content from outside the area of interest. A hovering pointer gesture is optionally displayed within the area of interest or a selectable function is performed if display content within the area of interest corresponds to a user interface function or a navigation option. According to some embodiments, the window containing magnified display content is only generated and displayed when the display content within the area of interest does not correspond to a user selectable function.

Claims

1. A computer implemented method comprising:

providing, for display to a user by way of a display screen on a user client device, a graphical user interface comprising display content;

tracking, by way of a camera of the user client device, user eye movements with respect to the display content on the display screen;

identifying an area of interest associated with a location of eye focus on the display screen;

determining that the area of interest of the display content does not correspond to a graphical user interface object associated with a function;

generating, automatically without further user input in response to the area of interest of the display content not corresponding to the graphical user interface object, a window containing a first subset of the display content from the area of interest;

displaying the window together with a portion of the display content from outside the area of interest, wherein the first subset of the display content within the window is displayed at a higher zoom level than the portion of the display content from outside the area of interest;

determining a change of eye focus of the user within the window; and

based on the change of eye focus, modifying the display content within the window from the first subset of the display content from the area of interest to a second subset of the display content from the area of interest.

2. The method of claim 1, wherein identifying the area of interest comprises determining a first position on the display screen of the user client device where a left eye of a user is focused and a second position on the display screen where a right eye of the user is focused.

3. The method of claim 1, further comprising displaying a next page of the display content when the tracked eye movements reach a bottom of a current page.

4. The method of claim 1, wherein the higher zoom level of the display content within the window is adjustable.

5. The method of claim 1, further comprising removing the window when the eye movements in the window are not trackable.

6. The method of claim 2, further comprising:

determining that a distance between a left eye focal point and a right eye focal point falls below a width threshold; and

generate a smaller area of interest, wherein the smaller area of interest is larger than a predetermined threshold.

7. The method of claim 1, further comprising:

determining, based on the user eye movements, that a focal position remains on a selectable object of the display content for a threshold period of time; and

in response to determining that the focal position remains on the selectable object for the threshold period of time, automatically selecting the selectable object thereby causing performance of an operation associated with the selectable object.

8. A system for controlling display of digital content using eye movement comprising:

a display;

a camera;

at least one processor; and

a non-transitory storage medium comprising instructions thereon that, when executed by the at least one processor, cause the system to;

provide, for display to a user by way of the display, a graphical user interface comprising display content;

track, by way of the camera, eye movements with respect to the display content on the display;

map an eye movement to a portion of display content defining an area of interest displayed at a first zoom level;

determine that the area of interest of the display content does not correspond to a graphical user interface object associated with a function;

display, automatically without further user input in response to the area of interest of the display content not corresponding to the graphical user interface object, a magnification window over the display content, the magnification window comprising a first portion of the display content, wherein the first portion of the display content is located in the area of interest and displayed at a second zoom level, where the second zoom level is greater than the first zoom level;

determine a change of eye focus of the user within the magnification window; and

based on the change of eye focus modifying the display content within the magnification window from the first portion of the display content from the area of interest to a second portion of the display content from the area of interest.

9. The system of claim 8, further comprising instructions that, when executed by the at least one processor, cause the system to:

determine that a distance between a left eye focal point and a right eye focal point falls below a width threshold; and

generate a smaller area of interest, wherein the smaller area of interest is larger than a predetermined threshold.

10. The system of claim 8, further comprising instructions thereon that, when executed by the at least one processor, cause the system to:

adjust a location of the magnification window in response to detecting that the eye movements indicate a change in the area of interest.

11. The system of claim 8, further comprising instructions thereon that, when executed by the at least one processor, cause the system to adjust the second zoom level of the display content.

12. The system of claim 8, further comprising instructions thereon that, when executed by the at least one processor, cause the system to remove the magnification window when the eye movements in the magnification window are not trackable.

13. The system of claim 8, further comprising instructions that, when executed by the at least one processor, cause the system to:

determine, based on the eye movement, that a focal position remains on a selectable object of the display content for a threshold period of time by tracking the eye movement by way of the camera to identify a first position on the display where a left eye is focused and a second position on the display where a right eye is focused; and

in response to determining that the focal position remains on the selectable object for the threshold period of time, automatically select the selectable object thereby causing performance of an operation associated with the selectable object.

14. A non-transitory computer readable medium for storing computer instructions that, when executed by at least one processor, cause a computing device to:

provide, for display to a user by way of a display screen on a user client device, a graphical user interface comprising display content;

track, by way of a camera of the user client device, eye movements with respect to the display content on the display screen;

based on a tracked eye movement;

identify an area of interest in the display content on the display screen associated with a location of eye focus;

determine that the area of interest of the display content does not correspond to a graphical user interface object associated with a function;

generate, automatically without further user input in response to the area of interest of the display content not corresponding to the graphical user interface object, a window containing a magnified version of a first subset of the display content from within the area of interest;

display the magnified first subset of the display content within the window together with a portion of the display content from outside the area of interest, wherein the first subset of the display content within the window is displayed at a higher zoom level than the portion of the display content from outside the area of interest;

determine a change of eye focus of the user within the window; and

based on the change of eye focus, modify the display content within the window from the first subset of the display content from the area of interest to a second subset of the display content from the area of interest.

15. The non-transitory computer readable medium of claim 14, further comprising instructions, that when executed by at least one processor, cause the computing device to:

determine that a distance between a left eye focal point and a right eye focal point falls below a width threshold; and

generate a smaller area of interest, wherein the smaller area of interest is larger than a predetermined threshold.

16. The non-transitory computer readable medium of claim 14, further comprising instructions, that when executed by at least one processor, cause the computing device to:

display a next page of the displayed content when the tracked eye movements reach a bottom of a current page; and

remove the window when the eye movements in the window are not trackable.

17. The non-transitory computer readable medium of claim 14, further comprising instructions, that when executed by at least one processor, cause the computing device to maintain content boundaries of the displayed content outside the window.

18. The non-transitory computer readable medium of claim 17, wherein the displayed content outside the window is not altered and remains displayed outside the window.

19. The non-transitory computer readable medium of claim 14, further comprising instructions, that when executed by at least one processor, cause the computing device to:

determine, based on the eye movement, that a focal position remains on a selectable object of the display content for a threshold period of time by tracking eye movements by way of the camera to identify a first position on the display screen where a left eye of is focused and a second position on the display screen where a right eye is focused; and

in response to determining that the focal position remains on the selectable object for the threshold period of time, automatically select the selectable object thereby causing performance of an operation associated with the selectable object.

20. The non-transitory computer readable medium of claim 14, further comprising instructions, that when executed by at least one processor, cause the computing device to display a perspective view window as the window.