Image manipulation based on tracked eye movement

• US 8,564,533 B2
• Filed: 09/25/2009
• Issued: 10/22/2013
• Est. Priority Date: 07/10/2009
• Status: Active Grant

• Current Assignee: Peking University
• Original Assignee: Peking University
• Inventors: Yuan, Xiaoru
• Primary Examiner(s): SIMPSON, LIXI CHOW
• Application Number:
  US12/567,684
• Publication Number:
  US 20110006978A1
• Time in Patent Office: 1,488 Days
• Field of Search: None
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance

Abstract

The disclosure relates to controlling and manipulating an image of an object on a display device based on tracked eye movements of an observer. When the object is displayed according to an initial view, the observer'"'"'s eye movement is tracked and processed in order to determine the focus of the observer'"'"'s attention or gaze on the image. Thereafter, the displayed image is modified to provide a better view of the part of the object in which the observer is most interested. This is accomplished by modifying at least one of the spatial positioning of the object within the viewing area, the angle of view of the object, and the viewing direction of the object.

Claims

1. A method comprising:

displaying an image of an object in a viewing area;

tracking an eye gaze at a predetermined sampling interval while the image is displayed;

obtaining a data point of the tracked eye gaze at the predetermined sampling interval, the data point representing a detected location of the eye gaze within the viewing area;

analyzing the data point obtained during the predetermined sampling interval;

determining a range of the viewing area in which a predetermined concentration of the analyzed data point is located;

modifying the image when the determined range is smaller than a first predetermined range, wherein the displayed object of the modified image is magnified according to a zoom-in operation;

modifying the image when the determined range is larger than a second predetermined range, wherein the displayed object of the modified image is shrunk according to a zoom-out operation; and

displaying the modified image.

2. The method of claim 1, further comprising:

analyzing the data point to identify a region of the displayed object corresponding to a high degree of eye attention based on the tracked eye gaze; and

modifying the image by moving the displayed object so that, in the modified image, the identified region of the displayed object is moved to a different location of the viewing area.

3. The method of claim 2, wherein the modifying step moves the identified region of the displayed object to the center of the viewing area.

4. The method of claim 1, wherein the first and second predetermined ranges are defined according to predetermined angles of view, respectively.

5. The method of claim 4, further comprising:

computing an angle of view of the viewing area based on the tracked eye gaze,

wherein the first and second predetermined ranges are established relative to the computed angle of view of the entire viewing area.

6. The method of claim 1, wherein the first predetermined range is defined according to first set of height and width parameters, and the second predetermined range is defined according to a second set of height and width parameters.

7. A method comprising,

displaying an image of an object in a viewing area, wherein a surface of the displayed object is displayed three-dimensionally on the viewing area;

tracking an eye gaze at a predetermined sampling interval while the image is displayed;

obtaining a data point of the tracked eye gaze at the predetermined sampling interval, the data point representing a detected location of the tracked eye gaze within the viewing area;

identifying eye attention points on the surface of the displayed object by applying an unprojection process to translate the obtained data point to a point on the three-dimensionally displayed surface of the displayed object;

determining a modification to the image based on the identified eye attention points;

modifying the image in accordance with the modification; and

displaying the modified image.

8. The method of claim 7, further comprising:

mapping the identified eye attention points to a plurality of surface regions of the displayed object; and

determining the surface region with a highest degree of saliency based on the mapping; and

wherein the modification of the image is determined based on the determined surface region.

9. The method of claim 8, further comprising:

modifying the image by rotating the displayed object so that, in the modified image, the viewing direction of the displayed object faces the determined surface region.

10. The method of claim 8, wherein the unprojection process is applied by multiplying an unprojection matrix with coordinates of the two-dimensional gaze location, wherein the unprojection matrix is calculated based on a projection matrix used for projecting the surface of the displayed object onto the viewing area.

11. A method comprising:

displaying an image of an object in a viewing area, wherein a surface of the displayed object is displayed three-dimensionally on the viewing area;

tracking an eye gaze at a predetermined sampling interval while the image is displayed;

obtaining a data point of the tracked eye gaze at the predetermined sampling interval, the data point representing a detected location of the tracked eye gaze within the viewing area;

mapping a degree of eye attention to a plurality of regions of the viewing area based on the obtained data point;

when the region of the viewing area with the highest degree of eye attention is outside of the displayed object, modifying the image by rotating the displayed object toward the region of the viewing area having the highest degree of eye attention; and

displaying the modified image.

12. A non-transitory computer-readable medium storing executable instructions that, when executed, cause a computer to perform operations comprising:

displaying an image of an object in a viewing area;

tracking an eye gaze at a predetermined sampling interval while the image is displayed;

obtaining a data point of the tracked eye gaze at the predetermined sampling interval, the data point representing a detected location of the tracked eye gaze within the viewing area;

analyzing the data point obtained during the predetermined sampling interval;

determining a range of the viewing area in which a predetermined concentration of the analyzed data point is located;

modifying the image when the determined range is smaller than a first predetermined range, wherein the displayed object of the modified image is magnified according to a zoom-in operation;

modifying the image when the determined range is larger than a second predetermined range, wherein the displayed object of the modified image is shrunk according to a zoom-out operation; and

displaying the modified image.

13. The non-transitory computer readable medium of claim 12, performing further operations including:

analyzing the obtained data point to identify a region of the displayed object corresponding to a high degree of the eye attention based on the tracked eye gaze; and

modifying the image by moving the displayed object so that, in the modified image, the determined region of the displayed object is moved to the center of the viewing area.

14. The non-transitory computer readable medium of claim 12, wherein the first and second predetermined ranges are defined according to predetermined angles of view, respectively.

15. The non-transitory computer readable medium of claim 14, performing further operations including:

computing an angle of view of the entire viewing area based on the tracked eye gaze,

wherein the first and second predetermined ranges are established relative to the computed angle of view of the entire viewing area.

16. The non-transitory computer readable medium of claim 12, wherein the first predetermined range is defined according to first set of height and width parameters, and the second predetermined range is defined according to a second set of height and width parameters.

17. The non-transitory computer readable medium of claim 16, wherein the first and second predetermined ranges are established relative to the height and width of the viewing area.

18. The non-transitory computer readable medium of claim 12, performing further operations including:

receiving inputs, via an input device, that establish the first and second predetermined ranges.

19. The non-transitory computer readable medium of claim 12, wherein a surface of the displayed object is geometrically defined in a three-dimensional space, the computer program causing the computer to perform further operations including:

identifying eye attention points on the surface of the displayed object by applying an unprojection process to translate the obtained data point to a point within the three-dimensional space, and

determining the modification to the image based on the identified eye attention points.

20. The non-transitory computer readable medium of claim 19, performing further operations including:

mapping the identified eye attention points to a plurality of surface regions of the displayed object; and

determining the surface region with a highest degree of saliency based on the mapping,

wherein the modification of the image is determined based on the determined surface region.

21. The non-transitory computer readable medium of claim 20, wherein a surface of the displayed object is geometrically defined in a three-dimensional space, the computer program causing the computer to perform further operations including:

modifying the image by rotating the displayed object so that, in the modified image, the viewing direction of the displayed object faces the determined surface region.

22. The non-transitory computer readable medium of claim 19, wherein the unprojection process is applied by multiplying an unprojection matrix with coordinates of the two-dimensional gaze location, wherein the unprojection matrix is calculated based on a projection matrix used for projecting the surface of the displayed object onto the viewing area.

23. The non-transitory computer readable medium of claim 12, performing further operations including:

mapping a degree of eye attention to a plurality of regions of the viewing area based on the obtained data point; and

modifying the image when the region of the viewing area with the highest degree of eye attention is outside of the displayed object by rotating the displayed object toward the region of the viewing area having the highest degree of eye attention.

24. A system comprising:

a display device configured to display an image on a two-dimensional viewing area; and

one or more computer processors, storing executable instructions, operably connected to the display device, the instructions causing the one or more computer processors to perform operations including;

displaying an image of an object in a viewing area, wherein a surface of the displayed object is displayed three-dimensionally on the viewing area,

tracking an eye gaze at a predetermined sampling interval while the image is displayed,

obtaining a data point of the tracked eye gaze at the predetermined sampling interval, the data point representing a detected location of the tracked eye gaze within the viewing area,

identifying eye attention points on the surface of the displayed object by applying an unprojection process to translate the obtained data point to a point on the three-dimensionally displayed surface of the displayed object,

determining a modification to the image based on the identified eye attention points,

modifying the image in accordance with the modification, and

displaying the modified image.