IMAGE MANIPULATION BASED ON TRACKED EYE MOVEMENT

• US 20110006978A1
• Filed: 09/25/2009
• Published: 01/13/2011
• Est. Priority Date: 07/10/2009
• Status: Active Grant

• Current Assignee: Peking University
• Original Assignee: Peking University
• Inventors: YUAN, Xiaoru
• Granted Patent: US 8,564,533 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance

Abstract

The disclosure relates to controlling and manipulating an image of an object on a display device based on tracked eye movements of an observer. When the object is displayed according to an initial view, the observer'"'"'s eye movement is tracked and processed in order to determine the focus of the observer'"'"'s attention or gaze on the image. Thereafter, the displayed image is modified to provide a better view of the part of the object in which the observer is most interested. This is accomplished by modifying at least one of the spatial positioning of the object within the viewing area, the angle of view of the object, and the viewing direction of the object.

Claims

1. A method comprising:

causing a display device to display a first image of an object in a predefined viewing area;

obtaining data representative of a location of a tracked eye gaze with respect to the viewing area;

processing the obtained data to determine a modification of the first image with respect to at least one of;

a viewing direction of the object, an angle of view of the object, and a spatial position of the object with respect to the viewing area; and

causing the display device to display a second image of the object including the modification.

2. The method of claim 1, further comprising:

tracking the eye gaze at regular sampling intervals while the first image is displayed; and

obtaining data points of the tracked eye gaze for respective sampling intervals, at least one of the data points representing a detected location of the eye gaze within the viewing area.

3. The method of claim 2, further comprising:

analyzing the data points to identify a region of the object corresponding to a high degree of eye attention based on the tracked eye gaze; and

modifying the first image by moving the object so that, in the second image, the identified region of the object is moved to a different location of the viewing area.

4. The method of claim 3, wherein the modifying step moves the identified region of the object to the center of the viewing area.

5. The method of claim 2, further comprising:

analyzing the data points obtained during a predetermined period of time;

determining a range of the viewing area in which a predetermined concentration of the analyzed data points is located;

when the determined range is smaller than a first predetermined range, modifying the first image so that, in the second image, the object is magnified according to a zoom-in operation; and

when the determined range is larger than a second predetermined range, modifying the first image so that, in the second image, the object is shrunk according to a zoom-out operation.

6. The method of claim 5, wherein the first and second predetermined ranges are defined according to predetermined angles of view, respectively.

7. The method of claim 6, further comprising:

computing an angle of view of the entire viewing area based on the tracked eye gaze,

wherein the first and second predetermined ranges are established relative to the computed angle of view of the entire viewing area.

8. The method of claim 5, wherein the first predetermined range is defined according to first set of height and width parameters, and the second predetermined range is defined according to a second set of height and width parameters.

9. The method of claim 2, wherein a surface of the object is displayed three-dimensionally on the viewing area, the method further comprising:

identifying eye attention points on the surface of the object by applying an unprojection process to translate at least one of the obtained data points to a point on the three-dimensionally displayed surface of the object; and

determining the modification to the first image based on the identified eye attention points.

10. The method of claim 9, further comprising:

mapping the identified eye attention points to a plurality of surface regions of the object; and

determining the surface region with a highest degree of saliency based on the mapping; and

wherein the modification of the first image is determined based on the determined surface region.

11. The method of claim 10, further comprising:

modifying the first image by rotating the object so that, in the second image, the viewing direction of the object faces the determined surface region.

12. The method of claim 10, wherein the unprojection process is applied by multiplying an unprojection matrix with coordinates of the two-dimensional gaze location, wherein the unprojection matrix is calculated based on a projection matrix used for projecting the surface of the object onto the viewing area.

13. The method of claim 2, wherein a surface of the object is displayed three-dimensionally on the viewing area, the method further comprising:

mapping a degree of eye attention to a plurality of regions of the viewing area based on the obtained data points; and

when the region of the viewing area with the highest degree of eye attention is outside of the displayed object, modifying the first image by rotating the object toward the region of the viewing area having the highest degree of eye attention.

14. A computer readable medium on which is embodied a computer program which, when executed, causes a computer to perform the following:

cause a display device to display a first image of an object in a predefined viewing area;

obtain data representative of a location of a tracked eye gaze with respect to the viewing area;

process the obtained data to determine a modification of the first image with respect to at least one of a viewing direction of the object, an angle of view of the object, and a spatial position of the object with respect to the viewing area; and

cause the display device to display a second image of the object including the modification.

15. The computer readable medium of claim 14, the program causing the computer to further perform the following:

track the eye gaze at regular sampling intervals while the first image is displayed; and

obtaining data points of the tracked eye gaze for respective sampling intervals, each of the data points representing a detected location of the eye gaze within the viewing area.

16. The computer readable medium of claim 15, the computer program causing the computer to further perform the following:

analyze the data points to identify a region of the object corresponding to a high degree of the eye attention based on the tracked eye gaze; and

modify the first image by moving the object so that, in the second image, the determined region of the object is moved to the center of the viewing area.

17. The computer readable medium of claim 15, the computer program causing the computer to further perform the following:

analyze the data points obtained during a predetermined period of time;

determine a range of the viewing area in which a predetermined concentration of the analyzed data points is located;

if the determined range is within a first predetermined range, modify the first image so that, in the second image, the object is magnified according to a zoom-in operation; and

if the determined range extends beyond a second predetermined range, modify the first image so that, in the second image, the object is shrunk according to a zoom-out operation.

18. The computer readable medium of claim 17, wherein the first and second predetermined ranges are defined according to predetermined angles of view, respectively.

19. The computer readable medium of claim 18, the computer program causing the computer to further perform the following:

compute an angle of view of the entire viewing area based on the tracked eye gaze,

wherein the first and second predetermined ranges are established relative to the computed angle of view of the entire viewing area.

20. The computer readable medium of claim 17, wherein the first predetermined range is defined according to first set of height and width parameters, and the second predetermined range is defined according to a second set of height and width parameters.

21. The computer readable medium of claim 20, wherein the first and second predetermined ranges are established relative to the height and width of the viewing area.

22. The computer readable medium of claim 17, the computer program causing the computer to further perform the following:

receive inputs, via an input device, establishing the first and second predetermined ranges.

23. The computer readable medium of claim 15, wherein a surface of the object is geometrically defined in a three-dimensional space, the computer program causing the computer to further perform the following:

identify eye attention points on the surface of the object by applying an unprojection process to translate each of the obtained data points to a point within the three-dimensional space, and

determine the modification to the first image based on the identified eye attention points.

24. The computer readable medium of claim 23, the computer program causing the computer to further perform the following:

map the identified eye attention points to a plurality of surface regions of the object; and

determine the surface region with a highest degree of saliency based on the mapping,

wherein the modification of the first image is determined based on the determined surface region.

25. The computer readable medium of claim 24, wherein a surface of the object is geometrically defined in a three-dimensional space, the computer program causing the computer to further perform the following:

modify the first image by rotating the object so that, in the second image, the viewing direction of the object faces the determined surface region.

26. The computer readable medium of claim 23, wherein the unprojection process is applied by multiplying an unprojection matrix with coordinates of the two-dimensional gaze location, wherein the unprojection matrix is calculated based on a projection matrix used for projecting the surface of the object onto the viewing area.

27. The computer readable medium of claim 15, the computer program causing the computer to further perform the following:

map a degree of eye attention to a plurality of regions of the viewing area based on the obtained data points; and

if the region of the viewing area with the highest degree of eye attention is outside of the displayed object, modify the first image by rotating the object toward the region of the viewing area having the highest degree of eye attention.

28. A system comprising:

a display device configured to display an image on a two-dimensional viewing area; and

one or more computer processors operably connected to the display device, the one or more computer processors being programmed to;

cause the display device to display a first image of an object in a predefined viewing area,

obtain data representative of a location of a tracked eye gaze with respect to the viewing area,

process the obtained data to determine a modification of the first image with respect to at least one of;

a viewing direction of the object, an angle of view of the object, and a spatial position of the object with respect to the viewing area, and

cause the display device to display a second image of the object including the modification.

29. The system of claim 28, the one or more computer processors further being programmed to:

track the eye gaze at regular sampling intervals while the first image is displayed; and

obtain data points of the tracked eye gaze for respective sampling intervals, at least one of the data points representing a detected location of the eye gaze within the viewing area.

30. The system of claim 29, wherein a surface of the object is displayed three-dimensionally on the viewing area, the one or more computer processors being further programmed to:

identify eye attention points on the surface of the object by applying an unprojection process to translate each of the obtained data points to a point within the three-dimensional space, and

determine the modification to the first image based on the identified eye attention points.