Gaze tracking system

• US 20040075645A1
• Filed: 10/09/2003
• Published: 04/22/2004
• Est. Priority Date: 10/09/2002
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Taylor, Richard Ian, Rowe, Simon Michael
• Granted Patent: US 7,158,097 B2
• Time in Patent Office: Days
• US Class Current: 345/157
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

A gaze tracking system (1,3,8) is provided which obtains images of an operator (9) and processes the images to associate the images with coordinates of where the operators gaze is directed towards.

In order to identify images where an operator is looking in the same direction, images obtained by a camera (8) are first processed using the retinex algorithm. Image patches from the processed image are then compared with stored feature images and an initial classification based upon the correspondence between areas of the processed image and the stored feature images is determined. This initial classification is then further processed and each image is assigned a single classification. The areas identified by different classifications are then determined so that classifications assigned to adjacent images identify points closer together and classifications never assigned to an adjacent images identify points further apart.

Claims

1. Apparatus for associating gaze data representative of the location of gaze of an operator with classifications of images of an operator looking at different locations, comprising:

a receiver operable to receive a video stream defining a sequence of images representative of an operator at different points in time;

a classification unit operable to assign one of a number of classifications to images in the video stream received by said receiver, said classification unit being operable to assign the same classifications to images of an operator looking at the same locations; and

a calibration unit comprising;

a classification store configured to store data identifying the pairs of different classifications assigned to pairs of images in a video stream received by said receiver representative of an operator at different times separated by less than a preset time period;

a gaze conversion store configured to store data associating each of said number of classifications with gaze data representative of a gaze location; and

an update unit operable to update gaze data stored in said gaze conversion store by updating said gaze data such that gaze data for pairs of different classifications identified by data stored in said classification store identify gaze locations which are closer together and gaze data for pairs of different classifications not identified by data stored in said classification store identify gaze locations which are further apart.

2. Apparatus in accordance with claim 1 wherein said calibration unit is responsive to the assignment of different classifications to images representative of an operator at different times separated by less than said pre set time period by said classification unit to determine whether data stored in said classification store is indicative of said pair of classifications and to update data stored in said classification store in dependence upon said determination.

3. Apparatus in accordance with claim 2 wherein said classification unit is operable to determine whether classifications assigned to consecutive images in a video stream received by said receiver are assigned to different classifications and to update data identifying pairs of classifications stored in said classification store if the different classifications for a pair of consecutive images are not identified by data in said classification store.

4. Apparatus in accordance with any claim 1 wherein said update unit is operable to determine the distance between locations identified by gaze data associated with pairs of classifications, said update unit being operable to update gaze data on the basis of said determined distances and said data stored in said classification store.

5. Apparatus in accordance with claim 4 wherein said update unit is operable to adjust gaze data associated with classifications identified by data in said classification store when the determined distance between the gaze locations identified by gaze data for said pair of classifications is greater than a predetermined threshold.

6. Apparatus in accordance with claim 4 wherein said update unit is operable to adjust gaze data associated with classifications not identified by data in said classification store when the determined distance between the gaze locations identified by gaze data for said pair of classifications is less than a predetermined threshold.

7. Apparatus in accordance with claim 1 wherein said gaze data comprises a co-ordinate data, said update unit being operable to update gaze data for classifications by adjusting said co-ordinate data by an amount determined utilising the difference between the coordinates associated with selected pairs of classifications.

8. Apparatus in accordance with claim 7 wherein said update unit us operable to adjust gaze data associated with a pair of classifications identified by data in said classification store when the distance between the locations identified by said gaze data is more than a predetermined threshold by adjusting the gaze data for at least one of said pair of classifications utilising the following equation:

9. Apparatus in accordance with claim 7 wherein said update unit is operable to adjust gaze data associated with a pair of classifications not identified by data in said classification store, when the distance between the locations identified by said gaze data is less than a predetermined threshold by adjusting the gaze data for at least one of said pair of classifications utilising the following equation:

10. Apparatus in accordance with claim 1 further comprising:

a display operable to display a user interface including a pointer;

a control device operable to control the location of said pointer on said display, said control device being operable to enable a user to select portions of said display utilising said pointer; and

a monitoring unit operable to monitor the selection of portions of said display utilising said control device, and to update gaze data associated with the classification of an image an operator obtained when a selection of a portion of the display is made using the control device utilising data identifying the location of the portion of the display selected.

11. Apparatus in accordance with the claim 10 wherein said control device comprises a mouse, said selection of a portion of the display being initiated by clicking a button on said mouse.

12. Apparatus in accordance with claim 1 further comprising:

a display operable to display a plurality of user interfaces identifying particular portions of said display,

wherein said calibration unit is operable to associate classifications of images obtained when a user interface identifying a particular portion of the display is shown with gaze data identifying the location of the portion of said display identified by said user interface.

13. Apparatus in accordance with claim 1 further comprising a data store operable to store data identifying the frequency with which images are associated with each of said classifications, said calibration unit being operable to update gaze data stored in said second data store by applying a selected transformation to said gaze data on the basis of said data stored in said data store.

14. Apparatus in accordance with claim 12 wherein said selected transformation comprises a transformation defining a reflection, a rotation, a translation or a combination of a reflection, rotation and a translation.

15. Apparatus in accordance with claim 1 wherein said classification unit comprises a self-organising map.

16. Apparatus in accordance with claim 1 further comprising an output unit operable to output for a video stream received by said receiver a sequence of gaze data comprising gaze data associated by said gaze conversion store with the classifications assigned to each image in said video stream by said classification unit.

17. A method associating gaze data representative of the location of gaze of an operator with classifications of images of an operator looking at different locations, comprising:

receiving a video stream defining a sequence of images representative of an operator at different points in time;

assigning one of a number of classifications to images in the received video stream, wherein the same classifications are assigned to images of an operator looking at the same locations;

storing data identifying the pairs of different classifications assigned to pairs of images in a received video stream representative of an operator at different times separated by less than a preset time period;

storing data associating each of said number of classifications with gaze data representative of a gaze location; and

updating stored gaze data by updating said gaze data such that gaze data for pairs of different classifications identified by stored data said gaze data identifies gaze locations which are closer together and gaze data for pairs of different classifications not identified by stored data are updated to identify gaze locations which are further apart.

18. A method in accordance with claim 17 further comprising responding to the assignment of different classifications to images representative of an operator at different times separated by less than said pre set time period by determining whether stored data identifies said pair of classifications and updating said stored data in dependence upon said determination.

19. A method in accordance with claim 18 wherein said determination comprises determining whether classifications assigned to consecutive images in a received video stream are assigned to different classifications and updating said stored data comprises storing data identifying the pair of different classifications if the different classifications for a pair of consecutive images are not already identified by stored data.

20. A method in accordance with claim 17 wherein said updating stored gazed data comprises determining the distance between locations identified by gaze data associated with pairs of classifications, and updating gaze data on the basis of said determined distances and said stored data.

21. A method in accordance with claim 20 wherein said updating stored gaze data comprises adjusting gaze data associated with classifications identified by stored data when the determined distance between the gaze locations identified by gaze data for said pair of classifications is greater than a predetermined threshold.

22. A method in accordance with claim 20 wherein said updating stored gaze data comprises adjusting gaze data associated with classifications not identified by stored data when the determined distance between the gaze locations identified by gaze data for said pair of classifications is less than a predetermined threshold.

23. A method in accordance with claim 17 wherein said gaze data comprises a co-ordinate data and said updating stored gaze data comprises updating gaze data for classifications by adjusting said co-ordinate data by an amount determined utilising the difference between the co-ordinates associated with selected pairs of classifications.

24. A method in accordance with claim 23 wherein said updating stored gaze data comprises adjusting gaze data associated with a pair of classifications identified by stored data when the distance between the locations identified by said gaze data is more than a predetermined threshold by adjusting the gaze data for at least one of said pair of classifications utilising the following equation:

25. A method in accordance with claim 23 wherein said updating stored gaze data comprises adjusting gaze data associated with a pair of classifications not identified by stored data when the distance between the locations identified by said gaze data is less than a predetermined threshold by adjusting the gaze data for at least one of said pair of classifications utilising the following equation:

26. A method in accordance with claim 17 further comprising the steps of:

displaying a user interface including a pointer;

a monitoring user input to identify the selection of portions of said user interface utilising said pointer under the control of a control device, and updating stored gaze data associated with the classification of an image an operator obtained when a selection of a portion of the user interface is determined to have been made utilising data identifying the location of the portion of the user interface selected with said pointer.

27. A method in accordance with the claim 26 wherein said control device comprises a mouse, said selection of a portion of the display being initiated by clicking a button on said mouse.

28. A method in accordance with claim 17 further comprising the steps of:

displaying a plurality of user interfaces identifying particular portions of a display; and

associating classifications of images obtained when a user interface identifying a particular portion of the display is shown with gaze data identifying the location of the portion of said display identified by said user interface.

29. A method in accordance with claim 17 further comprising the steps of:

storing data identifying the frequency with which images are associated with each of said classifications; and

updating gaze data by applying a selected transformation to said gaze data on the basis of said data identifying the frequency with which images are associated with each of said classifications.

30. A method in accordance with claim 29 wherein said selected transformation comprises a transformation defining a reflection, a rotation, a translation or a combination of a reflection, rotation and a translation.

31. A method in accordance with claim 17 further comprising the steps of:

outputting for a received video stream a sequence of gaze data comprising gaze data associated with the classifications assigned to each image in said video stream.

32. A data carrier storing computer implementable process steps for causing a programmable computer to perform a method in accordance with claim 17.

33. A data carrier in accordance with claim 32 comprising a computer disc.

34. A data carrier in accordance with claim 32 comprising an electric signal transferred via a network.

35. A computer disc in accordance with claim 33 wherein said computer disc comprises an optical, magneto-optical or magnetic disc.

36. Apparatus for classifying images comprising:

a receiver operable to receive image data defining images;

a patch generation module operable to generate for a number of points in an image defined by image data received by said receiver, image patches, said image patches being derived by processing image data defining portions of said image which include said points, a plurality of different image patches being derived for each of said points;

a patch comparison module operable to compare stored data with image patches generated by said patch generation module to identify stored data which most closely corresponds to said generated patches; and

a classification unit for generating a classification of an image defined by image data received by said receiver, wherein said classification comprises data identifying points in an image and for each of said points, data identifying stored data determined to most closely correspond to an image patch derived by said patch generation module from image data defining a portion of the image including said point.

37. A method of classifying images comprising:

receiving image data defining images;

generating for a number of points in an image defined by received image data, image patches, said image patches being derived by processing image data defining portions of said image which include said points, a plurality of different image patches being derived for each of said points;

comparing stored data with said generated image patches to identify stored data which most closely corresponds to said generated patches; and

generating a classification of an image defined by received image data, wherein said classification comprises data identifying points in an image and for each of said points, data identifying stored data determined to most closely correspond to an image patch generated from image data defining a portion of the image including said point.

38. Apparatus for comparing colour images independently of the illumination of a subject object in said images, comprising:

a receiver operable to receive image data defining images, said image data comprising colour data for a plurality of pixels representative of the colours of a subject object appearing in said images;

a processing unit operable to derive from image data received by said receiver, a colour reflectance image said colour reflectance image comprising colour data for said plurality of pixels representative of the contribution to the colour of a subject object in an image not arising due to the colour of the illumination of the subject object in said image, said processing unit being operable to derive said colour reflectance image such that the ratios of colour data of pixels in said colour reflectance image for a said subject object are independent of the colour of the illumination of said subject object in said image; and

a comparator operable to compare generated colour reflectance images with stored image data to determine the correspondence between said stored data and said generated images;

wherein said correspondence is determined on the basis of the comparison of the ratios of colour data for pixels in a said generated colour reflectance image and corresponding ratios of colour data for pixels of said stored image data.

39. A method of comparing colour images independently of the illumination of a subject object in said images, comprising:

receiving image data defining images, said image data comprising colour data for a plurality of pixels representative of the colours of a subject object appearing in said images;

processing a received image to derive from received image data a colour reflectance image said colour reflectance image comprising colour data for said plurality of pixels representative of the contribution to the colour of a subject object in an image not arising due to the colour of the illumination of the subject object in said image, said processing being such to derive said colour reflectance image such that the ratios of colour data of pixels in said colour reflectance image for a said subject object are independent of the colour of the illumination of said subject object in said image; and

comparing generated colour reflectance images with stored image data to determine the correspondence between said stored data and said generated images;

wherein said correspondence is determined on the basis of the comparison of the ratios of colour data for pixels in a said generated colour reflectance image and corresponding ratios of colour data for pixels of said stored image data.

40. Apparatus for associating data indicative of the orientation of a person'"'"'s head with classifications of images of a person'"'"'s head comprising:

a receiver operable to receive a video stream defining a sequence of images of a person'"'"'s head;

a classification unit operable to assign one of a number of classifications to images in the video stream received by said receiver, said classification unit being operable to assign the same classifications to images of a person'"'"'s head in substantially the same orientations; and

a calibration unit comprising;

a classification store configured to store data identifying pairs of different classifications assigned to pairs of images in a video stream received by said receiver representative of a person'"'"'s head at different times separated by less than a reference time period;

a conversion store configured to store data associating each of said number of classifications with data representative of at least one head orientation; and

an update unit operable to update data stored in said conversion store by updating said data such that data for pairs of different classifications identified by data stored in said classification store identify head orientations which are closer together and data for pairs of different classifications not identified by data stored in said classification store identify head orientations which are further apart.

41. A method of associating data indicative of the orientation of a person'"'"'s head with classifications of images of a person'"'"'s head comprising:

receiving a video stream defining a sequence of images of a person'"'"'s head;

assigning one of a number of classifications to images in the received video stream, wherein the same classifications are assigned to images of a person'"'"'s head in substantially similar orientations;

storing pair data identifying pairs of different classifications assigned to pairs of images in a received video stream representative of a person'"'"'s head at different times separated by less than a reference time period;

storing association data associating each of said number of classifications with data representative of at least one head orientation; and

updating said association data such that association data for pairs of different classifications identified by stored pair data identify head orientations which are closer together and association data for pairs of different classifications not identified by stored pair data identify head orientations which are further apart.

42. Apparatus for associating data indicative of the viewing direction of a person'"'"'s eyes with classifications of images of a person'"'"'s eyes comprising:

a receiver operable to receive a video stream defining a sequence of images of a person'"'"'s eyes;

a classification unit operable to assign one of a number of classifications to images in the video stream received by said receiver, said classification unit being operable to assign the same classifications to images of a person'"'"'s eyes in substantially the same viewing directions; and

a calibration unit comprising;

a classification store configured to store data identifying pairs of different classifications assigned to pairs of images in a video stream received by said receiver representative of a person'"'"'s eyes at different times separated by less than a reference time period;

a conversion store configured to store data associating each of said number of classifications with data representative of at least one viewing direction; and

an update unit operable to update data stored in said conversion store by updating said data such that data for pairs of different classifications identified by data stored in said classification store identify viewing directions which are closer together and data for pairs of different classifications not identified by data stored in said classification store identify viewing directions which are further apart.

43. A method of associating data indicative of the viewing directions of a person'"'"'s eyes with classifications of images of a person'"'"'s eyes comprising:

receiving a video stream defining a sequence of images of a person'"'"'s eyes;

assigning one of a number of classifications to images in the received video stream, wherein the same classifications are assigned to images of a person'"'"'s eyes in substantially similar viewing directions;

storing pair data identifying pairs of different classifications assigned to pairs of images in a received video stream representative of a person'"'"'s eyes at different times separated by less than a reference time period;

storing association data associating each of said number of classifications with data representative of at least one viewing direction; and

updating said association data such that association data for pairs of different classifications identified by stored pair data identify viewing directions which are closer together and association data for pairs of different classifications not identified by stored pair data identify viewing directions which are further apart.