Method and computer program product for determining an area of importance in an image using eye monitoring information

• US 7,046,924 B2
• Filed: 11/25/2002
• Issued: 05/16/2006
• Est. Priority Date: 11/25/2002
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Fedorovskaya, Elena A., Miller, Michael E., Cerosaletti, Cathleen D., Covannon, Edward A.
• Primary Examiner(s): PERKEY, WILLIAM B
• Application Number:
  US10/303,520
• Publication Number:
  US 20040103111A1
• Time in Patent Office: 1,268 Days
• Field of Search: 396/51, 348/231.3, 348/231.6
• US Class Current: 396/51
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 1/00326 with a data reading, recognizing or recording apparatus, e.g. with a bar-code apparatus arrangements for the associated working of recording or reproducing apparatus with related apparatus G11B31/00
• CPC Class Codes: H04N 1/00331 with an apparatus performing optical character recognition character recognition G06V30/10
• CPC Class Codes: H04N 1/2112 using still video cameras
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 2201/3204 of data relating to a user, sender, addressee, machine or electronic recording medium H04N2201/3256 and subgroups take precedence
• CPC Class Codes: H04N 2201/3225 of data relating to an image, a page or a document

Abstract

A method is provided for determining an area of importance in an archival image. In accordance with this method, eye information including eye gaze direction information captured during an image capture sequence for the archival image is obtained. An area of importance in the archival image is determined based upon the eye information. Area of importance data characterizing the area of importance is associated with the archival image.

Claims

1. A method for determining an area of importance in captured archival image, the method comprising the steps of successively:

capturing a plurality of eye images, in whole or part, when viewing a scene image to be photographed;

capturing the scene image during an image capture sequence, to obtain the captured archival image;

determining eye information from the plurality of eye images, including eye gaze direction information which describes eye gaze movement during the image capture sequence;

determining an area of importance in the captured archival image based upon the eye information;

generating area importance data; and

associating the area importance data with the archival image.

2. The method of claim 1, wherein the step of determining an area of importance in the captured archival image based upon the eye information comprises determining an eye gaze direction path based upon said eye gaze direction information and identifying an area of importance based upon the determined eye gaze direction path.

3. The method of claim 1, wherein the area of importance is determined by identifying a concentration of the eye gaze direction information within a region of the archival image.

4. The method of claim 1, wherein the step of determining an area of importance in the captured archival image comprises determining an eye gaze direction path based upon the obtained eye gaze direction information and identifying a predetermined pattern of the eye gaze direction information.

5. The method of claim 1, wherein the eye information further includes pupil size information captured during the image capture sequence and the step of determining an area of importance in the captured archival image based upon the eye information comprises determining the area of importance based upon the eye gaze direction information and the pupil size information.

6. The method of claim 1, wherein the step of determining an area of importance in the captured archival image based upon the eye information comprises determining the area of importance based upon the eye information and analysis of the archival image.

7. The method of claim 1, wherein the step of determining the area of importance in the captured archival image based upon the eye information comprises detecting a sequence of fixations in the eye gaze direction information and determining the area of importance based upon the sequence of fixations.

8. The method of claim 7, wherein the step of determining the area of importance in the captured archival image based upon the eye information comprises detecting a sequence of fixation in the eye gaze direction information, clustering the sequence of fixations, assigning a weight to each cluster, and selecting an area of importance based upon the weights of the clusters.

9. The method of claim 8, wherein the step of determining the weight of each cluster comprises determining the weight of each cluster based upon the order of the fixations in each cluster.

10. The method of claim 8, wherein the step of determining the weight of each cluster further comprises determining the weight of each cluster based upon the order of fixations in each cluster and the duration of the fixations within each cluster.

11. The method of claim 8, wherein the step of determining the weight of each cluster comprises determining the weight of each cluster based upon the duration of the fixations within each cluster.

12. The method of claim 8, wherein the step of determining the weight of each cluster comprises determining the weight of each cluster based at least in part upon a pattern of the eye gaze direction information between clusters.

13. The method of claim 1, wherein the step of determining the area of importance comprises identifying at least one of a pattern of positive importance and a pattern of negative importance in eye gaze direction information.

14. A method for determining an area of importance in captured archival image, the method comprising the steps of:

capturing a plurality of eye images, in whole or part, when viewing a scene image to be photographed;

capturing the scene image during an image capture sequence, to obtain the captured archival image;

determining eye information, including eye gaze direction information which describe eye gaze movement during the image capture sequence and is determined from the plurality of eye images, and context information, including scene composition change information which describes scene composition adjustments during the image capture sequence and is used to interpret the eye images;

determining an eye gaze direction path relative to the captured archival image based upon the eye gaze direction information and scene composition change information;

determining an area of importance in the captured archival image based upon the eye gaze direction path relative to the captured archival image;

generating area importance data; and

associating the area importance data with the captured archival image.

15. The method of claim 14, wherein the step of determining an area of importance in the captured archival image based upon the eye gaze direction path relative to the captured archival image comprises determining the area of importance in the captured archival image based upon eye gaze direction information that is determined to reflect eye gaze information within the captured archival image.

16. The method of claim 14, wherein the step of determining the eye gaze direction path relative to the captured archival image comprises determining the eye gaze direction path by adjusting the eye gaze direction information based upon the scene composition change information so that the eye gaze direction path is recorded relative to the scene image captured to obtain the captured archival image.

17. The method of claim 14, wherein the context information comprises a composite image containing image information composited from a set of evaluation images captured during composition of the captured archival image and wherein said eye gaze direction information is recorded relative to the composite image, the method further comprising the step of registering the position of the archival image with respect to the composite image wherein the step of determining the path of eye gaze direction relative to the archival image is based upon the registered position of the archival image with respect to the composite image.

18. The method of claim 14, wherein the context information comprises adjustable zoom setting information and wherein the step of determining a path of eye gaze direction relative to the archival image comprises determining the path of eye gaze information based upon the captured eye gaze direction, the scene composition change information and the adjustable zoom setting information.

19. The method of claim 14, wherein the context information characterizes changes in a field of view of the camera and the step of determining a path of eye gaze direction relative to the archival image comprises adjusting the path of eye gaze direction during a composition phase of the archival image capture process to compensate for changes in the field of view of the camera.

20. The method of claim 14, wherein the context information comprises a set of evaluation images captured during the image capture sequence, and wherein the step of determining a path of eye gaze direction relative to the archival image comprises analyzing the set of evaluation images and determining the path of eye gaze direction relative to the archival image based upon the eye gaze direction information and the analysis of the set of evaluation images.

21. The method of claim 14, wherein the eye gaze information comprises eye information captured from more than one eye wherein the path of eye gaze direction is determined relative to the archival image based upon the captured eye gaze direction information from each eye and scene composition change information.

22. The method of claim 14, wherein the eye information and context information are recorded on a photosensitive medium further comprising the step of extracting the eye information and context information from the photosensitive medium and associating the eye information and context information with an archival image recorded on the photosensitive medium.

23. The method of claim 14, wherein the eye information includes at least one image of an eye of a user, further comprising the step of determining the identity of the user of the camera system based upon the eye images and to associate the user identity with the scene image.

24. The method of claim 14, wherein the context information includes location information and wherein the steps of determining an area of importance in the archival image comprises determining the area of importance based upon the path of eye gaze direction relative to the archival image and the location information.

25. A computer program product for determining an area of importance in captured archival image, the computer program product comprising a computer readable storage medium having a computer program stored thereon for performing the steps of successively:

capturing a plurality of eye images, in whole or part, when viewing a scene image to be photographed;

capturing the scene image during an image capture sequence, to obtain the captured archival image;

determining eye information from the plurality of eye images, including eye gaze direction information which describes eye gaze movement during the image capture sequence;

determining an area of importance in the captured archival image based upon the eye information;

generating area importance data; and

associating the area importance data with the archival image.

26. The computer program product of claim 25, wherein the step of determining an area of importance in the archival image based upon the eye information comprises determining an eye gaze direction path based upon said eye gaze direction information and identifying an area of importance based upon the determined eye gaze direction path.

27. The computer program product of claim 25, wherein the area of importance is determined by identifying a concentration of the eye gaze direction information within a region of the archival image.

28. The computer program product of claim 25, wherein the step of determining an area of importance in the archival image comprises determining an eye gaze direction path based upon the obtained eye gaze direction information wherein the area of importance is determined by identifying a predetermined pattern of the eye gaze direction path.

29. The computer program product of claim 25, wherein the eye information further includes pupil size information captured during the image capture sequence and the step of determining an area of importance in the archival image based upon the eye gaze direction path comprises determining the area of importance based upon the eye gaze direction information and the pupil size information.

30. The computer program product of claim 25, further comprising the step of analyzing the archival image wherein the step of determining an area of importance in the archival image based upon the eye information comprises determining the area of importance based upon the eye information and the analysis of the archival image.

31. The computer program product of claim 25, wherein the step of determining the area of importance in the archival image based upon the eye information comprises detecting a sequence of fixations in the eye information and determining the area of importance based upon the sequence of fixations.

32. The computer program product of claim 25, wherein the step of determining the area of importance in the archival image based upon the eye information comprises detecting a sequence of fixations in the eye gaze direction information, clustering the sequence of fixations, assigning a weight to each cluster, and selecting an area of importance based upon the weights of the clusters.

33. The computer program product of claim 32, wherein the step of determining the weight of each cluster comprises determining the weight of each cluster based upon the order of the fixations in each cluster.

34. The computer program product of claim 32, wherein the step of determining the weight of each cluster further comprises determining the weight of each cluster based upon the order of fixations in each cluster and the duration of the fixations within each cluster.

35. The computer program product of claim 32, wherein the step of determining the weight of each cluster comprises determining the weight of each cluster based upon the duration of the fixations within each cluster.

36. The computer program product of claim 32, wherein the step of determining the weight of each cluster comprises determining the weight of each cluster based at least in part upon a pattern of the eye gaze information between clusters.

37. The computer program product of claim 25, wherein the step of determining the area of importance in the archival image based upon the eye information comprises identifying at least one of a pattern of positive importance and a pattern of negative importance in eye gaze direction information.

38. A computer program product for determining an area of importance in captured archival image the computer program product comprising a computer readable storage medium having a computer program stored thereon for performing the steps of:

capturing a plurality of eye images, in whole or part, when viewing a scene image to be photographed;

capturing the scene image during an image capture sequence, to obtain the captured archival image;

determining eye information, including eye gaze direction information which describe eye gaze movement during the image capture sequence and is determined from the plurality of eye images, and context information, including scene composition change information which describes scene composition adjustments during the image capture sequence and is used to interpret the eye images;

determining an eye gaze direction path relative to the captured archival image based upon the eye gaze direction information and scene composition change information;

determining an area of importance in the captured archival image based upon the eye gaze direction path;

generating area importance data; and

associating the area importance data with the captured archival image.

39. The computer program product of claim 38, wherein the step of determining an area of importance in the archival image based upon the eye gaze direction path comprises determining the area of importance in the archival image based only upon eye gaze direction information that is determined to reflect eye gaze information within the archival image.

40. The computer program product of claim 38, wherein the step of determining a path of eye gaze direction relative to the archival image comprises determining the path of eye gaze direction by adjusting the eye gaze direction information based upon the scene composition change information so that the determined path of eye gaze direction is recorded relative to the scene captured in the archival image.

41. The computer program product of claim 38, wherein the context information comprises a composite image containing image information composited from a set of evaluation images captured during composition of the archival image and wherein said eye gaze direction information is recorded relative to the composite image;

the computer program product further comprising the step of registering the position of the archival image with respect to the composite image and determining the path of eye gaze direction relative to the archival image based upon the registered position of the archival image with respect to the composite image.

42. The computer program product of claim 38, wherein the context information comprises adjustable zoom setting information and wherein step of determining a path of eye gaze direction relative to the archival image comprises determining the path of eye gaze information based upon the captured eye gaze direction information, the scene composition change information and the adjustable zoom setting information.

43. The computer program product of claim 38, wherein the context information characterizes changes in a field of view and the step of determining a path of eye gaze direction relative to the archival image comprises adjusting the path of eye gaze direction during a composition phase of the archival image capture process to compensate for changes in the field of view of the camera.

44. The computer program product of claim 38, wherein the context information comprises a set of evaluation images captured during the image capture sequence, and wherein the step of determining a path of eye gaze direction relative to the archival image comprises analyzing the set of evaluation images and determining the path of eye gaze direction relative to the archival image based upon the eye gaze direction information and the analysis of the set of evaluation images.

45. The computer program product of claim 38, wherein the eye information comprises eye information captured from more than one eye wherein the path of eye gaze direction is determined relative to the archival image based upon the captured eye gaze direction information from each eye and scene composition change information.

46. The computer program product of claim 38, wherein the eye information and context information are recorded on a photosensitive medium further comprising the step of extracting the eye information and context information from the photosensitive medium and associating the eye information and context information with an archival image recorded on the photosensitive medium.

47. The computer program product of claim 38, wherein the eye information includes at least one image of an eye of a user, further comprising the step of determining the identity of the user of the camera system based upon the eye images and associating the user identity with the scene image.

48. The computer program product of claim 38, wherein the context information includes location information and wherein the steps of determining an area of importance is based upon the path of eye gaze direction relative to the archival image and the location information.