Camera system with eye monitoring

• US 7,206,022 B2
• Filed: 11/25/2002
• Issued: 04/17/2007
• Est. Priority Date: 11/25/2002
• Status: Expired due to Fees

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Fedorovskaya, Elena A., Miller, Michael E., Cerosaletti, Cathleen D., Covannon, Edward
• Primary Examiner(s): Vu; Ngoc-Yen
• Assistant Examiner(s): Bemben; Richard M.
• Application Number:
  US10/303,978
• Publication Number:
  US 20040100567A1
• Time in Patent Office: 1,604 Days
• Field of Search: 348/78, 348/46, 348/107, 348/333.03
• US Class Current: 348/333.03
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: H04N 1/00127 Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
• CPC Class Codes: H04N 1/2112 using still video cameras
• CPC Class Codes: H04N 1/32128 attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image for information embedded in the image H04N1/32144
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 2201/3204 of data relating to a user, sender, addressee, machine or electronic recording medium H04N2201/3256 and subgroups take precedence
• CPC Class Codes: H04N 2201/3225 of data relating to an image, a page or a document
• CPC Class Codes: H04N 2201/3274 Storage or retrieval of prestored additional information
• CPC Class Codes: H04N 23/50 Constructional details
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera

Abstract

In one aspect of the invention a camera system is provided having an image capture system adapted to capture an image of a scene during an image capture sequence and an eye monitoring system adapted to determine eye information including a direction of the gaze of an eye of a user of the camera system. A controller is adapted to store the determined eye information including information characterizing eye gaze direction during the image capture sequence and to associate the stored eye information with the scene image.

Claims

1. A camera system comprising:

an image capture system for capturing an archival image of a scene during an archival image capture sequence including a composition phase before an image capture phase during which the archival image is captured;

an eye monitoring system adapted to determine eye information from including a direction of the gaze of an eye of a user;

a source of context information; and

a controller adapted to use the eye monitoring system to determine eye information from which a path of eye gaze direction can be determined that represents a path of eye gaze direction during a range of different times during the archival image capture sequence; and

to store the eye information and context information during the archival image capture sequence and to associate the stored information with the archival image captured during the archival image capture sequence;

wherein the controller causes the image capture system to capture a set of evaluation images during the composition phase of the archival image capture sequence and the source of context information comprises an image processor adapted to determine image information characterizing changes in the composition of the scene during the composition phase based on image analysis of the set of evaluation images and wherein the controller causes the image capture system to capture a set of evaluation images during the archival image capture sequence; and

,

wherein said image processor adapted to determine information characterizing changes in the field of view of the archival image capture system by assembling a composite image mapping all portions of the scene imaged by any of the set of captured evaluation images and said processor records eye gaze information in relation to the assembled composite.

2. The camera system of claim 1, wherein the eye information determined by the eye monitoring system further includes information characterizing changes in the diameter of the pupil during the image capture sequence.

3. The camera system of claim 1, wherein the eye information determined by the eye monitoring system further includes information characterizing eye temperature during the image capture sequence.

4. The camera system of claim 1, wherein the controller determines eye information further includes data from which the amount of dwell time at each location, or the acceleration of eye gaze position during movement can be determined.

5. The camera system of claim 1, wherein the eye information determined by the eye monitoring system further includes eye information from both eyes of the user.

6. The camera system of claim 5, wherein the controller is further adapted to determine focus distance information during the image capture sequence based upon parallax differences between the eye gaze direction of both eyes of observer.

7. The camera system of claim 5, wherein the controller is further adapted to compare eye gaze direction information determined from monitoring of both eyes of the observer for validation.

8. The camera system of claim 1, further comprising an input transducer adapted to generate signals in response to manual input from a user of the camera system and wherein said controller is adapted to store input transducer information during the image capture sequence and to associate the input transducer information with the scene image.

9. The camera system of claim 1, wherein the image capture system is a digital image capture system operable to form evaluation images during image composition and further comprising a display to present the evaluation images.

10. The camera system of claim 1, further comprising an additional image capture system adapted to store scene images on a photosensitive medium and having a write head to optically record the stored eye information on the photosensitive medium.

11. The camera system of claim 1, wherein the controller stores the eye information apart from the scene image and the controller associates the scene image with the eye information by recording data with the scene image which can be used to locate the eye information.

12. The camera system of claim 1, wherein the source of context information comprises a location detector adapted to determine where the archival image was captured.

13. The camera system of claim 1, wherein the controller causes the image capture system to capture a set of evaluation images during a composition phase of the archival image capture sequence and the source of context information comprises an image processor adapted to determine image information characterizing changes in the composition of the scene during the composition phase based on image analysis of the set of evaluation images.

14. The camera system of claim 1, wherein the controller causes the image capture system to capture a set of evaluation images during a composition phase of the archival image capture sequence and the source of context information comprises an image processor adapted to determine image information characterizing changes in the composition of the scene during the composition phase based on image analysis of the set of evaluation images to locate fiducials in the image.

15. The camera system of claim 1, wherein said controller comprises said source of context information and said controller is adapted to cause the image capture system to capture evaluation images during the image capture sequence and to store the captured evaluation images in association with the captured archival image with said evaluation images being captured at different times than said archival image.

16. The camera system of claim 1, wherein the source of context information comprises a manual input and the controller monitors the manual input during the image capture sequence and associates information characterizing changes in the status of the manual input during the image capture process with the captured archival image.

17. The camera system of claim 16, wherein the image capture system comprises a taking lens unit having an adjustable zoom setting and the manual input comprises a zoom setting input.

18. The camera system of claim 16, wherein the image capture system comprises a taking lens unit having an adjustable zoom setting and the manual input comprises a zoom setting input with the controller adjusting the eye gaze information based upon the zoom setting input.

19. The camera system of claim 1, wherein the source of context data comprises a scene brightness detector.

20. The camera system of claim 1, wherein the image capture system captures the archival image in the form of a digital archival image signal and the controller stores the electronic archival image signal in association with the eye movement data signal and the context information in digital form.

21. The camera system of claim 1, further comprising an archival image capture system that captures the archival image on a photosensitive element and further comprising a recording head for optically recording the eye information and context information on the photosensitive element.

22. The camera system of claim 1, further comprising an archival capture system that captures images on a photosensitive element and the archival image capture system comprises a film data recorder for recording the eye information and context information onto the film.

23. The camera system of claim 1, wherein the eye monitoring system comprises an optical system that presents light from the scene to form an image of the scene at an eye of an observer and an eye camera that captures images of the eye during the image capture sequence.

24. The camera system of claim 23, wherein a beam splitter is disposed in the optical path between the scene and the eye of the observer, with the beam splitter adapted to deflect an image of the eye to the eye camera.

25. The camera system of claim 23, wherein the eye monitoring system further comprises an image processor to examine the eye images and to determine eye gaze direction.

26. The camera of claim 23, wherein the eye monitoring system comprises an eye camera that further detects pupil dilation and wherein the controller is further adapted to incorporate pupil dilation information in the eye information.

27. The camera system of claim 1, wherein said processor is further adapted to perform an evaluation image capture sequence during composition and wherein an evaluation image display is adapted to present captured evaluation images to a display space for observation by a user and wherein the user can observe the scene either directly or through the display, an eye imager capturing images of the display space and wherein said controller is further adapted to receive images from the eye imager and to map the eye gaze direction of an eye of the user relative to the scene both when the eye in the display space is directed at the scene directly or at the scene as displayed on the display in order to determine eye information characterizing eye gaze direction of the eye.

28. The camera system of claim 27, further comprising a beam splitter interposed between the display and the eye with the beam splitter reflecting light from the eye onto the eye imager.

29. The camera system of claim 1, wherein the camera system is incorporated into a set of glasses.

30. A method for operating a camera system comprising the steps of:

capturing an archival image of a scene during an archival image capture sequence including a composition phase before an image capture phase during which the archival image is captured;

obtaining context information;

determining eye information including a direction of the gaze of an eye of a user from which a path of eye gaze direction can be determined that represents a path of eye gaze direction during a range of different times during the archival image capture sequence;

storing the eye information and context information during the archival image capture sequence; and

associating the stored eye information and context information with the archival image captured during the archival image capture sequence;

wherein a set of evaluation images is captured during the composition phase of the archival image capture sequence and the step of obtaining context information comprises determining image information characterizing changes in the composition of the scene during the composition phase based on image analysis of the set of evaluation images and,

wherein the image information characterizing changes in the field of view of the archival image capture system is determined by assembling a composite image mapping all portions of the scene imaged by any of the set of captured evaluation images and the eye gaze information is mapped in relation to the assembled composite.

31. The method of claim 30, further comprising the step of storing the eye gaze information with the archival image in a memory outside of the camera.

32. The method of claim 30, wherein the step of monitoring eye information includes monitoring a diameter of a pupil of the eye, and

wherein the step of determining eye information includes determining a pattern of changes in pupil diameter during the composition phase and the image capture phase.

33. The method of claim 30, further comprising the steps of forming an evaluation image corresponding to the captured archival image and presenting the evaluation image to the user of the camera system;

monitoring eye information including eye gaze information during presentation of the evaluation image; and

determining further eye information representing a pattern of changes in eye gaze direction during presentation of the evaluation image over a time period after capture of the archival image and wherein the evaluation image is presented.

34. The method of claim 30, wherein the camera system has a user interface adapted to receive user inputs and the step of determining context information comprises monitoring changes in the user inputs.

35. A method for operating a camera system comprising the steps of:

means for capturing an archival image of a scene during an archival image capture sequence including a composition phase before an image capture phase during which the archival image is captured;

means for using an eye monitoring system to determine eye information including a direction of the gaze of an eye of a user from which a path of eye gaze direction can be determined that represents a path of eye gaze direction during a range of different times during the archival image capture sequence;

means for obtaining context information;

means for storing the eye information and context information during the archival image capture sequence; and

means for associating the stored eye information and context information with the archival image captured during the archival image capture sequence;

wherein a set of evaluation images is captured during the composition phase of the archival image capture sequence and the step of obtaining context information comprises determining image information characterizing changes in the composition of the scene during the composition phase based on image analysis of the set of evaluation images and,

wherein the image information characterizing changes in the field of view of the archival image capture system is determined by assembling a composite image mapping all portions of the scene imaged by any of the set of captured evaluation images and the eye gaze information is mapped in relation to the assembled composite.