Method and apparatus for determining the position of eyes and for correcting eye-defects in a captured frame

• US 6,134,339 A
• Filed: 09/17/1998
• Issued: 10/17/2000
• Est. Priority Date: 09/17/1998
• Status: Expired due to Term

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Luo, Jiebo
• Primary Examiner(s): Au, Amelia
• Assistant Examiner(s): Bali, Vikkram
• Application Number:
  US09/154,684
• Time in Patent Office: 761 Days
• Field of Search: 382/115, 382/117, 382/190, 382/192, 382/194, 382/218, 382/274, 382/291, 382/103, 382/221, 382/100, 382/141, 351/206, 351/207, 351/208, 351/247, 351/212, 348/576
• US Class Current: 382/115
• CPC Class Codes: G06T 2207/30216 Redeye defect
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

The present apparatus and method captures at least a pair of images in relatively quick succession with an illumination source being fired during the capture of one of the images. The illumination source is located adjacent to the optical lens or close to the optical axis of the camera. Next the illumination level of the images is compensated to make them equal in illumination. Following that an analysis of the images is performed to determine the difference between the illumination compensated images. After the illumination compensation the difference has to be caused by eye-defects or specular reflections from glasses. The pixel coordinates of the transitory eye-defect regions are recorded and analyzed to determine the spacing of the potential eye locations and/or the spatial structure of the eyes for plausibility. Once confirmed the coordinates are output and recorded.

Claims

1. A method for determining the position of eyes within a frame of captured image data comprising the steps of:

capturing two color frames of an image scene wherein one frame is captured with an ambient illumination and the other frame is captured with an additional illumination;

storing said frames of said image scene in a memory;

compensating said frames for the overall illumination change between said two captured frames to obtain a compensated frame;

determining the difference between said compensated frame and an uncompensated frame of said two captured frames to obtain a compensated color difference image;

scanning said compensated color difference image and determining pairs of regions of high intensity pixel value in at least one color channel; and

outputting the data of pixel pair coordinates wherein each coordinate corresponds to a center of each region of high intensity pixel value in said color channel within said captured image scene.

2. The method according to claim 1, wherein said compensating step comprises:

a) converting the value of each pixel in said captured frames into log metric;

b) calculating the mean log intensity value of each of said converted images;

c) adding the difference between said mean log intensity values to the converted image corresponding to the subtrahend of said difference; and

d) converting said compensated log image to the inverse log metric to obtain a compensated frame.

3. The method according to claim 1, wherein said determined pairs of regions of high intensity pixel value, are processed in the red channel for human eyes, according to color, size, shape structure, and spacing.

4. The method according to claim 1, wherein said determined pairs of regions of high intensity pixel value, are processed in the green channel for animal eyes, according to color, size, shape structure, and spacing.

5. The method according to claim 1, wherein said determined pairs of regions of high intensity pixel value, preferably in the red and green channels for human and animal eyes, are processed according to color, size, shape structure, and spacing.

6. The method according to claim 1 further comprising the step of:

analyzing the spacing of a potential eye location and the spatial structure for plausibility.

7. The method according to claim 1 wherein the outputting step comprises recording the pixel coordinates of the center of said regions of high intensity pixel value.

8. The method according to claim 1 wherein said frames of said image scene are consecutive pictures of a video image stream.

9. An apparatus for determining the position of eyes within a frame of captured image data comprising:

means for capturing at least two color frames of an image scene, said means is provided with an optical lens defining an optical axis;

means for storing the data of at least two consecutive captured frames of an image scene;

an illumination source for illuminating one of said captured frames of said image scene wherein said illumination source is positioned as close as possible to said optical axis; and

electronic means;

a) for compensating said frames for the overall illumination change between said two captured frames to obtain a compensated frame;

b) for determining the difference between said compensated frame and an uncompensated frame of said two captured frames to obtain a compensated color difference image;

c) for scanning said compensated color difference image and determining pairs of regions of high intensity pixel value in at least one color channel; and

d) for outputting the data of pixel pair coordinates wherein each coordinate corresponds to a center of each region of high intensity pixel value in said color channel within said captured image scene.

10. Apparatus according to claim 9 wherein the capturing means is a video camera for capturing consecutive frames of an image scene.

11. Apparatus according to claim 9 wherein the illumination source is a flash light emitting source for triggering direct illumination in synchronization with the capturing of the second image frame of said image scene.

12. Apparatus according to claim 9 wherein the electronic means is a computer to which a keyboard, a monitor, an external storage, and a printer are connected.

13. A method, used in a digital camera, for locating eye-defect regions comprises the steps of:

selecting a function of the digital camera for eye-defect free picture taking;

capturing two color frames of an image scene with a minimum delay in between to minimize spatial registration error, wherein one frame is captured with an ambient illumination and the other frame is captured with an additional camera triggered illumination;

storing data, representing said two captured frames in a memory;

compensating said frames for the overall illumination change between said two captured frames to obtain a compensated frame;

determining the difference between said compensated frame and an uncompensated frame of said two captured frames to obtain a compensated color difference image;

scanning said compensated color difference image and determining pairs of regions of high intensity pixel value in at least one color channel; and

obtaining the data of pixel pair coordinates of said regions of bright intensity pixel value wherein each coordinate corresponds to a center of each of said eye-defect regions.

14. The method according to claim 13, wherein said compensating step comprises:

a) converting the value of each pixel in said captured frames into log metric;

b) calculating the mean log intensity value of each of said converted images;

c) adding the difference between said mean log intensity values to the converted image corresponding to the subtrahend of said difference; and

d) converting said compensated log image to the inverse log metric to obtain a compensated frame.

15. The method according to claim 13 further comprising the step of:

analyzing the spacing of a potential eye-defect region and the spatial structure for plausibility.

16. The method according to claim 13 wherein the outputting step comprises recording the pixel coordinates of the center of said eye-defect regions.

17. The method according to claim 15, wherein said determined eye- defect regions of high intensity pixel value, are processed in the red channel for human eyes, according to color, size, shape structure, and spacing.

18. The method according to claim 15, wherein said determined eye- defect regions of high intensity pixel value, are processed in the green channel for animal eyes, according to color, size, shape structure, and spacing.

19. The method according to claim 15, wherein said determined eye- defect regions of high intensity pixel value, preferably in the red and green channels for human and animal eyes, are processed according to color, size, shape structure, and spacing.

20. Method according to claim 13 wherein based on the user selection the digital camera performs an automatic eye-defect correction of the captured image scene.

21. Method according to claim 13 wherein, based on the user selection, the digital camera stores the data of the frame bearing said camera triggered illumination along with the obtained spatial data of the centers of each of said eye-defect regions.

22. A digital camera capable of locating eye-defect regions comprising:

a user input control for initiating the eye-defect correction function of the digital camera;

means for capturing two color frames of an image scene with a minimum delay between to minimize spatial registration error, wherein said capturing means is provided with an optical lens defining an optical axis;

an illumination source for illuminating one of the two frames of said image scene wherein said illumination source is positioned as close as possible to said optical axis;

means for storing the data of said two captured frames of said image scene; and

electronic means;

for compensating said frames for the overall illumination change between said two captured frames to obtain a compensated frame;

for determining the difference between said compensated frame and an uncompensated frame of said two captured frames to obtain a compensated color difference image; and

for scanning said compensated color difference image and thereby determining pairs of regions of eye-defect, and for calculating spatial coordinates of said eye-defects.

23. The digital camera according to claim 22 wherein said means for storing is used to store the captured image having the eye-defect correction automatically performed.

24. The digital camera according to claim 22 wherein said means for storing is used to store the data of the frame bearing said camera triggered illumination along with the calculated data of the spatial coordinates of the centers of each of said eye-defect regions.

25. Apparatus according to claim 22 wherein the illumination source is a flash light emitting source.

26. Apparatus according to claim 22 wherein the electronic means is a built-in central processing unit which controls the functions of the digital camera and an attached memory.

27. A method, used in a photographic film camera, for locating eye-defect regions comprising the steps of:

selecting a function of the photographic film camera for eye-defect free picture taking;

exposing the photographic film with a frame of an image scene bearing the camera triggered illumination and simultaneously storing data of said frame bearing the camera triggered illumination in a memory;

capturing a further frame of said image scene with a minimum delay in between to minimize spatial registration error, wherein said frame is captured with an ambient illumination, and storing data representing said frame in said memory;

compensating said frames for the overall illumination change between said two captured frames to obtain a compensated frame;

determining the difference between said compensated frame and an uncompensated frame of said two captured frames to obtain a compensated color difference image;

scanning said compensated color difference image and determining pairs of regions of high intensity pixel value in at least one color channel;

obtaining data of pixel pair coordinates corresponding to the pairs of regions of high intensity pixel value wherein each coordinate corresponds to a center of each of said eye-defect regions; and

recording on the photographic film inside the camera coordinates of said pairs of regions of high intensity pixel value preferably in the red channel.

28. The method according to claim 27, wherein said compensating step comprises:

converting the value of each pixel in said captured frames into log metric;

calculating the mean log intensity value of each of said converted images;

adding the difference between said mean log intensity values to the converted image corresponding to the subtrahend of said difference; and

converting said compensated log image to the inverse log metric to obtain a compensated frame.

29. The method according to claim 27 further comprising the step of analyzing the spacing of a potential eye-defect region and the spatial structure for plausibility.

30. The method according to claim 29, wherein said determined eye-defect regions of high intensity pixel value, are processed in the red channel for human eyes, according to color, size, shape structure, and spacing.

31. The method according to claim 29, wherein said determined eye-defect regions of high intensity pixel value, are processed in the green channel for animal eyes, according to color, size, shape structure, and spacing.

32. The method according to claim 29, wherein said determined eye-defect regions of high intensity pixel value, preferably in the red and green channels for human and animal eyes, are processed according to color, size, shape structure, and spacing.

33. Method according to claim 27 wherein based on the user selection, the digital camera performs an automatic eye-defect correction of the captured image scene.

34. Method according to claim 27 wherein based on the user selection the digital camera stores the data of the frame bearing said camera triggered illumination along with the obtained spatial data of the centers of each of said eye-defect regions.

35. The method according to claim 27, wherein said film includes a magnetic layer and, wherein the recording on the photographic film is magnetic recording on said magnetic layer.

36. A photographic film camera capable of locating eye-defects comprises:

a user input control for initiating the eye-defect free function of the film camera;

means for capturing two frames of an image scene with a minimum delay between to minimize spatial registration error, wherein said capturing means is provided with an optical lens defining an optical axis;

an illumination source for illuminating one of said two frames of said image scene, wherein said illumination source is positioned as close as possible to said optical axis;

a photographic film for capturing said illuminated frame of said image scene, said film including a recording layer;

means for temporarily storing data of said illuminated frame and data of said frame bearing ambient illumination of said image scene;

electronic means

for compensating said frames for the overall illumination change between said two captured frames to obtain a compensated frame;

for determining the difference between said compensated frame and an uncompensated frame of said two captured frames to obtain a compensated color difference image;

for scanning said compensated color difference image and thereby determining pairs of regions of eye-defect, and calculating spatial coordinates of said eye-defects; and

recording means for writing the spatial coordinates of the eye-defects in said recording layer on said photographic film inside said photographic film camera.

37. The photographic film camera according to claim 36 wherein the capturing means comprise a semi transparent mirror located in said optical axis in order to simultaneously illuminate said photographic film and an image sensor, said illumination of said photographic film is controlled by a shutter, positioned prior to said photographic film.

38. The photographic film camera according to claim 36 wherein the recording layer on said photographic film is a magnetic recording layer and the recording means are designed for magnetic recording of data on said magnetic recording layer.

39. The photographic film camera according to claim 36 wherein the illumination source is a flash light emitting source.

40. The photographic film camera according to claim 36 wherein said electronic means is a camera built-in central processing unit which controls the main camera functions, a memory for temporary storing and said recording means.