Method and apparatus for removal of bright or dark spots by the fusion of multiple images

• US 6,088,470 A
• Filed: 01/27/1998
• Issued: 07/11/2000
• Est. Priority Date: 01/27/1998
• Status: Expired due to Fees

• Current Assignee: Sensar, Inc. (Advent International Corporation), Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: Sensar, Inc. (Advent International Corporation), Sarnoff Corporation (SRI International, Inc.)
• Inventors: Camus, Theodore A., Hanna, Keith James, Salganicoff, Marcus, Chmielewski, Thomas A. Jr.
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Werner, Brian P.
• Application Number:
  US09/013,758
• Time in Patent Office: 896 Days
• Field of Search: 382/103, 382/117, 382/118, 382/274, 382/275, 382/299, 382/101, 382/102, 382/142, 351/206, 351/207, 396/18, 396/51, 396/158, 348/25, 348/78
• US Class Current: 382/117
• CPC Class Codes: A61B 3/156 for blocking
• CPC Class Codes: G06F 21/83 input devices, e.g. keyboards, mice or controllers thereof
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

A reliable method and apparatus for illuminating and imaging eyes uses multiple light sources producing multiple images of a subject each created under illumination by different illuminators. A composite image of the subject is formed by selecting pixels based upon their gray scale values or using pyramid image processing. A composite image can be created which is free of bright spots commonly caused by reflection of illumination from eyeglasses that may be worn by a subject or for which is free of dark shadows or which is free of both bright spots and dark shadows.

Claims

1. A method for creating an image of at least a portion of a head of a person who is wearing eyeglasses or contact lens and who is positioned at a subject location which image is free of specularities comprising:

a. positioning a first illuminator and at least one additional illuminator at a selected distance from the subject location and spaced apart from one another;

b. capturing a first image of at least a portion of a head of a person who is wearing eyeglasses or contact lens while the person'"'"'s head is illuminated by the first illuminator and not illuminated by the at least one additional illuminator wherein the first image is comprised of a set of pixels each pixel corresponding to a specific location relative to the person'"'"'s head and having a gray scale value;

c. capturing at least one additional image of at least a portion of a head of a person who is wearing eyeglasses or contact lens while the person'"'"'s head is illuminated by the at least one additional illuminator and not illuminated by the first illuminator;

wherein the second image is comprised of a set of pixels each pixel corresponding to a specific location relative to the person'"'"'s head and having a gray scale value;

d. creating sets of pixels such that each set contains a pixel from the first image and a pixel from each additional image and all pixels in each set correspond to a same location relative to the subject; and

e. constructing an image of the subject that is free of specularities by selecting one pixel from each set of pixels by choosing that pixel which has a minimum gray scale value.

2. The method of claim 1 wherein there is only one additional illuminator and only one additional image.

3. The method of claim 1 wherein the illuminators emit infrared light.

4. The method of claim 1 wherein at least one of the illuminators is an illuminator selected from the group consisting of light-emitting diodes, laser diodes, lasers fitted with a diverging lens, and an incandescent lamp.

5. The method of claim 1 also comprising the step of using the image of the at least a portion of a head of a person that is free of specularities to identify the person.

6. The method of claim 5 wherein the person is identified using a biometric identification method.

7. The method of claim 6 wherein the biometric identification method is iris identification.

8. The method of claim 1 wherein the first image is captured by:

a. capturing a first preliminary video image of the at least a portion of a head of a person which contains a field created while the at least a portion of a head of a person is illuminated by the first illuminator and not illuminated by the at least one additional illuminator and a second field which was created while the at least a portion of a head of a person was not illuminated by any illuminator wherein the first image is comprised of a set of pixels each pixel corresponding to a specific location relative to the at least a portion of a head of a person and having a gray scale value; and

b. forming the first image by subtraction of the second field from the first field.

9. The method of claim 8 wherein the at least one additional image is captured by:

a. capturing a second preliminary video image of the subject which contains two fields created while the at least a portion of a head of a person is illuminated by the at least one additional illuminator and not illuminated by the first illuminator and a second field which was created while the at least a portion of a head of a person was not illuminated by the first illuminator and wherein the second image is comprised of a set of pixels each pixel corresponding to a specific location relative to the at least a portion of a head of a person and having a gray scale value; and

b. forming the second image by either subtraction from the first field of the second preliminary image of the second field of the first preliminary image or by subtraction from the second field of the second preliminary image of the second field of the first preliminary image.

10. A method for creating an image of a subject positioned at a subject location which image is free of specularities comprising:

a. positioning a first illuminator and at least one additional illuminator at a selected distance from the subject location and spaced apart from one another;

b. capturing a first image of the subject while the subject is illuminated by he first illuminator and not illuminated by the at least one additional illuminator wherein the first image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value;

c. capturing at least one additional image of the subject while the subject is illuminated by the at least one additional illuminator and not illuminated by the first illuminator;

wherein the second image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value;

d. creating sets of pixels such that each set contains a pixel from the first image and a pixel from each additional image and all pixels in each set correspond to a same location relative to the subject;

e. constructing an image of the subject that is free of specularities by selecting one pixel from each set of pixels by choosing that pixel which has a minimum gray scale value;

f. capturing a third image of the subject wherein the third image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value; and

g. including a pixel from the third image in at least some of the sets of pixels.

11. A method for creating an image of a subject that is located in a package at least a portion of which package is light transmissive which image is free of specularities comprising:

a. positioning a first illuminator and at least one additional illuminator at a selected distance from the subject location and spaced apart from one another;

b. capturing a first image of the subject while the subject is illuminated by the first illuminator and not illuminated by the at least one additional illuminator wherein the first image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value;

c. capturing at least one additional image of the subject while the subject is illuminated by the at least one additional illuminator and not illuminated by the first illuminator;

wherein the second image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value;

d. creating sets of pixels such that each set contains a pixel from the first image and a pixel from each additional image and all pixels in each set correspond to a same location relative to the subject; and

e. constructing an image of the subject that is free of specularities by selecting one pixel from each set of pixels by choosing that pixel which has a minimum gray scale value.

12. method of claim 11 wherein the illuminators emit infrared light.

13. The method of claim 11 wherein at least one of the illuminators is an illuminator selected from the group consisting of light-emitting diodes, laser diodes, lasers fitted with a diverging lens, and an incandescent lamp.

14. The method of claim 11 also comprising the step of using the image of the subject that is free of specularities to identify the subject.

15. The method of claim 11 wherein the first image is captured by:

a. capturing a first preliminary video image of the subject which contains a field created while the subject is illuminated by the first illuminator and not illuminated by the at least one additional illuminator and a second field which was created while the subject was not illuminated by any illuminator wherein the first image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value; and

b. forming the first image by subtraction of the second field from the first field.

16. The method of claim 15 wherein the at least one additional image is captured by:

a. capturing a second preliminary video image of the subject which contains two fields created while the subject is illuminated by the at least one additional illuminator and not illuminated by the first illuminator and a second field which was created while the subject was not illuminated by the first illuminator and wherein the second image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value; and

b. forming the second image by either subtraction from the first field of the second preliminary image the second field of the first preliminary image or by subtraction from the second field of the second preliminary image the second field of the first preliminary image.

17. A method for creating a shadow free image of a subject positioned at a subject location comprising:

a. positioning a first illuminator and at least one additional illuminator at a selected distance from the subject location and spaced apart from one another;

b. capturing a first image of the subject while the subject is illuminated by the first illuminator and not illuminated by the at least one additional illuminator wherein the first image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value;

c. capturing at least one additional image of the subject while the subject is illuminated by the at least one additional illuminator and not illuminated by the first illuminator;

wherein the second image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value;

d. creating sets of pixels such that each set contains a pixel from the first image and a pixel from each additional image and all pixels in each set correspond to a same location relative to the subject; and

e. constructing an image of the subject by selecting one pixel from each set of pixels by choosing that pixel which has a maximum gray scale value that is less than a threshold gray scale value.

18. The method of claim 17 wherein the threshold gray scale value is 255.

19. The method of claim 17 wherein there is only one additional illuminator and only one additional image.

20. The method of claim 17 wherein the subject is at least a portion of a head of a person who is wearing eyeglasses or contact lenses.

21. The method of claim 17 wherein the subject is located in a package at least a portion of which is light transmissive.

22. The method of claim 17 wherein the illuminators emit infrared light.

23. The method of claim 17 wherein at least one of the illuminators is an illuminator selected from the group consisting of light-emitting diodes, laser diodes, lasers fitted with a diverging lens, and an incandescent lamp.

24. A method for creating an image of a subject positioned at a subject location comprised of:

a. positioning a first illuminator and at least one additional illuminator at a selected distance from the subject location and spaced apart from one another;

b. capturing a first image of the subject while the subject is illuminated by the first illuminator and not illuminated by the at least one additional illuminator wherein the first image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value;

c. capturing at least one additional image of the subject while the subject is illuminated by the at least one additional illuminator and not illuminated by the first illuminator;

wherein the second image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value;

d. computing a Laplacian pyramid for each image;

e. selecting a first particular level of the Laplacian pyramids computed for each image such that a same level is selected for each image;

f. creating sets of pixels such that each set contains a pixel from the selected level of the Laplacian pyramid for each of the first image and each additional image and all pixels in each set correspond to a same location relative to the subject;

g. identifing a pixel in each set which has a maximum value and identifying the image from which each selected pixels was derived; and

h. constructing a first composite image of the subject by selecting one pixel from one of the first image and each additional image that corresponds to each pixel identified in step g.

25. The method of claim 24 wherein there is only one additional illuminator and only one additional image.

26. The method of claim 24 also comprising:

a. selecting a second particular level of the Laplacian pyramids computed for each image such that a same level is selected for each image and the second particular level selected is adjacent to the first selected level;

b. creating sets of pixels such that each set contains a pixel from the second selected level of the Laplacian pyramid for each of the first image and each additional image and all pixels in each set correspond to a same location relative to the subject;

c. identifying a pixel in each set which has a minimum value and identifying the image from which each selected pixels was derived; and

d. constructing a second composite image of the subject by selecting one pixel from one of the first image and each additional image that corresponds to each pixel identified in step c.

27. The method of claim 26 also comprising combining the first composite image and the second composite image to form a combined composite image.

28. The method of claim 26 also comprising:

a. selecting at least one additional particular level of the Laplacian pyramids computed for each image such that a same level is selected for each;

b. creating sets of pixels for each selected additional particular level such that each set contains a pixel from the additional particular selected level of the Laplacian pyramid for each of the first image and each additional image and all pixels in each set correspond to a same location relative to the subject;

c. identifying a pixel in each set which has a minimum value and identifying the image from which each selected pixels was derived; and

d. constructing an additional composite image of the subject for each additional selected level by selecting one pixel from one of the first image and each additional image that corresponds to each pixel identified in step c.

29. The method of claim 28 also comprising:

a. creating a constructed Laplacian pyramid which contains the first composite image, the second composite image and at least one additional composite images; and

b. constructing an image of the subject by applying an inverse Laplacian transform procedure to the created Laplacian pyramid.

30. The method of claim 26 also comprising:

a. selecting at least one additional particular level of the Laplacian pyramids computed for each image such that a same level is selected for each;

b. creating sets of pixels for each selected additional particular level such that each set contains a pixel from the additional particular selected level of the Laplacian pyramid for each of the first image and each additional image and all pixels in each set correspond to a same location relative to the subject;

c. identifying a pixel in each set which has a maximum value and identifying the image from which each selected pixels was derived; and

d. constructing an additional composite image of the subject for each additional selected level by selecting one pixel from one of the first image and each additional image that corresponds to each pixel identified in step c.

31. The method of claim 30 also comprising:

a. creating a constructed Laplacian pyramid which contains the first composite image, the second composite image and at least one additional composite images; and

b. constructing an image of the subject by applying an inverse Laplacian transform procedure to the created Laplacian pyramid.

32. A method for creating an image of a subject positioned at a subject location comprised of:

a. positioning a first illuminator and a at least one additional illuminator at a selected distance from the subject location and spaced apart from one another;

b. capturing a first image of the subject while the subject is illuminated by the first illuminator and not illuminated by the at least one additional illuminator wherein the first image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value;

c. capturing at least one additional image of the subject while the subject is illuminated by the at least one additional illuminator and not illuminated by the first illuminator;

wherein the second image is comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value;

d. computing a Laplacian pyramid for each image;

e. selecting a first particular level of the Laplacian pyramids computed for each image such that a same level is selected for each image;

f. creating sets of pixels such that each set contains a pixel from the selected level of the Laplacian pyramid for each of the first image and each additional image and all pixels in each set correspond to a same location relative to the subject;

g. identifying a pixel in each set which has a minimum value and identifying the image from which each selected pixels was derived; and

h. constructing a first composite image of the subject by selecting one pixel from one of the first image and each additional image that corresponds to each pixel identified in step g.

33. The method of claim 32 wherein there is only one additional illuminator and only one additional image.

34. An apparatus for creating an image of a subject positioned at a subject location which image is free of specularities or shadows comprising:

a. a camera for taking an image of a subject located at a subject location;

b. a first illuminator and at least one additional illuminator at a selected distance from the subject location and spaced apart from one another;

c. a controller attached to the camera and the illuminators for turning on a selected illuminator whenever the camera captures an image of the subject; and

d. an image processor connected to the camera for receiving a first image of the subject taken when the first illuminator is illuminated and the at least one additional illuminator is not illuminated and receiving at least one additional image of the subject taken when the first illuminator is not illuminated and the at least one additional illuminator is illuminated and combining the first image and at least one additional image to produce a composite image of the subject wherein the images are each comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value and wherein the image processor contains a program for combining the first and second images which program constructs a Laplacian pyramid for each image and uses at least some of levels of the Laplacian pyramids to form a composite image of the subject.

35. The apparatus of claim 34 wherein there is only one additional illuminator and only one additional image.

36. An apparatus for creating an image of a subject positioned at a subject location which image is free of specularities or shadows comprising:

a. a camera for taking an image of a subject located at a subject location;

b. a first illuminator and at least one additional illuminator at a selected distance from the subject location and spaced apart from one another;

c. a controller attached to the camera and the illuminators for turning on a selected illuminator whenever the camera captures an image of the subject; and

d. an image processor connected to the camera for receiving a first image of the subject taken when the first illuminator is illuminated and the at least one additional illuminator is not illuminated and receiving at least one additional image of the subject taken when the first illuminator is not illuminated and the at least one additional illuminator is illuminated and combining the first image and at least one additional image to produce a composite image of the subject wherein the images are each comprised of a set of pixels each pixel corresponding to a specific location relative to the subject and having a gray scale value and wherein the image processor contains a program for combining the first and second images by;

a. creating sets of pixels such that each set contains a pixel from the first image and a pixel from each additional image and all pixels in each set correspond to a same location relative to the subject; and

b. creating a constructed image of the subject by selecting one pixel from each set of pixels by choosing that pixel which has a maximum gray scale value that is less than a threshold gray scale value.