Rapid auto-focus using classifier chains, MEMS and/or multiple object focusing

• US 8,659,697 B2
• Filed: 11/11/2010
• Issued: 02/25/2014
• Est. Priority Date: 11/11/2010
• Status: Expired due to Fees

• Current Assignee: FotoNation Limited (Adeia, Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia, Inc.)
• Inventors: Capata, Adrian, Petrescu, Stefan, Bigioi, Petronel, Corcoran, Peter
• Primary Examiner(s): Tran, Nhan T
• Application Number:
  US12/944,703
• Publication Number:
  US 20120120269A1
• Time in Patent Office: 1,202 Days
• Field of Search: 348/345, 348/346, 348/169, 348/239, 348/349, 348352-354, 382/118, 382/255
• US Class Current: 348/345
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06V 10/7747 Organisation of the process, e.g. bagging or boosting
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/675 comprising setting of focusing regions

Abstract

A smart-focusing technique includes identifying an object of interest, such as a face, in a digital image. A focus-generic classifier chain is applied that is trained to match both focused and unfocused faces and/or data from a face tracking module is accepted. Multiple focus-specific classifier chains are applied, including a first chain trained to match substantially out of focus faces, and a second chain trained to match slightly out of focus faces. Focus position is rapidly adjusted using a MEMS component.

Claims

1. A method of digital image processing using face detection, comprising:

acquiring a scene with a digital image acquisition device including multiple features;

identifying a first group of pixels that correspond to a first feature of a face within the scene;

adjusting an optic in real time using a MEMS component to a first focus position to focus the device on the first feature;

capturing a first digital image at the first focus position;

applying multiple focus-specific classifier chains, including a first trained to match substantially out of focus faces, and a second trained to match slightly out of focus faces, and a third focus-specific classifier chain trained to match sharply focused faces; and

when a second face is determined not to be sharply focused as significantly matching the first or second focus-specific classifier chains, or both, adjusting focus position using a MEMS component one or more times based on a degree of focus determined at one or more preceding focus positions and, such as to ultimately focus on the second face using a series of one or more selected smart-focus adjustments based on application of the focus-specific classifier chains; and

one or more of (i), (ii)(a), or (ii)(b), as follows;

(i) wherein when the second face is determined to be substantially unfocused as best matching the first focus-specific classifier chain, then the adjusting comprises coarsely adjusting to a second focus position and repeating the applying of the multiple focus-specific classifier chains, or

(ii) wherein when the second face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and

(a) repeating the applying of the multiple focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain;

or

(b) repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the second face; and

capturing a second digital image having a second feature at the second focus position;

registering the first and second digital images including the first and second features; and

storing, transmitting, combining, capturing or displaying the first and second digital images together, or combinations thereof.

2. The method of claim 1, further comprising determining that the first feature appears blurry in the second digital image or that the second feature appears blurry in the first image, or both, and wherein the storing, transmitting or displaying of the first and second digital images together, or combinations thereof, comprises generating a composite image including the first feature from the first digital image and the second feature from the second digital image, such that both of the first and second features appear to be sharp in the composite image.

3. The method of claim 2, wherein the generating a composite image comprises blending the first and second images.

4. The method of claim 2, wherein the generating a composite image comprises morphing the first and second images.

5. The method of claim 1, further comprising wherein the storing, transmitting or displaying of the first and second digital images together, or combinations thereof, comprises providing a toggle feature to call the first and second digital images together and toggle between them.

6. The method of claim 5, further comprising receiving a selection of one of the first and second digital images for further storing, transmitting, displaying or further processing, or combinations thereof.

7. The method of claim 5, wherein the toggle feature permits a display to appear to demonstrate toggling between the first and second focus positions in a same image.

8. The method of claim 1, wherein the registering comprises aligning the first and second digital images by applying digital or mechanical image stabilization, or both.

9. The method of claim 1, wherein the identifying of the first or second groups of pixels respectively comprises applying face detection to the acquired scene and identifying the first or second features as corresponding to first or second faces.

10. The method of claim 9, further comprising applying face tracking to the first or second group of pixels, or both, for respectively continuing to identify in subsequent images the first or second groups of pixels as corresponding to the first or second face.

11. A method of smart-focusing on a detected face region in a scene, comprising:

acquiring a digital image using a camera-enabled device including a lens, an image sensor, a memory and a processor;

identifying a group of pixels that contain a face, including applying a focus-generic classifier chain trained to match both focused and unfocused faces or accepting data from a face tracking module, or both;

applying multiple focus-specific classifier chains, including a first trained to match substantially out of focus faces, and a second trained to match slightly out of focus faces, and a third trained to match sharply focused faces; and

when the face is determined not to be sharply focused as significantly matching the first or second focus-specific classifier chains, or both, adjusting focus position using a MEMS component one or more times based on a degree of focus determined at one or more preceding focus positions and, such as to ultimately focus on the face using a series of one or more selected smart-focus adjustments based on application of focus-specific classifier chains; and

one or more of (i), (ii)(a), or (ii)(b), as follows;

(i) wherein when a second face is determined to be substantially unfocused as best matching the first focus-specific classifier chain, then the adjusting comprises coarsely adjusting to a second focus position and repeating the applying of the multiple focus-specific classifier chains, or

(ii) wherein when the second face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and

(a) repeating the applying of the multiple focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain;

or

(b) repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the second face.

12. The method of claim 11, wherein when the face is determined to be substantially unfocused at the second focus position, then the method further comprises coarsely adjusting focus to a third focus position oppositely as the adjusting to the second focus position in a reduced or increased amount as the adjusting to the second focus position, or both.

13. The method of claim 11, wherein the multiple focus-specific classifier chains further comprise a third focus-specific classifier chain trained to match sharply focused faces, and wherein when the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and repeating the applying of the at least three focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain.

14. The method of claim 11, wherein the multiple focus-specific classifier chains further comprise a third focus-specific classifier chain trained to match sharply focused faces, and wherein when the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and repeating the applying of the multiple focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain.

15. The method of claim 11, wherein when the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the face.

16. The method of claim 11, further comprising applying one or more specific facial feature classifier chains each trained to enable optimal focus on a feature of the detected face including one or both eyes, mouth, chin, nose, one or both ears, hairline, forehead, profile or other partial face region, or combinations thereof.

17. The method of claim 16, further comprising adjusting focus position to obtain optimal focus on a selected specific facial feature of the detected face.

18. The method of claim 11, further comprising repeating the applying of at least one of the multiple focus-specific classifier chains at each focus position.

19. One or more non-transitory processor-readable media having code embedded therein for programming a processor to perform a method of digital image processing using face detection, wherein the method comprises:

acquiring a scene with a digital image acquisition device including multiple features;

identifying a first group of pixels that correspond to a first feature of a face within the scene;

adjusting an optic in real time using a MEMS component to a first focus position to focus the device on the first feature;

capturing a first digital image at the first focus position;

applying multiple focus-specific classifier chains, including a first trained to match substantially out of focus faces, and a second trained to match slightly out of focus faces, and a third focus-specific classifier chain trained to match sharply focused faces; and

when a second face is determined not to be sharply focused as significantly matching the first or second focus-specific classifier chains, or both, adjusting focus position using a MEMS component one or more times based on a degree of focus determined at one or more preceding focus positions and, such as to ultimately focus on the second face using a series of one or more selected smart-focus adjustments based on application of focus-specific classifier chains; and

one or more of (i), (ii)(a), or (ii)(b), as follows;

(i) wherein when the second face is determined to be substantially unfocused as best matching the first focus-specific classifier chain, then the adjusting comprises coarsely adjusting to a second focus position and repeating the applying of the multiple focus-specific classifier chains, or

(ii) wherein when the second face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and

(a) repeating the applying of the multiple focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain;

or

(b) repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the second face; and

capturing a second digital image having a second feature at the second focus position;

registering the first and second digital images including the first and second features; and

storing, transmitting, combining, capturing or displaying the first and second digital images together, or combinations thereof.

20. The one or more non-transitory processor-readable media of claim 19, wherein the method further comprises determining that the first feature appears blurry in the second digital image or that the second feature appears blurry in the first image, or both, and wherein the storing, transmitting or displaying of the first and second digital images together, or combinations thereof, comprises generating a composite image including the first feature from the first digital image and the second feature from the second digital image, such that both of the first and second features appear to be sharp in the composite image.

21. The one or more non-transitory processor-readable media of claim 20, wherein the generating a composite image comprises blending the first and second images.

22. The one or more non-transitory processor-readable media of claim 20, wherein the generating a composite image comprises morphing the first and second images.

23. The one or more non-transitory processor-readable media of claim 19, wherein the method further comprises wherein the storing, transmitting or displaying of the first and second digital images together, or combinations thereof, comprises providing a toggle feature to call the first and second digital images together and toggle between them.

24. The one or more non-transitory processor-readable media of claim 23, wherein the method further comprises receiving a selection of one of the first and second digital images for further storing, transmitting, displaying or further processing, or combinations thereof.

25. The one or more non-transitory processor-readable media of claim 23, wherein the toggle feature permits a display to appear to demonstrate toggling between the first and second focus positions in a same image.

26. The one or more non-transitory processor-readable media of claim 19, wherein the registering comprises aligning the first and second digital images by applying digital or mechanical image stabilization, or both.

27. The one or more non-transitory processor-readable media of claim 19, wherein the identifying of the first or second groups of pixels respectively comprises applying face detection to the acquired scene and identifying the first or second features as corresponding to first or second faces.

28. The one or more non-transitory processor-readable media of claim 27, wherein the method further comprises applying face tracking to the first or second group of pixels, or both, for respectively continuing to identify in subsequent images the first or second groups of pixels as corresponding to the first or second face.

29. One or more non-transitory processor-readable media having code embedded therein for programming a processor to perform a method of smart-focusing on a detected face region in a scene, wherein the method comprises:

acquiring a digital image using a camera-enabled device including a lens, an image sensor, a memory and a processor;

identifying a group of pixels that contain a face, including applying a focus-genetic classifier chain trained to match both focused and unfocused faces or accepting data from a face tracking module, or both;

applying multiple focus-specific classifier chains, including a first trained to match substantially out of focus faces, and a second trained to match slightly out of focus faces, and a third trained to match sharply focused faces; and

when the face is determined not to be sharply focused as significantly matching the first or second focus-specific classifier chains, or both, adjusting focus position using a MEMS component one or more times based on a degree of focus determined at one or more preceding focus positions and, such as to ultimately focus on the face using a series of one or more selected smart-focus adjustments based on application of focus-specific classifier chains; and

one or more of (i), (ii)(a), or (ii)(b), as follows;

(i) wherein when a second face is determined to be substantially unfocused as best matching the first focus-specific classifier chain, then the adjusting comprises coarsely adjusting to a second focus position and repeating the applying of the multiple focus-specific classifier chains, or

(ii) wherein when the second face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and

(a) repeating the applying of the multiple focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain;

or

(b) repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the second face.

30. The one or more non-transitory processor-readable media of claim 29, wherein when the face is determined to be substantially unfocused at the second focus position, then the method further comprises coarsely adjusting focus to a third focus position oppositely as the adjusting to the second focus position in a reduced or increased amount as the adjusting to the second focus position, or both.

31. The one or more non-transitory processor-readable media of claim 29, wherein the multiple focus-specific classifier chains further comprise a third focus-specific classifier chain trained to match sharply focused faces, and wherein when the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and repeating the applying of the at least three focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain.

32. The one or more non-transitory processor-readable media of claim 29, wherein the multiple focus-specific classifier chains further comprise a third focus-specific classifier chain trained to match sharply focused faces, and wherein when the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and repeating the applying of the multiple focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain.

33. The one or more non-transitory processor-readable media of claim 29, wherein when the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the face.

34. The one or more non-transitory processor-readable media of claim 29, wherein the method further comprises applying one or more specific facial feature classifier chains each trained to enable optimal focus on a feature of the detected face including one or both eyes, mouth, chin, nose, one or both ears, hairline, forehead, profile or other partial face region, or combinations thereof.

35. The one or more non-transitory processor-readable media of claim 34, wherein the method further comprises adjusting focus position to obtain optimal focus on a selected specific facial feature of the detected face.

36. The one or more non-transitory processor-readable media of claim 29, wherein the method further comprises repeating the applying of at least one of the multiple focus-specific classifier chains at each focus position.

37. A digital image acquisition device, comprising:

a lens and image sensor for acquiring digital images;

a processor; and

one or more non-transitory processor-readable media having code embedded therein for programming a processor to perform a method of digital image processing using face detection, wherein the method comprises;

acquiring a scene with a digital image acquisition device including multiple features;

identifying a first group of pixels that correspond to a first feature of a face within the scene;

adjusting an optic in real time using a MEMS component to a first focus position to focus the device on the first feature;

capturing a first digital image at the first focus position;

applying multiple focus-specific classifier chains, including a first trained to match substantially out of focus faces, and a second trained to match slightly out of focus faces, and a third focus-specific classifier chain trained to match sharply focused faces; and

when a second face is determined not to be sharply focused as significantly matching the first or second focus-specific classifier chains, or both, adjusting focus position using a MEMS component one or more times based on a degree of focus determined at one or more preceding focus positions and, such as to ultimately focus on the second face using a series of one or more selected smart-focus adjustments based on application of focus-specific classifier chains; and

one or more of (i), (ii)(a), or (ii)(b), as follows;

(i) wherein when the second face is determined to be substantially unfocused as best matching the first focus-specific classifier chain, then the adjusting comprises coarsely adjusting to a second focus position and repeating the applying of the multiple focus-specific classifier chains, or (ii) wherein when the second face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and

(a) repeating the applying of the multiple focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain;

or

(b) repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the face; and

capturing a second digital image having a second feature at the second focus position;

registering the first and second digital images including the first and second features; and

storing, transmitting, combining, capturing or displaying the first and second digital images together, or combinations thereof.

38. The device of claim 37, wherein the method further comprises determining that the first feature appears blurry in the second digital image or that the second feature appears blurry in the first image, or both, and wherein the storing, transmitting or displaying of the first and second digital images together, or combinations thereof, comprises generating a composite image including the first feature from the first digital image and the second feature from the second digital image, such that both of the first and second features appear to be sharp in the composite image.

39. The device of claim 38, wherein the generating a composite image comprises blending the first and second images.

40. The device of claim 38, wherein the generating a composite image comprises morphing the first and second images.

41. The device of claim 37, wherein the method further comprises wherein the storing, transmitting or displaying of the first and second digital images together, or combinations thereof, comprises providing a toggle feature to call the first and second digital images together and toggle between them.

42. The device of claim 41, wherein the method further comprises receiving a selection of one of the first and second digital images for further storing, transmitting, displaying or further processing, or combinations thereof.

43. The device of claim 41, wherein the toggle feature permits a display to appear to demonstrate toggling between the first and second focus positions in a same image.

44. The device of claim 37, wherein the registering comprises aligning the first and second digital images by applying digital or mechanical image stabilization, or both.

45. The device of claim 37, wherein the identifying of the first or second groups of pixels respectively comprises applying face detection to the acquired scene and identifying the first or second features as corresponding to first or second faces.

46. The device of claim 45, wherein the method further comprises applying face tracking to the first or second group of pixels, or both, for respectively continuing to identify in subsequent images the first or second groups of pixels as corresponding to the first or second face.

47. A digital image acquisition device, comprising:

a lens and image sensor for acquiring digital images;

a processor; and

one or more non-transitory processor-readable media having code embedded therein for programming a processor to perform a method of smart-focusing on a detected face region in a scene, wherein the method comprises;

acquiring a digital image using a camera-enabled device including a lens, an image sensor, a memory and a processor;

identifying a group of pixels that contain a face, including applying a focus-generic classifier chain trained to match both focused and unfocused faces or accepting data from a face tracking module, or both;

applying multiple focus-specific classifier chains, including a first trained to match substantially out of focus faces, and a second trained to match slightly out of focus faces; and

when the face is determined not to be sharply focused as significantly matching the first or second focus-specific classifier chains, or both, adjusting focus position using a MEMS component one or more times based on a degree of focus determined at one or more preceding focus positions and, such as to ultimately focus on the face using a series of one or more selected smart-focus adjustments based on application of focus-specific classifier chains; and

one or more of (i), (ii)(a), or (ii)(b) as follows;

(i) wherein when the face is determined to be substantially unfocused as best matching the first focus-specific classifier chain, then the adjusting comprises coarsely adjusting to a second focus position and repeating the applying of the multiple focus-specific classifier chains, or

(ii) wherein when a second face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and

(a) repeating the applying of the multiple focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain;

or

(b) repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the face.

48. The device of claim 47, wherein when the face is determined to be substantially unfocused at the second focus position, then the method further comprises coarsely adjusting focus to a third focus position oppositely as the adjusting to the second focus position in a reduced or increased amount as the adjusting to the second focus position, or both.

49. The device of claim 47, wherein the multiple focus-specific classifier chains further comprise a third focus-specific classifier chain trained to match sharply focused faces, and wherein when the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and repeating the applying of the at least three focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain.

50. The device of claim 47, wherein the multiple focus-specific classifier chains further comprise a third focus-specific classifier chain trained to match sharply focused faces, and wherein when the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and repeating the applying of the multiple focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain.

51. The device of claim 47, wherein when the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting comprises finely adjusting to a second focus position and repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the face.

52. The device of claim 47, wherein the method further comprises applying one or more specific facial feature classifier chains each trained to enable optimal focus on a feature of the detected face including one or both eyes, mouth, chin, nose, one or both ears, hairline, forehead, profile or other partial face region, or combinations thereof.

53. The device of claim 52, wherein the method further comprises adjusting focus position to obtain optimal focus on a selected specific facial feature of the detected face.

54. The device of claim 47, wherein the method further comprises repeating the applying of at least one of the multiple focus-specific classifier chains at each focus position.