Face and other object detection and tracking in off-center peripheral regions for nonlinear lens geometries

• US 8,982,180 B2
• Filed: 04/02/2011
• Issued: 03/17/2015
• Est. Priority Date: 03/31/2011
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Corcoran, Peter, Bigioi, Petronel, Stec, Piotr
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Le, Peter D
• Application Number:
  US13/078,971
• Publication Number:
  US 20120249726A1
• Time in Patent Office: 1,445 Days
• Field of Search: 348/36
• US Class Current: 348/36
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A technique of enhancing a scene containing one or more off-center peripheral regions within an initial distorted image captured with a large field of view includes determining and extracting an off-center region of interest (hereinafter “ROI”) within the image. Geometric correction is applied to reconstruct the off-center ROI into a rectangular or otherwise undistorted or less distorted frame of reference as a reconstructed ROI.

Claims

1. Within an image acquisition system comprising a non-linear, wide-angled lens and an imaging sensor, a method of enhancing a scene containing one or more off-center peripheral regions, the method comprising:

acquiring, using the non-linear, wide-angled lens and the imaging sensor, an initial distorted image with a large field of view;

determining and extracting an off-center region of interest (hereinafter “

ROI”

) within said initial distorted image;

wherein the ROI includes a portion from said initial distorted image that has uncorrected distortion;

applying an initial object detection process to said off-center ROI that includes the portion from said initial distorted image having uncorrected distortion;

responsive to the initial object detection process successfully detecting an object within the off-center ROI;

applying geometric correction to remove distortion from the off-center ROI to produce a undistorted reconstructed ROI, without applying the geometric correction to said entire initial distorted image;

applying a second object detection process to one or more regions within the undistorted reconstructed ROI;

wherein the second object detection process is different than the initial object detection process.

2. A method as in claim 1, further comprising compensating for global motion of the imaging device.

3. A method as in claim 1, further comprising performing an action based on a location of the object tracked within the undistorted reconstructed ROI.

4. A method as in claim 1, further comprising determining a quality of reconstructed pixels within said undistorted reconstructed ROI;

and selectively applying image analysis to said undistorted reconstructed ROI based on the quality of the reconstructed pixels.

5. A method as in claim 4, further comprising responsive to said image analysis and said quality of reconstructed pixels, selectively applying image enhancement and generating an enhanced output image.

6. A method as in claim 1, further comprising determining if any undetected faces below a size threshold are likely to exist in one or more reduced quality regions of the undistorted reconstructed ROI.

7. A method as in claim 6, further comprising, responsive to said determining if any undetected faces exist, acquiring one or more additional initial distorted images, extracting and reconstructing one or more matching ROIs, and combining said one or more matching ROIs with reduced quality pixels of the original reconstructed ROI using a super-resolution technique, and generating, based on the combining, one or more enhanced ROIs.

8. A method as in claim 7, further comprising applying additional face detecting or tracking, or both, to the one or more enhanced ROIs to confirm a presence and location of a face below an original size threshold.

9. A digital image acquisition device, comprising:

a non-linear, wide-angled lens and an imaging sensor configured to capture digital images of scenes containing one or more off-center peripheral regions, including an initial distorted image with a large field of view;

a processor;

a memory having code embedded therein for programming the processor to perform a method of enhancing a scene containing one or more off-center peripheral regions, wherein the method comprises;

acquiring, using the non-linear, wide-angled lens and the imaging sensor, an initial distorted image with a large field of view;

determining and extracting an off-center region of interest (hereinafter “

ROI”

) within said initial distorted image;

wherein the ROI includes a portion from said initial distorted image that has uncorrected distortion;

applying an initial object detection process to said off-center ROI that includes the portion from said initial distorted image having uncorrected distortion

responsive to the initial object detection process successfully detecting an object within the off-center ROI;

applying geometric correction to remove distortion from the off-center ROI to produce a undistorted reconstructed ROI, without applying the geometric correction to said entire initial distorted image;

applying a second object detection process to one or more regions within the undistorted reconstructed ROI;

wherein the second object detection process is different than the initial object detection process.

10. A device as in claim 9, wherein the method further comprises compensating for global motion of the imaging device.

11. A device as in claim 9, wherein the method further comprises performing an action based on a location of the object tracked within the first reconstructed ROI.

12. A device as in claim 9, wherein the method further comprises determining a quality of reconstructed pixels within said first reconstructed ROI;

and selectively applying image analysis to said reconstructed ROI based on the quality of the reconstructed pixels.

13. A device as in claim 12, wherein the method further comprises responsive to said image analysis and said quality of reconstructed pixels, selectively applying image enhancement and generating an enhanced output image.

14. A device as in claim 9, wherein the method further comprises determining if any undetected faces below a size threshold are likely to exist in one or more reduced quality regions of the reconstructed ROI.

15. A device as in claim 14, wherein the method further comprises, responsive to said determining if any undetected faces exist, acquiring one or more additional initial distorted images, extracting and reconstructing one or more matching ROIs, and combining said one or more matching ROIs with reduced quality pixels of the original reconstructed ROI using a super-resolution technique, and generating, based on the combining, one or more enhanced ROIs.

16. A device as in claim 15, wherein the method further comprises applying additional face detecting or tracking, or both, to the one or more enhanced ROIs to confirm a presence and location of a face below an original size threshold.

17. One or more non-transitory, processor-readable storage media having code embedded therein for programming a processor to perform a method of enhancing a scene captured with a non-linear, wide-angled lens and containing one or more off-center peripheral regions, wherein the method comprises:

determining and extracting an off-center region of interest (hereinafter “

ROI”

) within an initial distorted image;

wherein the ROI includes a portion from said initial distorted image that has uncorrected distortion;

applying an initial object detection process to said off-center ROI that includes the portion from said initial distorted image having uncorrected distortion;

responsive to the initial object detection process successfully detecting an object within the off-center ROI;

applying geometric correction to remove distortion from the off-center ROI to produce a undistorted reconstructed ROI, without applying the geometric correction to said entire initial distorted image;

applying a second object detection process to one or more regions within the undistorted reconstructed ROI;

wherein the second object detection process is different than the initial object detection process.

18. One or more non-transitory, processor-readable storage media as in claim 17, wherein the method further comprises compensating for global motion of the imaging device.

19. One or more non-transitory, processor-readable storage media as in claim 17, wherein the method further comprises performing an action based on a location of the object tracked within the first reconstructed ROI.

20. One or more non-transitory, processor-readable storage media as in claim 17, wherein the method further comprises determining a quality of reconstructed pixels within said first reconstructed ROI;

and selectively applying image analysis to said reconstructed ROI based on the quality of the reconstructed pixels.

21. One or more non-transitory, processor-readable storage media as in claim 20, the method further comprises responsive to said image analysis and said quality of reconstructed pixels, selectively applying image enhancement and generating an enhanced output image.

22. One or more non-transitory, processor-readable storage media as in claim 17, wherein the method further comprises determining if any undetected faces below a size threshold are likely to exist in one or more reduced quality regions of the reconstructed ROI.

23. One or more non-transitory, processor-readable storage media as in claim 22, wherein the method further comprises, responsive to said determining if any undetected faces exist, acquiring one or more additional initial distorted images, extracting and reconstructing one or more matching ROIs, and combining said one or more matching ROIs with reduced quality pixels of the original reconstructed ROI using a super-resolution technique, and generating, based on the combining, one or more enhanced ROIs.

24. One or more non-transitory, processor-readable storage media as in claim 23, wherein the method further comprises applying additional face detecting or tracking, or both, to the one or more enhanced ROIs to confirm a presence and location of a face below an original size threshold.