Scene enhancements in off-center peripheral regions for nonlinear lens geometries

• US 8,947,501 B2
• Filed: 03/31/2011
• Issued: 02/03/2015
• Est. Priority Date: 03/31/2011
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Corcoran, Peter, Bigioi, Petronel, Stec, Piotr
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Le, Peter D
• Application Number:
  US13/077,936
• Publication Number:
  US 20120250937A1
• Time in Patent Office: 1,405 Days
• Field of Search: 348/36
• US Class Current: 348/36
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation

Abstract

A technique of enhancing a scene containing one or more off-center peripheral regions within an initial distorted image captured with a large field of view includes determining and extracting an off-center region of interest (hereinafter “ROI”) within the image. Geometric correction is applied to reconstruct the off-center ROI into a rectangular frame of reference as a reconstructed ROI. A quality of reconstructed pixels is determined within the reconstructed ROI. Image analysis is selectively applied to the reconstructed ROI based on the quality of the reconstructed pixels.

Claims

1. Within an image acquisition system comprising a non-linear, wide-angled lens and an imaging sensor, a method of enhancing a scene containing one or more off-center peripheral regions, the method comprising:

acquiring, using a nonlinear, wide-angled lens and an imaging sensor, an initial distorted image with a large field of view;

generating a reconstructed image from the initial distorted image, wherein the reconstructed image includes a plurality of reconstructed regions;

wherein each reconstructed region of the plurality of reconstructed regions has a corresponding region quality;

wherein the region quality of each reconstructed region of the plurality of reconstructed regions is based on reconstruction factors of reconstructed pixels within the reconstructed region;

wherein the reconstruction factor for a reconstructed pixel reflects how much data, from the image sensor, the reconstructed pixel was extrapolated based on;

based on the region qualities of said plurality of reconstructed regions, determining one or more reduced quality regions within said plurality of reconstructed regions;

determining whether any undetected faces below a size threshold are likely to exist in the one or more reduced quality regions; and

in response to determining that any undetected faces below the size threshold are likely to exist in the one or more reduced quality regions, performing the steps of;

generating one or more enhanced regions of interest that correspond to the one or more reduced quality regions; and

applying face detecting or tracking, or both, to the one or more enhanced regions of interest to confirm presence of a face below the size threshold.

2. A method as in claim 1, further comprising compensating for global motion of the image acquisition system.

3. A method as in claim 1, wherein the generating the one or more enhanced regions of interest comprises:

acquiring one or more additional initial distorted images;

extracting and reconstructing one more matching regions of interest;

combining said one or more matching regions of interest with reduced quality pixels of the one or more reconstructed regions using a super-resolution technique; and

based on the combining of said one or more matching regions of interest with reduced quality pixels of the one or more reconstructed regions, generating the one or more enhanced regions of interest.

4. A method as in claim 1, further comprising, prior to applying a geometric correction, applying an initial face detection process to one or more off-center regions.

5. A method as in claim 4, wherein said applying the geometric correction is performed in response to an initial determination that a face exists within at least one of the one or more off-center regions.

6. A method as in claim 5, wherein said face detecting or tracking, or both, is applied to the one or more reconstructed regions to refine or confirm, or both, the initial face detection process.

7. A digital image acquisition device, comprising:

a non-linear, wide-angled lens and an imaging sensor configured to capture digital images of scenes containing one or more off-center peripheral regions, including an initial distorted image with a large field of view;

a processor;

a memory having code embedded therein for programming the processor to perform a method of enhancing a scene containing one or more off-center peripheral regions, wherein the method comprises;

generating a reconstructed image from the initial distorted image, wherein the reconstructed image includes a plurality of reconstructed regions;

wherein each reconstructed region of the plurality of reconstructed regions has a corresponding region quality;

wherein the region quality of each reconstructed region of the plurality of reconstructed regions is based on reconstruction factors of reconstructed pixels within the reconstructed region;

wherein the reconstruction factor for a reconstructed pixel reflects how much data, from the image sensor, the reconstructed pixel was extrapolated based on;

based on the region qualities of said plurality of reconstructed regions, determining one or more reduced quality regions within said plurality of reconstructed regions;

determining whether any undetected faces below a size threshold are likely to exist in the one or more reduced quality regions; and

in response to determining that any undetected faces below the size threshold are likely to exist in the one or more reduced quality regions, performing the steps of;

generating one or more enhanced regions of interest that correspond to the one or more reduced quality regions; and

applying face detecting or tracking, or both, to the one or more enhanced regions of interest to confirm presence of a face below the size threshold.

8. A device as in claim 7, wherein the method further comprises compensating for global motion of the digital image acquisition device.

9. A device as in claim 7, wherein the generating the one or more enhanced regions of interest comprises:

acquiring one or more additional initial distorted images;

extracting and reconstructing one more matching regions of interest;

combining said one or more matching regions of interest with reduced quality pixels of the one or more reconstructed regions using a super-resolution technique; and

based on the combining of said one or more matching regions of interest with reduced quality pixels of the one or more reconstructed regions, generating the one or more enhanced regions of interest.

10. A device as in claim 7, wherein the method further comprises, prior to applying a geometric correction, applying an initial face detection process to one or more off-center regions.

11. A device as in claim 10, wherein said applying the geometric correction is performed in response to an initial determination that a face exists within at least one of the off-center regions.

12. A device as in claim 11, wherein said face detecting or tracking, or both, is applied to the one or more reconstructed region to refine or confirm, or both, the initial face detection process.

13. One or more non-transitory, processor-readable storage media having code embedded therein for programming a processor to perform a method of enhancing a scene captured with a non-linear, wide-angled lens and containing one or more off-center peripheral regions, wherein the method comprises:

acquiring using a nonlinear, wide-angled lens and an imaging sensor, an initial distorted image with a large field of view;

generating a reconstructed image from the initial distorted image, wherein the reconstructed image includes a plurality of reconstructed regions;

wherein each reconstructed region of the plurality of reconstructed regions has a corresponding region quality;

wherein the region quality of each reconstructed region of the plurality of reconstructed regions is based on reconstruction factors of reconstructed pixels within the reconstructed region;

wherein the reconstruction factor for a reconstructed pixel reflects how much data, from the image sensor, the reconstructed pixel was extrapolated based on;

based on the region qualities of said plurality of reconstructed regions, determining one or more reduced quality regions within said plurality of reconstructed regions;

determining whether any undetected faces below a size threshold are likely to exist in the one or more reduced quality regions; and

in response to determining that any undetected faces below the size threshold are likely to exist in the one or more reduced quality regions, performing the steps of;

generating one or more enhanced regions of interest that correspond to the one or more reduced quality regions; and

applying face detecting or tracking, or both, to the one or more enhanced regions of interest to confirm presence of a face below the size threshold.

14. One or more non-transitory, processor-readable storage media as in claim 13, further comprising compensating for global motion of an imaging device.

15. One or more non-transitory, processor-readable storage media as in claim 13, wherein the generating the one or more enhanced regions of interest comprises:

acquiring one or more additional initial distorted images;

extracting and reconstructing one more matching regions of interest;

combining said one or more matching regions of interest with reduced quality pixels of the one or more reconstructed regions using a super-resolution technique; and

based on the combining of said one or more matching regions of interest with reduced quality pixels of the one or more reconstructed regions, generating the one or more enhanced regions of interest.

16. One or more non-transitory, processor-readable storage media as in claim 13, further comprising, prior to applying a geometric correction, applying an initial face detection process to one or more off-center regions.

17. One or more non-transitory, processor-readable storage media as in claim 16, wherein said applying the geometric correction is performed in response to an initial determination that a face exists within at least one of the one or more off-center regions.

18. One or more non-transitory, processor-readable storage media as in claim 17, wherein said face detecting or tracking, or both, is applied to the one or more reconstructed regions to refine or confirm, or both, the initial face detection process.