Image capture device with contemporaneous image correction mechanism

• US 9,767,539 B2
• Filed: 10/24/2014
• Issued: 09/19/2017
• Est. Priority Date: 06/21/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Zamfir, Adrian, Bigioi, Petronel, Corcoran, Peter, Drimbarean, Alexandru, Steinberg, Eran
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Quiett, Carramah J
• Application Number:
  US14/523,816
• Publication Number:
  US 20150063692A1
• Time in Patent Office: 1,061 Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 2207/30216 Redeye defect
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 25/44 by partially reading an SSIS array
• CPC Class Codes: H04N 25/46 by combining or binning pixels
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/907 using static stores, e.g. storage tubes or semiconductor memories H04N5/91 takes precedence
• CPC Class Codes: H04N 9/8042 : involving data reduction

Abstract

A hand-held or otherwise portable or spatial or temporal performance-based image capture device includes one or more lenses, an aperture and a main sensor for capturing an original main image. A secondary sensor and optical system are for capturing a reference image that has temporal and spatial overlap with the original image. The device performs an image processing method including capturing the main image with the main sensor and the reference image with the secondary sensor, and utilizing information from the reference image to enhance the main image. The main and secondary sensors are contained together within a housing.

Claims

1. A computer-implemented method comprising:

receiving a plurality of images of a same scene captured by a camera;

wherein the plurality of images comprises a particular image of the same scene, a preview image of the same scene and a postview image of the same scene;

wherein the preview image, from the plurality of images, is an image acquired by the camera just shortly before the particular image was acquired by the camera;

wherein the postview image, from the plurality of images, is an image acquired by the camera just after the particular image was acquired by the camera;

wherein each of the preview image and the postview image has a resolution different than a resolution of the particular image;

wherein each of the preview image and the postview image depicts the same objects and subjects as those depicted in the particular image of the plurality of images;

determining whether the particular image, of the plurality of images, comprises a particular region that includes a defective depiction of one or more eyes;

in response to determining that the particular image comprises the particular region that includes the defective depiction of the one or more eyes;

selecting, either the preview image or the postview image that comprises a certain region;

wherein;

the certain region includes a non-defective depiction of the one or more eyes; and

the certain region corresponds to the particular region in the particular image; and

generating a combination image based, at least in part, on the particular image and the certain region by replacing the particular region in the particular image with the certain region;

wherein the method is performed by one or more processors of a computing device configured as an optical system.

2. The method of claim 1, wherein one or more images of the plurality of images are obtained using one or more infra-red sensors.

3. The method of claim 1, wherein the determining whether the particular image comprises the particular region that includes the defective depiction of the one or more eyes comprises determining whether the particular image comprises a region that includes a depiction of a blinking-eye, and if so, determining a degree to which the blinking-eye is shut.

4. The method of claim 3, wherein the degree to which the blinking-eye is shut is determined based on analyzing the particular image and one or more images, of the plurality of images, acquired within a certain time interval before or after the particular image was acquired.

5. The method of claim 3, wherein the determining whether the particular region comprises the depiction of the blinking-eye comprises determining a degree of blurriness of one or more eye lids depicted in the particular region.

6. The method of claim 1, further comprising:

determining whether any portion of a pupil, an iris, one or more eye lids, or an eye sclera is depicted in the particular region;

performing a shape analysis of shapes identified in the particular region;

performing a color analysis of pixels included in the particular region.

7. The method of claim 1, further comprising:

applying a set of sub-filters to the particular image to determine whether any pixel grouping in the particular image is compatible with one or more known depictions of eye-defects;

wherein the set of sub-filters comprise one or more of;

geometric filters, additional region compactness filters, boundary continuity filters, or region size filters; and

wherein the applying of the set of sub-filters to the particular image comprises determining one or more of;

an eye shape, eyebrows, iris regions, or a red-eye region.

8. An interactive computing device comprising a processor and a display, wherein the processor is arranged to perform:

receiving a plurality of images of a same scene captured by a camera;

wherein the plurality of images comprises a particular image of the same scene, a preview image of the same scene and a postview image of the same scene;

wherein the preview image, from the plurality of images, is an image acquired by the camera just shortly before the particular image was acquired by the camera;

wherein the postview image, from the plurality of images, is an image acquired by the camera just after the particular image was acquired by the camera;

wherein each of the preview image and the postview image has a resolution different than a resolution of the particular image;

wherein each of the preview image and the postview image depicts the same objects and subjects as those depicted in the particular image of the plurality of images;

determining whether the particular image, of the plurality of images, comprises a particular region that includes a defective depiction of one or more eyes;

in response to determining that the particular image comprises the particular region that includes the defective depiction of the one or more eyes;

selecting, either the preview image or the postview image that comprises a certain region;

wherein;

the certain region includes a non-defective depiction of the one or more eyes; and

the certain region corresponds to the particular region in the particular image; and

generating a combination image based, at least in part, on the particular image and the certain region by replacing the particular region in the particular image with the certain region.

9. The interactive computing device of claim 8, wherein one or more images of the plurality of images are obtained using one or more infra-red sensors.

10. The interactive computing device of claim 8, wherein the determining whether the particular image comprises the particular region that includes the defective depiction of the one or more eyes comprises determining whether the particular image comprises a region that includes a depiction of a blinking-eye, and if so, determining a degree to which the blinking-eye is shut.

11. The interactive computing device of claim 10, wherein the degree to which the blinking-eye is shut is determined based on analyzing the particular image and one or more images, of the plurality of images, acquired within a certain time interval before or after the particular image was acquired.

12. The interactive computing device of claim 10, wherein the determining whether the particular region comprises the depiction of the blinking-eye comprises determining a degree of blurriness of one or more eye lids depicted in the particular region.

13. The interactive computing device of claim 8, wherein the processor is further arranged to perform:

determining whether any portion of a pupil, an iris, one or more eye lids, or an eye sclera is depicted in the particular region;

performing a shape analysis of shapes identified in the particular region;

performing a color analysis of pixels included in the particular region.

14. The interactive computing device of claim 8, wherein the processor is further arranged to perform:

applying a set of sub-filters to the particular image to determine whether any pixel grouping in the particular image is compatible with one or more known depictions of eye-defects;

wherein the set of sub-filters comprise one or more of;

geometric filters, additional region compactness filters, boundary continuity filters, or region size filters; and

wherein the applying of the set of sub-filters to the particular image comprises determining one or more of;

an eye shape, eyebrows, iris regions, or a red-eye region.

15. A non-transitory, computer-readable storage medium storing one or more executable instructions which, when executed by one or more processors of an interactive computing device, cause performance of:

receiving a plurality of images of a same scene captured by a camera;

wherein the plurality of images comprises a particular image of the same scene, a preview image of the same scene and a postview image of the same scene;

wherein the preview image, from the plurality of images, is an image acquired by the camera just shortly before the particular image was acquired by the camera;

wherein the postview image, from the plurality of images, is an image acquired by the camera just after the particular image was acquired by the camera;

wherein each of the preview image and the postview image has a resolution different than a resolution of the particular image;

wherein each of the preview image and the postview image depicts the same objects and subjects as those depicted in the particular image of the plurality of images;

determining whether the particular image, of the plurality of images, comprises a particular region that includes a defective depiction of one or more eyes;

in response to determining that the particular image comprises the particular region that includes the defective depiction of the one or more eyes;

selecting either the preview image or the postview image that comprises a certain region;

wherein;

the certain region includes a non-defective depiction of the one or more eyes; and

the certain region corresponds to the particular region in the particular image; and

generating a combination image based, at least in part, on the particular image and the certain region by replacing the particular region in the particular image with the certain region.

16. The non-transitory, computer-readable storage medium of claim 15, wherein one or more images of the plurality of images are obtained using one or more infra-red sensors.

17. The non-transitory, computer-readable storage medium of claim 15, comprising additional instructions which, when executed by the one or more processors, cause performance of:

determining whether the particular image comprises a region that includes a depiction of a blinking-eye, and if so, determining a degree to which the blinking-eye is shut.

18. The non-transitory, computer-readable storage medium of claim 17, wherein the degree to which the blinking-eye is shut is determined based on analyzing the particular image and one or more images, of the plurality of images, acquired within a certain time interval before or after the particular image was acquired.

19. The non-transitory, computer-readable storage medium of claim 17, comprising additional instructions which, when executed by the one or more processors, cause performance of:

determining a degree of blurriness of one or more eye lids depicted in the particular region.

20. The non-transitory, computer-readable storage medium of claim 15, storing additional instructions which, when executed by the one or more processors, cause performance of:

determining whether any portion of a pupil, an iris, one or more eye lids, or an eye sclera is depicted in the particular region;

performing a shape analysis of shapes identified in the particular region;

performing a color analysis of pixels included in the particular region.

21. The non-transitory, computer-readable storage medium of claim 15, storing additional instructions which, when executed by the one or more processors, cause performance of:

applying a set of sub-filters to the particular image to determine whether any pixel grouping in the particular image is compatible with one or more known depictions of eye-defects;

wherein the set of sub-filters comprise one or more of;

geometric filters, additional region compactness filters, boundary continuity filters, or region size filters; and

wherein the applying of the set of sub-filters to the particular image comprises determining one or more of;

an eye shape, eyebrows, iris regions, or a red-eye region.