Methods and apparatus for compensating for motion and/or changing light conditions during image capture

• US 10,051,182 B2
• Filed: 10/05/2016
• Issued: 08/14/2018
• Est. Priority Date: 10/05/2015
• Status: Active Grant

• Current Assignee: Blue River Technology Inc. (Deere & Company)
• Original Assignee: Light Labs Inc.
• Inventors: Pulli, Kari, Shroff, Nitesh, Shroff, Sapna A.
• Primary Examiner(s): HANNETT, JAMES M
• Application Number:
  US15/286,517
• Publication Number:
  US 20170099439A1
• Time in Patent Office: 678 Days
• Field of Search: None
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2625 for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect sequence generated by event triggered capturing H04N7/188
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Methods and apparatus for compensating for motion and/or changing light conditions during image capture, e.g., in video, through use of multiple camera modules and/or images captured by multiple camera modules are described. During image capture time periods a plurality of camera modules capture images. During a first image capture time period a first camera module captures an image including a complete image of a user selected scene area of interest. During an additional image capture time period the first camera module captures an image including a portion of the scene area of interest; however, a portion of the scene area image is missing from the captured image, e.g., due to camera motion, occlusion and/or lighting conditions. Captured images from other camera modules and/or from during different image capture time periods which include the missing portion are identified and ranked; the highest ranked image is used in generating a composite image.

Claims

1. A method of generating one or more images, the method comprising:

receiving user input indicating a scene area of interest;

capturing using a first camera module, during a first image capture time period, an image including a complete image of said scene area of interest, said first camera module having a first focal length;

capturing multiple images using a set of camera modules during a second image capture time period, said set of camera modules including at least said first camera module and a second camera module, said first camera module capturing a first image including a first portion of said scene area of interest during said second image capture time period but missing a second portion of said scene area of interest during said second image capture time period, wherein said first camera module misses a portion of said scene area of interest during the second image capture time period due to a change in camera position between said first and second image capture time periods;

identifying, for use in generating a composite image, an image including said missing second portion of said scene area of interest, said identified image being;

(i) an image captured by said second camera module in said set of camera modules or (ii) an image captured during a different image capture time period than said second image capture time period; and

generating the composite image from the first portion of said scene area of interest included in said first image captured by said first camera module during the second image capture time period and the identified image including said missing second portion of said scene area of interest.

2. The method of claim 1, wherein said missing portion is an occluded portion of said scene area of interest which is occluded from a field of view of said first camera module during said second image capture time period but not during said first image capture time period.

3. A method of generating one or more images, the method comprising:

receiving user input indicating a scene area of interest;

capturing using a first camera module, during a first image capture time period, an image including a complete image of said scene area of interest, said first camera module having a first focal length;

capturing multiple images using a set of camera modules during a second image capture time period, said set of camera modules including at least said first camera module and a second camera module, said first camera module capturing a first image including a first portion of said scene area of interest during said second image capture time period but missing a second portion of said scene area of interest during said second image capture time period, wherein said first camera module misses a portion of said scene area of interest during said second image capture time period due to a change in lighting conditions between said first and second image capture time periods, said missing second portion of said scene area of interest corresponding to a portion of a first sensor which was saturated due to lighting levels higher during said second image capture time period than during said first image capture time period;

identifying, for use in generating a composite image, an image including said missing second portion of said scene area of interest, said identified image being;

(i) an image captured by said second camera module in said set of camera modules or (ii) an image captured during a different image capture time period than said second image capture time period; and

generating the composite image from the first portion of said scene area of interest included in said first image captured by said first camera module during the second image capture time period and the identified image including said missing second portion of said scene area of interest.

4. A method of generating one or more images, the method comprising:

capturing multiple images using a set of camera modules during a second image capture time period, said set of camera modules including at least a first camera module and a second camera module, said first camera module capturing a first image including a first portion of a scene area of interest during said second image capture time period but missing a second portion of said scene area of interest during said second image capture time period;

identifying, for use in generating a composite image, an image including said missing second portion of said scene area of interest, said identified image being;

(i) an image captured by said second camera module in said set of camera modules or (ii) an image captured during a different image capture time period than said second image capture time period, wherein identifying, for use in generating a composite image, said image including said missing second portion of said scene area of interest includes;

i) identifying one or more images including said second missing portion; and

ii) ranking the identified one or more images including said missing second portion of said scene area of interest based on at least one of resolution or time of image capture; and

iii) selecting, based on the ranking of the identified one or more images including said missing second portion, a highest ranked one of the identified one or more images including said second missing portion for use as the identified image for use in combining with the first image captured by the first camera module; and

generating the composite image from the first portion of said scene area of interest included in said first image captured by said first camera module during the second image capture time period and the identified image including said missing second portion of said scene area of interest.

5. The method of claim 4, wherein ranking the identified one or more images including the second missing portion includes weighting the identified one or more images on a per image basis based on the resolution of the image being ranked.

6. The method of claim 5, wherein ranking the identified one or more images including the second missing portion includes weighting images based on image resolution, higher resolution images being weighted more heavily than lower resolution images.

7. The method of claim 6, wherein ranking the identified one or more images including the second missing portion includes weighting images based on temporal proximity to the second image capture time period, images captured during the second image capture time period weighted more heavily than images captured during another image capture time period.

8. The method of claim 6, wherein ranking the identified one or more images includes weighting images based on temporal proximity to the second image capture time period, images captured during an image capture time period immediately preceding or subsequent to said second image capture time period being weighted more heavily than images captured in a time period two or more image capture time periods away from said second image capture time period.

9. The method of claim 8, wherein when an image corresponding to a different image capture time period and an image corresponding to the second image capture time period are identified as including the second missing scene portion, ranking the identified one or more images includes:

determining the resolution of the image captured during the second image capture time period including the second missing scene portion;

determining the resolution of the image captured during a different image capture time period including the second missing scene portion; and

when the resolution of the image captured during a different image capture time period including the second missing scene portion is higher than the resolution of the image captured during the second image capture time period including the second missing scene portion;

comparing the content of the second missing scene portion of the images including the missing second scene portion to determine if they match; and

if the lower resolution content of the second missing scene portion from the second image capture time period matches the content of the second missing scene portion of the higher resolution image captured in a different image capture time period, increasing the ranking of the higher resolution image captured in the different time period so that it is used instead of the lower resolution image captured in the second image capture time period.

10. The method of claim 4, wherein said image including the second missing portion selected for combining with the first image is an image captured by a camera module having smaller focal length than said first camera module, said selected image including the second missing portion being of lower resolution than said first image.

11. The method of claim 10, wherein generating a composite image includes combining the first portion of said scene area of interest included in the image captured by said first camera module during the second image capture time period and having a first resolution with the missing second portion of said scene area of interest included in the identified image including the second missing scene portion, the identified image including the missing second image portion being an image captured by the second camera module during the second image capture time period and having a second resolution which is different from the resolution of the image captured by the first camera module.

12. The method of claim 11, wherein the second camera module has a smaller focal length than said first camera module, said second resolution being lower than said first resolution.

13. A system comprising:

a plurality of optical chains;

an image capture controller configured to control a first optical chain in said plurality of optical chains to capture, during a first period of time, an image of a scene area of interest, the first optical chain having a first focal length, and to control a first set of optical chains in the plurality of optical chains to capture multiple images during a second time period, the first set of optical chains including said first optical chain, which captures a first portion of said scene area of interest during said second period of time but misses a second portion of said scene area of interest during said second period of time due to a change in camera position between said first and second periods of time, and at least a second optical chain, which captures a second image during said second period of time; and

a processor configured to;

identify at least one image including said missing second portion of said scene area of interest captured during said second period of time, said identified at least one image being an image captured during said second period of time by another optical chain in said first set of optical chains; and

generate a composite image from the first portion of said scene area of interest included in an image captured by said first optical chain during said second period of time and the missing second portion of said scene area of interest included in an image captured by said another optical chain during said second period of time.

14. The system of claim 13, wherein the second optical chain has a different focal length than said first optical chain.

15. The system of claim 14, further comprising:

a user input device for receiving user input indicating a scene area of interest; and

wherein the image capture controller is configured to control the first optical chain, during a first image capture time period to capture an image including a complete image of said scene area of interest, said first optical chain having a first focal length.

16. The system of claim 15,

wherein said user input device is a touch sensitive display;

wherein said first set of optical chains are camera modules which each include a separate sensor and lens; and

wherein said first set of optical chains includes optical chains of at least three different focal lengths.

17. The system of claim of claim 16, wherein said system is a handheld camera.

18. The system of claim 16, wherein said system includes:

a handheld camera that includes the plurality of optical chains and the image capture controller; and

a processing system coupled to said handheld camera, said processing system including said processor which is configured to generate the composite image.