Methods and apparatus for using a camera device to support multiple modes of operation

• US 9,912,864 B2
• Filed: 10/17/2015
• Issued: 03/06/2018
• Est. Priority Date: 10/17/2014
• Status: Active Grant

• Current Assignee: Blue River Technology Inc. (Deere & Company)
• Original Assignee: Light Labs Inc.
• Inventors: Laroia, Rajiv, Shroff, Nitesh, Shroff, Sapna A.
• Primary Examiner(s): Monk, Mark T
• Application Number:
  US14/886,049
• Publication Number:
  US 20160112637A1
• Time in Patent Office: 871 Days
• Field of Search: None
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 5/265 Mixing

Abstract

In various embodiments a camera with multiple optical chains, e.g., camera modules, is controlled to operate in one of a variety of supported modes of operation. The modes include a non-motion mode, a motion mode, a normal burst mode and/or a reduced data burst mode. Motion mode is well suited for capturing an image including motion, e.g., moving object(s) with some modules being used to capture scene areas using a shorter exposure time than other modules and the captured images then being combined taking into consideration locations of motion. A reduced data burst mode is supported in some embodiments in which camera modules with different focal lengths capture images at different rates. While the camera modules of different focal length operate at different image capture rates in the reduced data burst mode, images are combined to support a desired composite image output rate, e.g., a desired frame rate.

Claims

1. A method, the method comprising:

operating a first optical chain of a camera device including a plurality of optical chains to capture a first image of a first scene area using a first exposure time, said first optical chain having a first focal length;

operating a second optical chain of the camera device to capture a second image of a second scene area using a second exposure time which is shorter than said first exposure time, said second optical chain having a second focal length which is longer than said first focal length, a time in which said second image is captured overlapping a time during which said first image is captured; and

generating a composite image from said first and second images, said step of generating a composite image including performing a weighted combination of pixel values from said first and second images for at least a first portion of said first scene area which is overlapped by said second scene area as a function of whether the first portion of the scene area is an area of motion, said step of performing a weighted combination of pixel values including weighting a first pixel value of the second image corresponding to a first scene area which is an area of motion more heavily than a first pixel value in said first image corresponding to the first scene area which is the area of motion and summing the first weighted pixel value of the first image with the first weighted pixel value of the second image to generate a first pixel value of the composite image corresponding to said first scene area.

2. The method of claim 1, further comprising:

controlling the exposure time of the second optical chain with the second focal length which is longer than the first focal length as a function of motion.

3. The method of claim 1, wherein said first and second images are captured during a motion mode of operation, the method further comprising:

controlling the exposure time of the first optical chain without regard to motion while the exposure time of the second optical chain is controlled as a function of motion.

4. The method of claim 3, wherein the first image captured by the first camera module includes a portion corresponding to an area of motion.

5. The method of claim 1, wherein said weighted combination of pixel values weights a pixel value in said first image more heavily than a pixel value in said second image when the pixels of said first and second images which are being combined correspond to an area where there is no motion.

6. The method of claim 1, wherein said weighting is a function of an amount of motion in the area to which pixel values being combined correspond.

7. The method of claim 6, further comprising:

estimating an amount of motion based on analysis of at least one of said first and second images.

8. The method of claim 7, wherein estimating the amount of motion includes determining a level of blurriness in said first image.

9. The method of claim 7, wherein estimating the amount of motion includes comparing a portion of said first and second images to determine an estimate of how much said first and second images differ from one another in the area being compared.

10. The method claim 1, further comprising:

operating a third optical chain to capture a third image of a third scene area using said first exposure time, said third scene area corresponding to a scene area with no motion or little motion.

11. The method of claim 10, wherein said step of generating a composite image further includes using said third image to generate said composite image, said second and third images corresponding to different portions of a scene area to which the first image corresponds.

12. The method of claim 11, wherein said second and third optical chains have longer focal lengths than said first optical chain.

13. The camera device of claim 3, wherein said processor is configured, as part of performing the weighted combination of pixel values, to weight a pixel value in said second image more heavily than a pixel value in said first image when the pixels of said first and second images which are being combined correspond to an area of motion, the heavier the weight of a pixel value the greater the contribution of the pixel value to a combined pixel value generated by said weighted combination of pixel values.

17. The camera device of claim 13, wherein said weighting is a function of an amount of motion in the area to which pixel values being combined correspond.

18. The camera device of claim 17, wherein said processor is further configured to estimate an amount of motion based on analysis of at least one of said first and second images.

14. A camera device comprising:

a plurality of optical chains including a first optical chain and a second optical chain; and

a processor configured to;

operate the first optical chain of the camera device including a plurality of optical chains to capture a first image of a first scene area using a first exposure time, said first optical chain having a first focal length;

operate the second optical chain of the camera device to capture a second image of a second scene area using a second exposure time which is shorter than said first exposure time, said second optical chain having a second focal length which is longer than said first focal length, a time in which said second image is captured overlapping a time during which said first image is captured; and

generate a composite image from said first and second images, said step of generating a composite image including performing a weighted combination of pixel values from said first and second images for at least a first portion of said first scene area which is overlapped by said second scene area as a function of whether the first portion of the scene area is an area of motion, said step of performing a weighted combination of pixel values including weighting a first pixel value of the second image corresponding to a first scene area which is an area of motion more heavily than a first pixel value in said first image corresponding to the first scene area which is the area of motion and summing the first weighted pixel value of the first image with the first weighted pixel value of the second image to generate a first pixel value of the composite image corresponding to said first scene area.

15. The camera device of claim 14, wherein said second optical chain has a larger focal length than said first optical chain;

and

wherein said second scene area overlaps a portion of said first scene area.

16. The camera device of claim 15, wherein said first and second images are captured during a motion mode of camera operation;

and

wherein said processor is further configured to generate a composite image from said first and second images, said processor being configured to perform a weighted combination of pixel values from said first and second images for a portion of said first scene area which is overlapped by said second scene area, as part of being configured to generate the composite image.

19. The camera device of claim 14, further comprising:

a third optical chain; and

wherein said processor is further configured to control said third optical chain to capture a third image of a third scene area using said first exposure time, said third scene area corresponding to a scene area with no motion or little motion.

20. A non-transitory computer readable medium including computer executable instructions which when executed by a processor control a camera device to:

operate a first optical chain of the camera device including a plurality of optical chains to capture a first image of a first scene area using a first exposure time, said first optical chain having a first focal length;

operate a second optical chain of the camera device to capture a second image of a second scene area using a second exposure time which is shorter than said first exposure time, said second optical chain having a second focal length which is longer than said first focal length, a time in which said second image is captured overlapping a time during which said first image is captured; and

generate a composite image from said first and second images, said step of generating a composite image including performing a weighted combination of pixel values from said first and second images for at least a first portion of said first scene area which is overlapped by said second scene area as a function of whether the first portion of the scene area is an area of motion, said step of performing a weighted combination of pixel values including weighting a first pixel value of the second image corresponding to a first scene area which is an area of motion more heavily than a first pixel value in said first image corresponding to the first scene area which is the area of motion and summing the first weighted pixel value of the first image with the first weighted pixel value of the second image to generate a first pixel value of the composite image corresponding to said first scene area.