Operating a device to capture high dynamic range images

• US 10,038,855 B2
• Filed: 08/15/2016
• Issued: 07/31/2018
• Est. Priority Date: 07/05/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Johnson, Garrett M., Orr, IV, James Edmund
• Primary Examiner(s): Akhvannik, Hadi
• Application Number:
  US15/237,348
• Publication Number:
  US 20170048442A1
• Time in Patent Office: 715 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Some embodiments provide a method of operating a device to capture an image of a high dynamic range (HDR) scene. Upon the device entering an HDR mode, the method captures and stores multiple images at a first image exposure level. Upon receiving a command to capture the HDR scene, the method captures a first image at a second image exposure level. The method selects a second image from the captured plurality of images. The method composites the first and second images to produce a composite image that captures the HDR scene. In some embodiments, the method captures multiple images at multiple different exposure levels.

Claims

1. A non-transitory machine readable medium of a device that captures images, the medium storing a program that when executed by at least one processing unit generates a high dynamic range (HDR) image of a scene, the program comprising sets of instructions for:

capturing a plurality of images of the scene at different exposure levels, wherein the different exposure levels are selected based at least in part upon detected lighting conditions within the scene, including;

capturing and storing, prior to receiving a command to generate the HDR image of the scene, at least one image at a first exposure level;

detecting the lighting conditions within the scene based on an analysis of at least one of the at least one image captured and stored; and

capturing, upon receiving the command to generate the HDR image of the scene, at least a first image at a second exposure level selected based at least in part upon the detected lighting conditions within the scene; and

compositing the plurality of images to generate the HDR image of the scene.

2. The non-transitory machine readable medium of claim 1, wherein the sets of instructions for capturing a plurality of images of the scene at different exposure levels comprises sets of instructions for:

capturing and storing, upon the device entering an HDR mode, a first plurality of images at the first exposure level; and

capturing, upon receiving the command to generate the HDR image of the scene, at least one image at one or more different exposure levels, wherein the one or more different exposure levels are selected based at least in part upon the detected lighting conditions within the scene.

3. The non-transitory machine readable medium of claim 2, wherein the program further comprises sets of instructions for:

computing the first exposure level based on the detected lighting conditions within the scene; and

computing the one or more different exposure levels based on the first exposure level.

4. The non-transitory machine readable medium of claim 2, wherein the set of instructions for capturing and storing, upon the device entering the HDR mode, the first plurality of images at the first exposure level comprises a set of instructions for analyzing at least one of the first plurality of images to detect the lighting conditions within the scene.

5. The non-transitory machine readable medium of claim 2, wherein:

capturing at least one image at one or more different exposure levels comprises;

capturing at least the first image at the second exposure level when the device is determined to be operating in a first set of lighting conditions, where the determination is based at least in part upon analyzing at least one image of the captured first plurality of images; and

capturing at least a second image at the same first exposure level when the device is determined to be operating in low light conditions, where the determination is based at least in part upon analyzing at least one image of the captured first plurality of images; and

compositing the plurality of images to generate the HDR image of the scene comprises;

compositing at least the first image and a third image from the captured first plurality of images, when the device is determined to be operating in the first set of lighting conditions, in order to generate a first composite HDR image; and

compositing at least the second image and the third image when the device is determined to be operating in low light conditions in order to generate a second composite HDR image with reduced noise.

6. The non-transitory machine readable medium of claim 1, wherein the sets of instructions for compositing the plurality of images to generate the HDR image of the scene comprises sets of instructions for:

compositing at least one particular image selected from a plurality of images captured at a particular exposure level of the different exposure levels.

7. The non-transitory machine readable medium of claim 6, wherein the set of instructions for selecting the particular image from the plurality of images comprises a set of instructions for selecting a sharpest image of the plurality of images.

8. A method of operating a device to generate a high dynamic range (HDR) image of a scene, the method comprising:

capturing a plurality of images of the scene at a plurality of selected exposure levels, wherein the exposure levels are selected based at least in part upon detected lighting conditions within the scene, including;

capturing and storing, prior to receiving a command to generate the HDR image of the scene, at least one image at a first exposure level;

detecting the lighting conditions within the scene based on an analysis of at least one of the at least one image captured and stored; and

capturing, upon receiving the command to generate the HDR image of the scene, at least a first image at a second exposure level selected based at least in part upon the detected lighting conditions within the scene; and

compositing the plurality of images to generate the HDR image of the scene.

9. The method of claim 8, wherein capturing a plurality of images of the scene at different exposure levels comprises:

capturing and storing, upon the device entering an HDR mode, a first plurality of images at the first exposure level; and

capturing, upon receiving the command to generate the HDR image of the scene, at least one image at one or more different exposure levels, wherein the one or more different exposure levels are selected based at least in part upon the detected lighting conditions within the scene.

10. The method of claim 9, comprising:

computing the first exposure level based on the detected lighting conditions within the scene; and

computing the one or more different exposure levels based on the first exposure level.

11. The method of claim 9, wherein capturing and storing, upon the device entering the HDR mode, the first plurality of images at the first exposure level comprises analyzing at least one of the first plurality of images to detect the lighting conditions within the scene.

12. The method of claim 9, wherein:

capturing at least one image at one or more different exposure levels comprises;

capturing at least the first image at the second exposure level when the device is determined to be operating in a first set of lighting conditions, where the determination is based at least in part upon analyzing at least one image of the captured first plurality of images; and

capturing at least a second image at the same first exposure level when the device is determined to be operating in low light conditions, where the determination is based at least in part upon analyzing at least one image of the captured first plurality of images; and

compositing the plurality of images to generate the HDR image of the scene comprises;

compositing at least the first image and a third image from the captured first plurality of images, when the device is determined to be operating in the first set of lighting conditions, in order to generate a first composite HDR image; and

compositing at least the second image and the third image when the device is determined to be operating in low light conditions in order to generate a second composite HDR image with reduced noise.

13. The method of claim 8, wherein the sets of instructions for compositing the plurality of images to generate the HDR image of the scene comprises sets of instructions for:

compositing at least one particular image selected from a particular plurality of images captured at a particular exposure level of the different exposure levels.

14. The method of claim 13, wherein selecting the particular image from the particular plurality of images comprises selecting a sharpest image of the particular plurality of images.

15. A mobile device that captures images, the device comprising:

a camera configured to capture a plurality of images of a scene at different image exposure durations, wherein the different image exposure durations are selected based at least in part upon detected lighting conditions of the scene, including the camera configured to;

capture and store, prior to receiving a command to generate a high dynamic range (HDR) image of the scene, at least one image at a first exposure duration;

detect the lighting conditions within the scene based on an analysis of at least one of the at least one image captured and stored; and

capture, upon receiving the command to generate the HDR image of the scene, at least a first image at a second exposure duration selected based at least in part upon the detected lighting conditions within the scene; and

an image processing module configured to composite the plurality of captured images to generate the HDR image of the scene.

16. The mobile device of claim 15, wherein to capture a plurality of images of a scene at different image exposure durations, the camera is configured to:

capture, upon the mobile device entering an HDR mode, a plurality of images at the first exposure duration; and

capture, upon receiving the command to generate the HDR image of the scene, at least one image at one or more different exposure durations, wherein the one or more different exposure durations are selected based at least in part upon the detected lighting conditions of the scene.

17. The mobile device of claim 16, comprising an image capture module configured to:

compute the first exposure duration based on the detected lighting conditions of the scene; and

compute the one or more different exposure durations based on the first exposure level.

18. The mobile device of claim 17, wherein, to compute the first exposure duration based on the detected lighting conditions within the scene, the image capture module is configured to analyze at least one of the plurality of images to detect the lighting conditions of the scene.

19. The mobile device of claim 15, wherein to composite the plurality of captured images to generate the HDR image of the scene, the image processing module is configured to:

composite at least one particular image selected from a particular plurality of images captured at a particular exposure duration of the one or more different exposure durations.

20. The mobile device of claim 19, wherein, to composite at least one particular image selected from a particular plurality of images captured at a particular exposure duration, the image processing module is configured to select a sharpest image of the particular plurality of images.