Operating a device to capture high dynamic range images

• US 10,341,574 B2
• Filed: 07/30/2018
• Issued: 07/02/2019
• Est. Priority Date: 07/05/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Johnson, Garrett M., Orr, IV, James Edmund
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US16/049,412
• Publication Number:
  US 20180338078A1
• Time in Patent Office: 337 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Some embodiments provide a method of operating a device to capture an image of a high dynamic range (HDR) scene. Upon the device entering an HDR mode, the method captures and stores multiple images at a first image exposure level. Upon receiving a command to capture the HDR scene, the method captures a first image at a second image exposure level. The method selects a second image from the captured plurality of images. The method composites the first and second images to produce a composite image that captures the HDR scene. In some embodiments, the method captures multiple images at multiple different exposure levels.

Claims

1. A non-transitory machine readable medium of a device that captures images, the medium storing a program that when executed by at least one processing unit generates an image of a scene, the program comprising sets of instructions for:

capturing a plurality of images of the scene at different exposure levels, wherein the different exposure levels are selected based at least in part upon detected lighting conditions within the scene, comprising;

capturing and storing, prior to receiving input to generate the image, at least one image at a first exposure level;

detecting the lighting conditions within the scene based on an analysis of at least one of the at least one image captured and stored; and

capturing, upon receiving the input to generate the image, at least a different image at a second exposure level selected based at least in part upon the detected lighting conditions within the scene; and

compositing two or more of the plurality of images at different exposure levels to generate the image of the scene.

2. The non-transitory machine readable medium of claim 1, wherein the sets of instructions includes instructions for selecting an image from the at least one image at the first exposure level that was captured before receiving the input to generate the image.

3. The non-transitory machine readable medium of claim 1, wherein the sets of instructions further comprises instructions for selecting for compositing at least one image that was captured at the first exposure level while the device moved less than a threshold amount.

4. The non-transitory machine readable medium of claim 3, wherein the sets of instructions includes instructions for producing a plurality of different values related to motion of the device at a plurality of different times, wherein the threshold amount relates to the values produced.

5. The non-transitory machine readable medium of claim 3, wherein the device has a circuit that produces a plurality of different values related to motion of the device at a plurality of different times, wherein the program comprises sets of instructions for:

for each of a plurality of values produced by the circuit, computing a motion parameter that expresses the motion of the device at a corresponding time;

correlating each of the at least one image captured within the time period to a corresponding motion parameter;

selecting at least one of the at least one image captured within the time period based on a comparison of each motion parameter of each image captured within the time period, to the threshold amount.

6. The non-transitory machine readable medium of claim 1, wherein the sets of instructions for compositing two or more of the plurality of images comprises instructions for selecting a sharpest focus image of the at least one image, wherein the two or more of the plurality of images composited includes the sharpest focus image.

7. The non-transitory machine readable medium of claim 1, wherein the first exposure level is a normal exposure level, and the second exposure level is one of an underexposed level or an overexposed level.

8. A method comprising:

capturing a plurality of images of a scene at different exposure levels, wherein the different exposure levels are selected based at least in part upon detected lighting conditions within the scene, comprising;

capturing and storing, prior to receiving input to generate the image, at least one image at a first exposure level;

detecting the lighting conditions within the scene based on an analysis of at least one of the at least one image captured and stored; and

capturing, upon receiving the input to generate the image, at least a different image at a second exposure level selected based at least in part upon the detected lighting conditions within the scene; and

compositing two or more of the plurality of images at different exposure levels to generate the image of the scene.

9. The method of claim 8, further comprising selecting an image from the at least one image at the first exposure level that was captured before receiving the input to generate the image.

10. The method of claim 8, wherein compositing two or more of the plurality of images comprises selecting for compositing, at least one image that was captured at the first exposure level while the device moved less than a threshold amount.

11. The method of claim 10, further comprising producing a plurality of different values related to motion of the device at a plurality of different times, wherein the threshold amount relates to the values.

12. The method of claim 10, further comprising:

producing a plurality of different values related to motion of a device via which the images are captured, at a plurality of different times;

for each of a plurality of values produced, computing a motion parameter that expresses the motion of the device at a corresponding time;

correlating each of the at least one image captured within the time period to a corresponding motion parameter; and

selecting at least one of the at least one image captured within the time period based on a comparison of each motion parameter of each image captured within the time period, to the threshold amount.

13. The method of claim 8, wherein compositing two or more of the plurality of images comprises selecting a sharpest focus image of the at least one image, wherein the two or more of the plurality of images composited includes the sharpest focus image.

14. The method of claim 8, wherein the first exposure level is a normal exposure level, and the second exposure level is one of an underexposed level or an overexposed level.

15. A mobile device that captures images, the mobile device comprising:

a camera configured to capture a plurality of images of a scene at different exposure levels, wherein the different exposure levels are selected based at least in part upon detected lighting conditions within the scene, including the camera configured to;

capture and store, prior to receiving input to generate the image, at least one image at a first exposure level;

detect the lighting conditions within the scene based on an analysis of at least one of the at least one image captured and stored; and

capture, upon receiving the input to generate the image, at least a different image at a second exposure level selected based at least in part upon the detected lighting conditions within the scene; and

an image processing module configured to composite two or more of the plurality of images at different exposure levels to generate the image of the scene.

16. The mobile device of claim 15, wherein the mobile device is further configured to select an image from the at least one image at the first exposure level that was captured before receiving the input to generate the image.

17. The mobile device of claim 15, wherein compositing two or more of the plurality of images comprises selecting for compositing, at least one image that was captured at the first exposure level while the device moved less than a threshold amount.

18. The mobile device of claim 17, wherein the mobile device further comprises a circuit configured to produce a plurality of different values related to motion of the device at a plurality of different times, wherein the threshold amount relates to the values produced by the circuit.

19. The mobile device of claim 17, wherein the image processing module is further configured to:

produce a plurality of different values related to motion of a device via which the images are captured, at a plurality of different times;

for each of a plurality of values produced, compute a motion parameter that expresses the motion of the device at a corresponding time;

correlate each of the at least one image captured within the time period to a corresponding motion parameter; and

select at least one of the at least one image captured within the time period based on a comparison of each motion parameter of each image captured within the time period, to the threshold amount.

20. The mobile device of claim 15, wherein compositing two or more of the plurality of images comprises selection of a sharpest focus image of the at least one image, wherein the two or more of the plurality of images composited includes the sharpest focus image.