Operating a device to capture high dynamic range images

• US 8,885,978 B2
• Filed: 09/03/2010
• Issued: 11/11/2014
• Est. Priority Date: 07/05/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Johnson, Garrett M., Orr, James Edmund IV
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): HEIDEMANN, JASON E
• Application Number:
  US12/876,100
• Publication Number:
  US 20120002898A1
• Time in Patent Office: 1,530 Days
• Field of Search: None
• US Class Current: 382/284
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Some embodiments provide a method of operating a device to capture an image of a high dynamic range (HDR) scene. Upon the device entering an HDR mode, the method captures and stores multiple images at a first image exposure level. Upon receiving a command to capture the HDR scene, the method captures a first image at a second image exposure level. The method selects a second image from the captured plurality of images. The method composites the first and second images to produce a composite image that captures the HDR scene. In some embodiments, the method captures multiple images at multiple different exposure levels.

Claims

1. A non-transitory machine readable medium of a device that captures images, the medium storing a program that when executed by at least one processing unit captures an image of a high dynamic range (HDR) scene, the program comprising sets of instructions for:

capturing lighting conditions of the HDR scene;

capturing and storing, upon the device entering an HDR mode, a plurality of images at a first exposure level;

capturing, upon receiving a command to capture the HDR scene, a first image at a second exposure level and a second image at a third exposure level;

selecting a third image from the captured plurality of images at the first exposure level; and

compositing the first, second, and third images to produce a composite image of the HDR scene;

wherein the first exposure level, second exposure level, and third exposure level are based on the captured lighting conditions.

2. The machine readable medium of claim 1, wherein the set of instructions for selecting the third image from the plurality of images comprises a set of instructions for selecting an image in the plurality of images that was captured immediately before receiving the command to capture the HDR scene.

3. The machine readable medium of claim 1, wherein the set of instructions for selecting the third image from the plurality of images comprises a set of instructions for selecting an image in the plurality of images that (i) was captured within a time period before receiving the command to capture the HDR scene, and (ii) was captured while the device moved less than a threshold amount.

4. The machine readable medium of claim 3, wherein the device has a circuit that produces a plurality of different values related to motion of the device at a plurality of different instances in time, wherein the threshold amount relates to the values produced by the circuit.

5. The machine readable medium of claim 3, wherein the device has a circuit that produces a plurality of different values related to motion of the device at a plurality of different instances in time, wherein the set of instructions for selecting the image comprises sets of instructions for:

computing, for each of the plurality of values produced by the circuit, a motion parameter that expresses the motion of the device at an instance in time;

correlating each of the Images captured within the time period to a motion parameter; and

selecting one image captured within the time period as the third image based on a comparison of the motion parameter of each image captured within the time period to the threshold amount.

6. The machine readable medium of claim 1, wherein the set of instructions for selecting the third image from the plurality of images comprises a set of instructions for selecting a sharpest image of the plurality of images.

7. The machine readable medium of claim 6, wherein the device has a circuit that produces a plurality of different values related to the motion of the device at a plurality of different instances in time, wherein the sharpest image from the plurality of images is the image that was captured at an instance when the value produced by the circuit at that instance indicated that the device moved the least.

8. The machine readable medium of claim 6, wherein the program further comprises a set of instructions for performing digital signal processing on each of a group of images captured and stored at the first exposure level, wherein the sharpest image from the plurality of images is an image with highest frequency content.

9. The machine readable medium of claim 1, wherein the first exposure level is a normal exposure level, wherein one of the second and third exposure levels is an underexposed level, wherein the other of the second and third exposure levels is an overexposed level.

10. The machine readable medium of claim 1, wherein the first exposure level is an overexposed level, wherein one of the second and third exposure levels is a normal exposure level, wherein the other of the second and third exposure level is an underexposed level.

11. The machine readable medium of claim 1, wherein the first exposure level is an underexposed level, wherein one of the second and third exposure levels is a normal exposure level, wherein the other of the second and third exposure level is an overexposed level.

12. The machine readable medium of claim 1, wherein the program further comprises sets of instructions for:

computing the first exposure level from the captured lighting conditions; and

computing the second and third exposure levels from the first exposure level.

13. The machine readable medium of claim 12, wherein the second exposure level is a first multiple of the first exposure level and the third exposure level is a second multiple of the first exposure level.

14. The machine readable medium of claim 13, wherein the program further comprises a set of instructions for computing the first multiple and the second multiple based on at least one histogram of at least one of the plurality of images.

15. The machine readable medium of claim 12, wherein the set of instructions for capturing the lighting conditions comprises a set of instructions for analyzing at least one of the plurality of images to identify the lighting conditions.

16. The machine readable medium of claim 12, wherein the set of instructions for capturing the lighting conditions comprises a set of instructions for analyzing a signal produced by a circuit of the device that quantifies the lighting conditions.

17. The machine readable medium of claim 1, wherein the set of instructions for capturing and storing the plurality of images comprises sets of instructions for:

storing each of the plurality of images in a storage of a device before receiving the HDR capture command;

computing the second and third exposure levels; and

notifying a controller of the device to enter a mode to capture, upon receiving the HDR capture command, second and third images at the second and third exposure levels.

18. A non-transitory machine readable medium of a device that captures images, the medium storing a program that when executed by at least one processing unit captures an image of a high dynamic range (HDR) scene, the program comprising sets of instructions for:

capturing, upon entering an HDR mode, a plurality of images at a first exposure level and analyzing the images to detect lighting conditions within the scene; and

upon receiving a command to capture the HDR scene;

capturing only a first image at a second exposure level when a first set of lighting conditions is detected;

capturing the first image at the second exposure level and capturing a second image at a third exposure level when a second set of lighting conditions is detected, wherein the second set of lighting conditions is different from the first set of lighting conditions;

selecting a third image from the captured plurality of images at the first exposure level;

compositing the first and third images when the first set of lighting conditions is detected in order to produce a first composite image of the HDR scene; and

compositing the first, second, and third images when the second set of lighting conditions is detected in order to produce a second composite image of the HDR scene.

19. The machine readable medium of claim 18, wherein the first exposure level is a normal exposure level, wherein one of the second and third exposure levels is an underexposed level, wherein the other of the second and third exposure levels is an overexposed level.

20. The machine readable medium of claim 18, wherein the first exposure level is an overexposed level, wherein one of the second and third exposure levels is a normal exposure level, wherein the other of the second and third exposure levels is an underexposed level.

21. The machine readable medium of claim 18, wherein the first exposure level is an underexposed level, wherein one of the second and third exposure levels is a normal exposure level, wherein the other of the second and third exposure levels is an overexposed level.

22. The machine readable medium of claim 18, wherein the set of instructions for analyzing the plurality of images comprises a set of instructions for computing at least one exposure level from the detected lighting conditions.

23. The machine readable medium of claim 22, wherein the computed exposure level is the first exposure level, wherein the program further comprises a set of instructions for computing the second and third exposure levels from the first exposure level.

24. The machine readable medium of claim 23, wherein the first image exposure level is a first multiple of the third image exposure level and the second image exposure level is a second multiple of the third image exposure level.

25. The machine readable medium of claim 24, wherein the program further comprises a set of instructions for computing the first multiple and the second multiple based on at least one histogram of at least one of the plurality of images.

26. The machine readable medium of claim 18, wherein the set of instructions for analyzing the images to detect the lighting conditions comprises a set of instructions for analyzing a signal produced by a circuit of the device that quantifies the lighting conditions.

27. The machine readable medium of claim 18, wherein the set of instructions for capturing the plurality of images comprises sets of instructions for:

computing the first, second and third exposure levels; and

directing a controller of the device to enter a mode to capture at least one image at at least one of the first, second and third exposure levels upon receipt of the HDR capture command.

31. The machine readable medium of claim 27, wherein the set of instructions for illuminating the scene with the flash light source comprises a set of instructions for activating the flash light source only when the first exposure duration is longer than the threshold duration.

28. A non-transitory machine readable medium of a device that captures images, the medium storing program that when executed by at least one processing unit captures an image of a scene that is dark, the program comprising sets of instructions for:

in response to receiving a command to capture an image;

capturing a first image at a first exposure duration;

illuminating, when the first exposure duration is longer than a particular threshold duration, the scene with a flash light source contemporaneously with capturing a second image at a second exposure duration that is longer than the first exposure duration;

capturing, when the first exposure duration is less than the particular threshold duration, the second image at the second exposure duration without using the flash light source; and

compositing the first and second images to generate a third composite image.

30. The machine readable medium of claim 28, wherein the second exposure duration is calculated using the first exposure duration.

32. The machine readable medium of claim 28, wherein the threshold duration is a particular fraction of a maximum exposure duration of the device.

33. The machine readable medium of claim 28, wherein the program further comprises sets of instructions for:

setting the second exposure duration to a multiple of the first exposure duration; and

reducing the multiple when the first exposure duration is longer than the threshold duration.

34. The machine readable medium of claim 28, wherein the set of instructions for compositing the images comprises a set of instructions for weighting the first and second images, wherein the second image is weighted less than the first image.

35. The machine readable medium of claim 28, wherein the first image is captured without using any flash light source.

29. The machine readable medium 28, wherein the third composite image has a higher dynamic range than the second image that was captured with the flash light source illuminated.

36. A method of operating a device to capture an image of a high dynamic range (HDR) scene, the method comprising:

capturing lighting conditions of the HDR scene;

upon the device entering an HDR mode, capturing and storing a plurality of images at a first exposure level;

upon receiving a command to capture the HDR scene, capturing a first image at a second exposure level;

selecting a second image from the captured plurality of images at the first exposure level; and

compositing the first and second images to produce a composite image of the HDR scene;

wherein the first exposure level and second exposure level are based on the captured lighting conditions.

37. The method of claim 36, wherein selecting the second image from the plurality of images comprises selecting the second image from the plurality of images based on a set of criteria.

38. The method of claim 36 further comprising:

analyzing at least one of the images of the captured plurality of images to determine lighting conditions within the scene; and

upon receiving a command to capture the HDR scene, capturing one of;

(i) only a first image when a first set of lighting conditions is determined to exist, and (ii) the first image and a third image at a third exposure level when a second set of lighting conditions is determined to exist, wherein

compositing the first and second images comprises compositing the first, second and third images to produce the composite image when the second set of lighting conditions exists.

39. A method of operating a device to capture an image of a high dynamic range (HDR) scene in low light conditions, the method comprising:

upon entering an HDR mode, capturing and storing a plurality of images at a first exposure level; and

upon receiving a command to capture an HDR scene;

capturing at least a first image at a second exposure when the device is determined to be operating in a first set of lighting conditions, where the determination is based at least in part upon analyzing at least one of the images in the captured plurality of images;

capturing a second image at the same first exposure level when the device is determined to be operating in low light conditions, where the determination is based at least in part upon analyzing at least one of the images in the captured plurality of images;

compositing at least the first image and a third image from the captured plurality of images that capture the same scene at the first exposure level, when the device is determined to be operating in the first set of lighting conditions, in order to produce a first composite HDR image; and

compositing the second image and the third image when the device is determined to be operating in low light conditions in order to produce a second composite HDR image with reduced noise.

40. The method of claim 39, wherein the second image is selected from the plurality of images based on a set of criteria.

41. The method of claim 39 further comprising capturing, upon receiving the command to capture an image, a third image at a second exposure level that is different from the first exposure level, wherein compositing the first and second images further comprises compositing the first, second and third images to produce the composite image.