Image processing apparatus and image processing method

• US 9,646,397 B2
• Filed: 03/19/2013
• Issued: 05/09/2017
• Est. Priority Date: 04/19/2012
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Nashizawa, Hiroaki, Minegishi, Yuka
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US13/981,284
• Publication Number:
  US 20150036878A1
• Time in Patent Office: 1,512 Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

An image processing apparatus which composes a plurality of images captured while changing an exposure amount, includes a displacement detection unit which detects a displacement amount between the plurality of images, a correction unit which corrects a displacement between the images based on the displacement amount detected by the displacement detection unit, a moving object region detection unit which detects a moving object region from the plurality of images for which the displacement has been corrected, an image composition unit which composes the plurality of images for which the displacement has been corrected, and a moving object processing unit which replaces a region corresponding to the moving object region of the composite image composed by the image composition unit by an image obtained by performing weighted addition of the plurality of images.

Claims

1. An image processing apparatus which composes a plurality of images captured while changing an exposure amount, comprising:

a CPU implementing the functions of a displacement detection unit, a correction unit, a moving object region detection unit, a moving object region composition unit, an image composition unit and a moving object processing unit, wherein;

the displacement detection unit is configured to detect a displacement amount between the plurality of images;

the correction unit is configured to correct a displacement between the images based on the displacement amount detected by the displacement detection unit;

the moving object region detection unit is configured to detect a moving object region from the plurality of images for which the displacement has been corrected;

the moving object region composition unit is configured to compose a detection result of an underexposed image and a correctly exposed image with a detection result of the underexposed image and an overexposed image;

the image composition unit j configured to compose the plurality of images for which the displacement has been corrected to generate a composite image; and

the moving object processing unit is configured to replace a region corresponding to the moving object region of the composite image with an image obtained by performing weighted addition of the plurality of images.

2. The apparatus according to claim 1, wherein the image composition unit determines addition ratios of the respective images based on a luminance of an image with a smallest exposure amount among the plurality of images.

3. The apparatus according to claim 1, wherein the moving object processing unit determines the addition ratios of the plurality of images based on setting of a shooting exposure width.

4. The apparatus according to claim 1, wherein the moving object processing unit determines the addition ratios of the plurality of images based on a correct ISO sensitivity.

5. The apparatus according to claim 1, further comprising

a development processing unit configured to perform gamma conversion to equalize luminance levels of the plurality of images,

wherein the development processing unit determines a gamma characteristic of gamma conversion performed for the plurality of images based on an exposure amount ratio of the plurality of images.

6. The apparatus according to claim 5, wherein the development processing unit sets parameters of noise reduction processing according to the gamma characteristic.

7. The apparatus according to claim 1, wherein the moving object region detection unit detects, as a moving object region, a difference in image signal between a reference image and an aligned image.

8. The apparatus according to claim 1, wherein the moving object region detection unit removes, from the detection result of the moving object region, a luminance region subject to a highlight-detail loss and a luminance region subject to a shadow-detail loss.

9. The apparatus according to claim 8, wherein the moving object region detection unit removes an isolated region from an image in which the moving object has been detected.

10. The apparatus according to claim 1, further comprising

an image capturing unit configured to capture a plurality of images at a predetermined exposure amount.

11. The apparatus according to claim 1, further comprising

a selection unit configured to select a moving object region representation method when said moving object processing unit composes the plurality of images,

wherein based on the representation method selected by said selection unit, said moving object processing unit determines the addition ratios of the plurality of images to generate a weighted addition image.

12. The apparatus according to claim 1, wherein the moving object processing unit generates a weighted addition image by increasing the addition ratio of an image with a later shooting time among the plurality of images.

13. The apparatus according to claim 1, wherein the moving object processing unit generates a weighted addition image by increasing the addition ratio of an image with an earlier shooting time among the plurality of images.

14. The apparatus according to claim 1, wherein the moving object processing unit generates a weighted addition image by equalizing the addition ratios of the plurality of images.

15. The apparatus according to claim 1, wherein the moving object processing unit generates a weighted addition image by maximizing the addition ratio of an image with an intermediate shooting time among the plurality of images.

16. The apparatus according to claim 1, wherein the moving object processing unit determines addition ratios of an underexposed image, an overexposed image, and the weighted addition image based on a luminance of a correctly exposed image.

17. The apparatus according to claim 1, wherein the moving object processing unit determines addition ratios of a correctly exposed image and the weighted addition image based on a luminance of an underexposed image.

18. The apparatus according to claim 1, wherein the moving object processing unit determines addition ratios of a correctly exposed image and the weighted addition image based on a luminance of an overexposed image.

19. The apparatus according to claim 1, further comprising

an operation unit configured to accept a user operation for selecting the representation method.

20. The apparatus according to claim 1, further comprising

an operation unit configured to accept a user operation for selecting whether to correct the displacement by the correction unit.

21. The apparatus according to claim 1, further comprising

an operation unit configured to accept a user operation for selecting whether to process the moving object region by the moving object processing unit.

22. The apparatus according to claim 1, wherein the selection unit automatically selects the representation method based on feature information of the moving object region.

23. The apparatus according to claim 22, wherein the feature information includes one of a ratio of the moving object region to the image, a motion vector detected from the moving object region, and information indicating whether the moving object region is obtained by composing the plurality of images.

24. The apparatus according to claim 1, further comprising

an image capturing unit configured to capture a plurality of images at a predetermined exposure amount, and

a control unit configured to control an image shooting order by the image capturing unit based on the representation method set by the selection unit.

25. an image processing apparatus which composes a plurality of images captured while changing an exposure amount, comprising:

a CPU implementing the functions of a displacement detection unit, a correction unit, a moving object region detection unit, an image composition unit and a moving object processing unit, wherein;

the displacement detection unit is configured to detect a displacement amount between the plurality of images;

the correction unit is configured to correct a displacement between the images based on the displacement amount detected by the displacement detection unit;

the moving object region detection unit is configured to detect a moving object region from the plurality of images for which the displacement has been corrected, wherein the moving object region detection unit detects a first moving object region for an object by comparing an underexposed image with a correctly exposed image, and a second moving object region for the same object by comparing an underexposed image with an overexposed image;

the image composition unit is configured to compose the plurality of images for which the displacement has been corrected to generate a composite image; and

the moving object processing unit is configured to replace a region corresponding to the first moving object region and the second moving object region of the composite image with an image obtained by performing weighted addition of the plurality of images.

26. The apparatus according to claim 25, wherein the image composition unit determines addition ratios of the respective images based on a luminance of an image with a smallest exposure amount among the plurality of images.

27. The apparatus according to claim 25, wherein the moving object processing unit determines the addition ratios of the plurality of images based on setting of a shooting exposure width.

28. The apparatus according to claim 25, wherein the moving object processing unit determines the addition ratios of the plurality of images based on a correct ISO sensitivity.

29. The apparatus according to claim 25, further comprising

a development processing unit configured to perform gamma conversion to equalize luminance levels of the plurality of images,

wherein the development processing unit determines a gamma characteristic of gamma conversion performed for the plurality of images based on an exposure amount ratio of the plurality of images.

30. The apparatus according to claim 29, wherein the development processing unit sets parameters of noise reduction processing according to the gamma characteristic.

31. The apparatus according to claim 25, wherein the moving object region detection unit detects, as a moving object region, a difference in image signal between a reference image and an aligned image.