IMAGE COMBINING DEVICE AND IMAGING APPARATUS

• US 20080056613A1
• Filed: 08/31/2007
• Published: 03/06/2008
• Est. Priority Date: 08/31/2006
• Status: Active Grant

• Current Assignee: Xacti Corporation (Office Support KK)
• Original Assignee: Sanyo Electric Company Limited (Panasonic Holdings Corporation)
• Inventors: Hatanaka, Haruo, Iida, Takashi
• Granted Patent: US 7,956,897 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/003 Deblurring; Sharpening vibration or motion blur correction for cameras comprising an electronic image sensor H04N23/682
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

A plurality of divided exposure images (first to fourth images) are obtained by dividing a normal exposure time. By an image matching method, a motion detection circuit calculates a plurality of cumulative total correlation values between different divided exposure images, and searches the plurality of cumulative total correlation values for the smallest cumulative total correlation value, thereby calculating a motion vector between the different divided exposure images. In addition, based on the level of the smallest cumulative total correlation value, the level of reliability B[i] indicating the accuracy of the motion vector is calculated. An image combining circuit uses the level of reliability B[i] as a mixture proportion. The image combining circuit positions the divided exposure images based on the motion vector, and combines the divided exposure images together one on top of another in mixture proportions (B[i]), so as to form one still image that suffers less from camera shake.

Claims

1. An image combining device comprising:

a motion vector calculating portion calculating a motion vector between different divided exposure images of a plurality of divided exposure images obtained as a result of continuous shooting,

wherein the image combining device positions the divided exposure images based on the motion vector thus calculated, and combines the divided exposure images together one on top of another so as to form one composite image,

wherein the image combining device further comprises;

a mixture proportion calculating portion calculating, based on information commensurate with a motion of an image between the different divided exposure images, a mixture proportion of each divided exposure image to the composite image,

wherein the image combining device combines the divided exposure images together one on top of another in mixture proportions thus calculated so as to form the composite image.

2. The image combining device of claim 1,

wherein the motion vector calculating portion calculates the motion vector based on a plurality of cumulative total correlation values between the different divided exposure images, the plurality of cumulative total correlation values being obtained by an image matching method,

wherein the mixture proportion calculating portion calculates the mixture proportions based on the plurality of cumulative total correlation values used as the information.

3. The image combining device of claim 2,

wherein each divided exposure image is provided with a detection region,

wherein the motion vector calculating portion calculates, by the image matching method, m cumulative total correlation values (m is an integer equal to or greater than 2) between the different divided exposure images with respect to the detection region,

wherein the mixture proportion calculating portion calculates the mixture proportions based on one or more evaluation values of first, second, and third evaluation values,

wherein the first evaluation value is an evaluation value based on a smallest cumulative total correlation value of the m cumulative total correlation values, or the first evaluation value is an evaluation value based on a cumulative total correlation value equal to or smaller than a value obtained by increasing the smallest cumulative total correlation value according to a predetermined rule,

wherein the second evaluation value is an evaluation value based on a value commensurate with an average of the m cumulative total correlation values,

wherein the third evaluation value is an evaluation value based on a number of smallest cumulative total correlation values of the m cumulative total correlation values, or the third evaluation value is an evaluation value based on a number of cumulative total correlation values equal to or smaller than a value obtained by increasing the smallest cumulative total correlation value according to a predetermined rule.

4. The image combining device of claim 2,

wherein each divided exposure image is provided with a plurality of detection regions,

wherein the motion vector calculating portion calculates, by the image matching method, m cumulative total correlation values (m is an integer equal to or greater than 2) between the different divided exposure images with respect to each detection region,

wherein the image combining device further comprises a checking portion checking whether each detection region is valid or invalid,

wherein, based on the m cumulative total correlation values, the motion vector calculating portion obtains, for each detection region, a vector indicating a motion of an image within the detection region as a regional motion vector,

wherein the motion vector calculating portion calculates, as a motion vector of an entire image, the motion vector from the regional motion vector corresponding to the detection region found to be valid,

wherein the mixture proportion calculating portion calculates the mixture proportions based on one or more evaluation values of first, second, third, and fourth evaluation values,

wherein the first evaluation value is calculated based on a smallest cumulative total correlation value of the m cumulative total correlation values, the smallest cumulative total correlation value being obtained for each detection region, or the first evaluation value is calculated based on a cumulative total correlation value equal to or smaller than a value obtained by increasing the smallest cumulative total correlation value according to a predetermined rule, the cumulative total correlation value being obtained for each detection region,

wherein the second evaluation value is calculated based on a value commensurate with an average of the m cumulative total correlation values, the value being obtained for each detection region,

wherein the third evaluation value is calculated based on a number of smallest cumulative total correlation values of the m cumulative total correlation values, the number being obtained for each detection region, or the third evaluation value is calculated based on a number of cumulative total correlation values equal to or smaller than a value obtained by increasing the smallest cumulative total correlation value according to a predetermined rule, the number being obtained for each detection region,

wherein the fourth evaluation value is calculated based on a number of detection regions of the plurality of detection regions, the number of detection regions being found to be valid by the checking portion.

9. An imaging apparatus comprising:

an imaging portion performing continuous shooting and outputting a plurality of image data indicating a plurality of divided exposure images; and

the image combining device of claim 1,

wherein the image combining device forms the composite image based on the plurality of image data.

5. An image combining device comprising:

a motion vector calculating portion calculating a motion vector between different divided exposure images of a plurality of divided exposure images obtained as a result of continuous shooting,

wherein the image combining device sets one of the plurality of divided exposure images as a reference image, positions the divided exposure images based on the motion vector with respect to the reference image, and combines the divided exposure images together one on top of another so as to form one composite image,

wherein the image combining device further comprises a reference image setting portion operable to change the reference image,

wherein the reference image setting portion sets the reference image based on information commensurate with a motion of an image between the different divided exposure images.

6. The image combining device of claim 5,

wherein the motion vector calculating portion calculates the motion vector based on a plurality of cumulative total correlation values between the different divided exposure images, the plurality of cumulative total correlation values being obtained by an image matching method,

wherein the reference image setting portion sets the reference image based on the plurality of cumulative total correlation values used as the information.

7. The image combining device of claim 6,

wherein each divided exposure image is provided with a detection region,

wherein the motion vector calculating portion calculates, by the image matching method, m cumulative total correlation values (In is an integer equal to or greater than 2) between the different divided exposure images with respect to the detection region,

wherein the reference image setting portion sets the reference image based on one or more evaluation values of first, second, and third evaluation values,

wherein the first evaluation value is an evaluation value based on a smallest cumulative total correlation value of the m cumulative total correlation values, or the first evaluation value is an evaluation value based on a cumulative total correlation value equal to or smaller than a value obtained by increasing the smallest cumulative total correlation value according to a predetermined rule,

wherein the second evaluation value is an evaluation value based on a value commensurate with an average of the m cumulative total correlation values,

wherein the third evaluation value is an evaluation value based on a number of smallest cumulative total correlation values of the m cumulative total correlation values, or the third evaluation value is an evaluation value based on a number of cumulative total correlation values equal to or smaller than a value obtained by increasing the smallest cumulative total correlation value according to a predetermined rule.

8. The image combining device of claim 6,

wherein each divided exposure image is provided with a plurality of detection regions,

wherein the motion vector calculating portion calculates, by the image matching method, m cumulative total correlation values (m is an integer equal to or greater than 2) between the different divided exposure images with respect to each detection region,

wherein the image combining device further comprises a checking portion checking whether each detection region is valid or invalid,

wherein, based on the m cumulative total correlation values, the motion vector calculating portion obtains, for each detection region, a vector indicating a motion of an image within the detection region as a regional motion vector,

wherein the motion vector calculating portion calculate, as a motion vector of an entire image, the motion vector from the regional motion vector corresponding to the detection region found to be valid,

wherein the reference image setting portion sets the reference image based on one or more evaluation values of first, second, third, and fourth evaluation values,

wherein the first evaluation value is calculated based on a smallest cumulative total correlation value of the m cumulative total correlation values, the smallest cumulative total correlation value being obtained for each detection region, or the first evaluation value is calculated based on a cumulative total correlation value equal to or smaller than a value obtained by increasing the smallest cumulative total correlation value according to a predetermined rule, the cumulative total correlation value being obtained for each detection region,

wherein the second evaluation value is calculated based on a value commensurate with an average of the m cumulative total correlation values, the value being obtained for each detection region,

wherein the third evaluation value is calculated based on a number of smallest cumulative total correlation values of the m cumulative total correlation values, the number being obtained for each detection region, or the third evaluation value is calculated based on a number of cumulative total correlation values equal to or smaller than a value obtained by increasing the smallest cumulative total correlation value according to a predetermined rule, the number being obtained for each detection region,

wherein the fourth evaluation value is calculated based on a number of detection regions of the plurality of detection regions, the number of detection regions being found to be valid by the checking portion.

10. An imaging apparatus comprising;

an imaging portion performing continuous shooting and outputting a plurality of image data indicating a plurality of divided exposure images; and

the image combining device of claim 5,

wherein the image combining device forms the composite image based on the plurality of image data.

11. An image combining method comprising the steps of:

calculating a motion vector between different divided exposure images of a plurality of divided exposure images obtained as a result of continuous shooting; and

positioning the divided exposure images based on the motion vector thus calculated, and combining the divided exposure images together one on top of another so as to form one composite image,

wherein, based on information commensurate with a motion of an image between the different divided exposure images, a mixture proportion of each divided exposure image to the composite image is calculated,

wherein the divided exposure images are combined together one on top of another in mixture proportions thus calculated so as to form the composite image.

12. An image combining method comprising the steps of:

calculating a motion vector between different divided exposure images of a plurality of divided exposure images obtained as a result of continuous shooting;

setting one of the plurality of divided exposure images as a reference image; and

positioning the divided exposure images based on the motion vector thus calculated with respect to the reference image, and combining the divided exposure images together one on top of another so as to form one composite image,

wherein the reference image can be changed based on information commensurate with a motion of an image between the different divided exposure images.