Image generation apparatus and image generation method

• US 20070189386A1
• Filed: 04/17/2007
• Published: 08/16/2007
• Est. Priority Date: 06/22/2005
• Status: Active Grant

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Imagawa, Taro, Azuma, Takeo
• Granted Patent: US 7,596,177 B2
• Time in Patent Office: Days
• US Class Current: 375/240.120
• CPC Class Codes: G06T 3/16 Spatio-temporal transformations, e.g. video cubism
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

The image generation apparatus includes: an image receiving unit which receives a first video sequence including frames having a first resolution and a second video sequence including frames having a second resolution which is higher than the first resolution, each frame of the first video sequence being obtained with a first exposure time, and each frame of the second video sequence being obtained with a second exposure time which is longer than the first exposure time; and an image integration unit which generates, from the first video sequence and the second video sequence, a new video sequence including frames having a resolution which is equal to or higher than the second resolution, at a frame rate which is equal to or higher than a frame rate of the first video sequence, by reducing a difference between a value of each frame of the second video sequence and a sum of values of frames of the new video sequence which are included within an exposure period of the frame of the second video sequence.

Claims

1. An image generation apparatus that generates, from plural video sequences respectively obtained by photographing a same object, a new video sequence which represents the same object, said apparatus comprising:

an image receiving unit operable to receive a first video sequence including frames having a first resolution and a second video sequence including frames having a second resolution which is higher than the first resolution, each frame of the first video sequence being obtained with a first exposure time, and each frame of the second video sequence being obtained with a second exposure time which is longer than the first exposure time; and

an image integration unit operable to generate, from the first video sequence and the second video sequence, a new video sequence including frames having a resolution which is equal to or higher than the second resolution, at a frame rate which is equal to or higher than a frame rate of the first video sequence, by reducing a difference between a value of each frame of the second video sequence and a sum of values of frames of the new video sequence which are included within an exposure period of the frame of the second video sequence.

2. The image generation apparatus according to claim 1,

wherein each of the frames of the second video sequence is obtained by open exposure between the frames.

3. The image generation apparatus according to claim 1, further comprising

a constraint unit operable to specify a constraint to be satisfied by pixel values of a new video sequence to be generated, based on continuity between pixel values of spatially and temporally adjacent pixels,

wherein said image integration unit is operable to generate the new video sequence in which the specified constraint is satisfied.

4. The image generation apparatus according to claim 1, further comprising:

a motion estimation unit operable to estimate a motion of the object from at least one of the plural video sequences received by said image receiving unit; and

a constraint unit operable to specify a constraint to be satisfied by pixel values of a new video sequence to be generated, based on a result of the motion estimation,

wherein said image integration unit is operable to generate the new video sequence in which the specified constraint is satisfied.

5. The image generation apparatus according to claim 4,

wherein said motion estimation unit is operable to calculate a confidence level of the motion estimation, and

said image integration unit is operable to generate the new video sequence, using the constraint specified based on the result of the motion estimation for an image area with a high confidence level calculated by said motion estimation unit, and using a predetermined constraint other than the constraint specified based on the motion estimation for an image area with a low confidence level.

6. The image generation apparatus according to claim 5,

wherein said motion estimation unit is operable to estimate the motion on the basis of each block obtained by dividing each of the frames which constitute the video sequence, and to calculate, as the confidence level, a negative value of a sum of squares of differences in pixel values between the blocks, and

said image integration unit is operable to generate the new video sequence by setting, as the image area with the high confidence level, a block with a confidence level of a value which is greater than a predetermined value, and setting, as the image area with the low confidence level, a block with a confidence level of a value which is equal to or smaller than the predetermined value.

7. The image generation apparatus according to claim 4,

wherein said motion estimation unit further includes an attitude sensor signal receiving unit operable to receive a signal from an attitude sensor that detects an attitude of an imaging device which images an object, and is operable to estimate the motion using the signal received by said attitude sensor receiving unit.

8. The image generation apparatus according to claim 1,

wherein said image integration unit is operable to extract chrominance information from the first video sequence, to generate a new intermediate video sequence from the second video sequence and luminance information obtained from the first video sequence, and to finally generate the new video sequence by adding the chrominance information to the generated intermediate video sequence.

9. The image generation apparatus according to claim 1,

wherein said image integration unit is operable to calculate an amount of temporal change between respective frames of at least one of the plural video sequences, and to generate the new video sequence by setting a temporal boundary between frames between which the calculated amount of change exceeds a predetermined value.

10. The image generation apparatus according to claim 1,

wherein said image integration unit is further operable to calculate a value indicating a confidence level of the generated new video sequence, and to output the calculated value as well as the new video sequence.

11. An image generation method for generating a new video sequence from plural video sequences, said method comprising:

receiving a first video sequence including frames having a first resolution and a second video sequence including frames having a second resolution which is higher than the first resolution, each frame of the first video sequence being obtained with a first exposure time, and each frame of the second video sequence being obtained with a second exposure time which is longer than the first exposure time; and

generating, from the first video sequence and the second video sequence, a new video sequence including frames having a resolution which is equal to or higher than the second resolution, at a frame rate which is equal to or higher than a frame rate of the first video sequence, by reducing a difference between a value of each frame of the second video sequence and a sum of values of frames of the new video sequence which are included within an exposure period of the frame of the second video sequence.

12. A program for generating a new video sequence from plural video sequences, said program causing a computer to execute:

receiving a first video sequence including frames having a first resolution and a second video sequence including frames having a second resolution which is higher than the first resolution, each frame of the first video sequence being obtained with a first exposure time, and each frame of the second video sequence being obtained with a second exposure time which is longer than the first exposure time; and

generating, from the first video sequence and the second video sequence, a new video sequence including frames having a resolution which is equal to or higher than the second resolution, at a frame rate which is equal to or higher than a frame rate of the first video sequence, by reducing a difference between a value of each frame of the second video sequence and a sum of values of frames of the new video sequence which are included within an exposure period of the frame of the second video sequence.