Image processing device, image processing method, computer program, recording medium storing the computer program, frame-to-frame motion computing method, and image processing method

• US 7,903,156 B2
• Filed: 05/13/2008
• Issued: 03/08/2011
• Est. Priority Date: 07/17/2007
• Status: Active Grant

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Nobori, Kunio, Motomura, Hideto, Imagawa, Taro, Azuma, Takeo
• Primary Examiner(s): Ometz; David L
• Assistant Examiner(s): Berhan; Ahmed A
• Application Number:
  US12/600,390
• Publication Number:
  US 20100157149A1
• Time in Patent Office: 1,029 Days
• Field of Search: 348/265, 348/267, 348271-273, 348277-279, 348/208.99, 348/208.13, 348/220.1, 348/222.1
• US Class Current: 348/272
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

An image processing apparatus and method is provided to generate a moving picture with a high resolution, a high frame rate and a high SNR by eliminating a decrease in SNR even if the intensity of incoming light has been halved by a half mirror. The apparatus generates a multi-color moving picture based on first and second moving pictures, which respectively have first and second frame rates (where the second rate is higher than the first rate) and are comprised of pictures representing a first color component and pictures representing a second color component different from the first color component. The resolution of the second moving picture is equal to or lower than that of the first moving picture. The apparatus includes: a first image synthesizing section, which generates, based on the data of the first and second moving pictures, a synthetic picture by modifying some of the pictures that form the first moving picture and which outputs a synthesized moving picture corresponding to the first moving picture at a higher frame rate than the first frame rate by using the first moving picture and the synthetic picture; and a second image synthesizing section, which receives and synthesizes together the synthesized moving picture and the second moving picture, thereby generating a multi-color moving picture including the first and second color components.

Claims

1. An image processing apparatus for generating multi-color moving picture data based on the data of first and second moving pictures,

the first moving picture having a first frame rate and being comprised of a number of pictures, each representing a first color component,

the second moving picture having a second frame rate, which is higher than the first frame rate, and being comprised of a number of pictures, each representing a second color component that is different from the first color component,

resolution of the second moving picture being equal to or lower than that of the first moving picture,

wherein the apparatus comprises:

a first image synthesizing section, which generates, based on the data of the first and second moving pictures, a synthesized moving picture corresponding to the first moving picture at a higher frame rate than the first frame rate; and

a second image synthesizing section, which receives and synthesizes together the synthesized moving picture and the second moving picture, thereby generating a multi-color moving picture including the first and second color components, and

wherein the first image synthesizing section comprises;

a motion estimating section, which receives the data of the second moving picture and which generates motion information based on a change between at least two of the pictures that form the second moving picture; and

a motion compensating section for generating a synthetic picture by modifying some of the pictures that form the first moving picture based on the motion information and outputting the synthesized moving picture corresponding to the first moving picture by using the synthetic picture and the first moving picture.

2. The apparatus of claim 1, wherein the first and second moving pictures have their respective base frame pictures, which have been shot at the same time, the second moving picture further having an intermediate frame picture that has been shot between two adjacent base frame pictures thereof, and

wherein the motion estimating section generates the motion information based on a change between the intermediate frame picture and the base frame picture of the second moving picture, and

wherein the motion compensating section modifies the base frame picture of the first moving picture in accordance with the motion information, thereby generating a synthetic picture corresponding to the intermediate frame of the second moving picture.

3. The apparatus of claim 2, wherein the motion estimating section generates the motion information based on a change between the intermediate frame picture of the second moving picture and a base frame picture thereof that has been shot before the intermediate frame picture.

4. The apparatus of claim 2, wherein the motion estimating section generates the motion information based on a change between the intermediate frame picture of the second moving picture and a base frame picture thereof that has been shot after the intermediate frame picture.

5. The apparatus of claim 2, wherein the motion estimating section generates motion information S based on a change between the intermediate frame picture U of the second moving picture and a base frame picture S thereof that has been shot before the intermediate frame picture, and also generates motion information T based on a change between the intermediate frame picture U of the second moving picture and a base frame picture T thereof that has been shot after the intermediate frame picture, and

wherein the motion compensating section modifies the base frame picture S of the first moving picture, corresponding to the base frame picture S of the second moving picture, in accordance with the motion information S, and also modifies the base frame picture T of the first moving picture, corresponding to the base frame picture T of the second moving picture, in accordance with the motion information T, thereby generating a synthetic picture corresponding to the intermediate frame U of the second moving picture.

6. The apparatus of claim 2, wherein the motion estimating section receives the data of the first and second moving pictures, generates not only a base frame resolution converted picture by converting the resolution of the base frame picture of the second moving picture using the base frame picture of the first moving picture but also an intermediate frame resolution converted picture by converting the resolution of the intermediate frame picture of the second moving picture using the base frame picture of the first moving picture, further generates the motion information based on a degree of similarity between the base frame resolution converted picture and the intermediate frame resolution converted picture, and outputs the motion information as the motion information of the second moving picture.

7. The apparatus of claim 6, wherein the motion estimating section generates the converted moving picture by converting the resolution of the second moving picture based on a spatial correlation between associated pictures of the first and second moving pictures.

8. The apparatus of claim 1, wherein the motion estimating section defines multiple areas for each of the at least two pictures, and generates motion information about each said area based on a change in image within that area, and

wherein the motion compensating section defines multiple areas for each of the pictures that form the first moving picture, and modifies each said area in accordance with the motion information, thereby generating a single synthetic picture.

9. The apparatus of claim 1, wherein by using a temporal correlation between the first and second moving pictures as an evaluation value, the motion compensating section generates a synthetic picture corresponding to the first moving picture at an arbitrary timing based on the data of the first and second moving pictures and the motion information.

10. The apparatus of claim 9, wherein by using a temporal correlation between the synthesized moving picture and the second moving picture as an evaluation value, the motion compensating section corrects the pixel values of the synthetic picture included in the synthesized moving picture.

11. The apparatus of claim 1, further comprising a resolution raising section, which receives the data of the synthesized moving picture corresponding to the first moving picture and the data of the second moving picture, and raises the resolution of the second moving picture based on information about pictures that form the synthesized moving picture, thereby generating a synthesized moving picture corresponding to the second moving picture,

wherein the synthesizing section receives the synthesized moving picture corresponding to the second moving picture instead of the second moving picture itself, and synthesizes the respective synthesized moving pictures corresponding to the first and second moving pictures together, thereby generating the multi-color moving picture.

12. The apparatus of claim 11, wherein the resolution raising section raises the resolution of the second moving picture in accordance with information about the pictures that form the synthesized moving picture corresponding to the first moving picture, and uses a spatial correlation between the second moving picture and the synthesized moving picture corresponding to the first moving picture as an evaluation value for generating the synthesized moving picture corresponding to the second moving picture.

13. The apparatus of claim 12, wherein the resolution raising section raises the resolution of the second moving picture in accordance with information about the pictures that form the synthesized moving picture corresponding to the first moving picture, and changes the degrees of correlation between the respective pictures that form the synthesized moving pictures corresponding to the first and second moving pictures according to the degree of correlation between the respective pictures that form the first moving picture and the ones that form the second moving picture.

14. The apparatus of claim 12, wherein the resolution raising section raises the resolution of the second moving picture in accordance with information about the pictures that form the synthesized moving picture corresponding to the first moving picture, and generates pictures to form the synthesized moving picture corresponding to the second moving picture by interpolating and expanding the pictures that form the second moving picture as for an area where there is a low degree of correlation between the respective pictures that form the first moving picture and the ones that form the second moving picture.

15. The apparatus of claim 11, wherein the resolution raising section raises the resolution of the second moving picture in accordance with information about the pictures that form the synthesized moving picture corresponding to the first moving picture, and adds high range components, which are included in the synthesized moving picture corresponding to the first moving picture, to the second moving picture when generating a synthesized moving picture corresponding to the second moving picture.

16. The apparatus of claim 11, wherein the resolution raising section raises the resolution of the second moving picture in accordance with information about the pictures that form the synthesized moving picture corresponding to the first moving picture, and increases the degree of correlation between the synthesized moving picture corresponding to the first moving picture and a synthetic picture corresponding to the second moving picture when generating a synthesized moving picture corresponding to the second moving picture.

17. The apparatus of claim 1, wherein the apparatus is able to generate multi-color moving picture data based on the data of a third moving picture, too, and

the third moving picture having a third frame rate, which is higher than the first frame rate, and being comprised of a number of pictures, each representing a third color component that is different from the first and second color components,

the resolution of the third moving picture being equal to or lower than that of the first moving picture,

wherein the motion estimating section further receives the data of the third moving picture and generates motion information based on a change between at least two of the pictures that form the second moving picture, and

wherein the motion compensating section generates the synthesized moving picture corresponding to the first moving picture based on the motion information about the second and third moving pictures and the data of the first moving picture, and

wherein the synthesizing section receives and synthesizes together the synthesized moving picture and the second and third moving pictures, thereby generating a multi-color moving picture including the first, second and third color components.

18. The apparatus of claim 17, wherein pixels are arranged in spatially different locations between the second and third moving pictures.

19. The apparatus of claim 18, wherein the motion estimating section generates a combined moving picture by calculating a weighted sum of the respective pictures that form the second and third moving pictures with the spatial arrangement of pixels taken into account and also generates the motion information based on a change between at least two of the pictures that form the combined moving picture generated.

20. The apparatus of claim 19, wherein the motion estimating section generates a combined moving picture by calculating a weighted sum of the respective pictures that form the second and third moving pictures with the spatial arrangement of pixels taken into account, also generates a base frame resolution converted picture by converting the resolution of the base frame picture of the combined moving picture using the base frame picture of the first moving picture and an intermediate frame resolution converted picture by converting the resolution of the intermediate frame picture of the combined moving picture using the base frame picture of the first moving picture, and further generates the motion information based on a change between the base frame resolution converted picture and the intermediate frame resolution converted picture.

21. The apparatus of claim 1, wherein the first color component is green.

22. The apparatus of claim 1, further comprising:

a color separating section for splitting light received into at least two light beams that have the first and second color components, respectively; and

an image capturing section for shooting the first and second moving pictures based on the split light beams with the first and second color components, respectively, and

wherein the image capturing section shoots the first moving picture with a resolution that is equal to or higher than the resolution of the second moving picture and at a frame rate that is less than the frame rate of the second moving picture.

23. The apparatus of claim 22, wherein the image capturing section controls the resolutions and the frame rates of the moving pictures representing multiple color components according to the type of the moving picture that has been shot.

24. An image processing method for generating multi-color moving picture data based on the data of first and second moving pictures,

the first moving picture having a first frame rate and being comprised of a number of pictures, each representing a first color component,

the second moving picture having a second frame rate, which is higher than the first frame rate, and being comprised of a number of pictures, each representing a second color component that is different from the first color component,

resolution of the second moving picture being equal to or lower than that of the first moving picture,

wherein the method comprises the steps of:

receiving the data of the second moving picture and generating motion information based on a change between at least two of the pictures that form the second moving picture;

generating, based on the motion information, a synthetic picture by modifying some of the pictures that form the first moving picture and outputting a synthesized moving picture corresponding to the first moving picture at a higher frame rate than the first frame rate by using the synthetic picture and the first moving picture; and

receiving and synthesizing together the synthesized moving picture and the second moving picture, thereby generating a multi-color moving picture including the first and second color components.

25. A computer program embodied in a non-transitory storage medium, that is defined to make a processor, built in an image processing apparatus, generate multi-color moving picture data based on the data of first and second moving pictures,

the first moving picture having a first frame rate and being comprised of a number of pictures, each representing a first color component,

the second moving picture having a second frame rate, which is higher than the first frame rate, and being comprised of a number of pictures, each representing a second color component that is different from the first color component, resolution of the second moving picture being equal to or lower than that of the first moving picture,

wherein the computer program is defined to make the processor perform the steps of:

receiving the data of the second moving picture and generating motion information based on a change between at least two of the pictures that form the second moving picture;

generating, based on the motion information, a synthetic picture by modifying some of the pictures that form the first moving picture and outputting a synthesized moving picture corresponding to the first moving picture at a higher frame rate than the first frame rate by using the synthetic picture and the first moving picture; and

receiving and synthesizing together the synthesized moving picture and the second moving picture, thereby generating a multi-color moving picture including the first and second color components.

26. A non-transitory storage medium having stored thereon the computer program of claim 25.

27. An inter-frame motion estimating method for estimating a motion between associated constituent frames of first and second moving pictures based on the data of the first and second moving pictures,

the first moving picture having a first frame rate and being comprised of a number of pictures,

the second moving picture having a second frame rate, which is higher than the first frame rate, and being also comprised of a number of pictures,

resolution of the second moving picture being equal to or lower than that of the first moving picture,

the first and second moving pictures having their respective base frame pictures, which have been shot at the same time, the second moving picture further having an intermediate frame picture that has been shot between two adjacent base frame pictures thereof, and

wherein the method comprises the steps of:

receiving the data of the first and second moving pictures, and generating a base frame resolution converted picture by converting the resolution of the base frame picture of the second moving picture using the base frame picture of the first moving picture;

generating an intermediate frame resolution converted picture by converting the resolution of the intermediate frame picture of the second moving picture using the base frame picture of the first moving picture; and

generating the motion information based on a degree of similarity between the base frame resolution converted picture and the intermediate frame resolution converted picture.

28. The method of claim 27, wherein the first moving picture is comprised of a number of pictures, each representing a first color component, and wherein the second moving picture is comprised of a number of pictures, each representing a second color component that is different from the first color component.

29. An image processing method for generating the data of a synthesized moving picture based on the data of first and second moving pictures,

the first moving picture having a first frame rate and being comprised of a number of pictures,

the second moving picture having a second frame rate, which is higher than the first frame rate, and being also comprised of a number of pictures,

the first and second moving pictures having their respective base frame pictures, which have been shot at the same time, the second moving picture further having an intermediate frame picture that has been shot between two adjacent base frame pictures thereof, and

wherein the method comprises the steps of:

receiving the data of the first and second moving pictures, and generating a base frame resolution converted picture by converting resolution of the base frame picture of the second moving picture using the base frame picture of the first moving picture;

generating an intermediate frame resolution converted picture by converting resolution of the intermediate frame picture of the second moving picture using the base frame picture of the first moving picture;

generating the motion information based on a degree of similarity between the base frame resolution converted picture and the intermediate frame resolution converted picture; and

generating, based on the data of the first and second moving pictures and the motion information, a synthetic picture by modifying some of the pictures that form the first moving picture and outputting a synthesized moving picture corresponding to the first moving picture at a higher frame rate than the first frame rate by using the first moving picture and the synthetic picture.