Video image synthesis method, video image synthesizer, image processing method, image processor, and programs for executing the synthesis method and processing method

• US 20040086193A1
• Filed: 08/25/2003
• Published: 05/06/2004
• Est. Priority Date: 08/28/2002
• Status: Active Grant

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Photo Film Co Limited (Fujifilm Holdings Corporation)
• Inventors: Ito, Wataru, Kameyama, Hirokazu
• Granted Patent: US 7,729,563 B2
• Time in Patent Office: Days
• US Class Current: 382/254
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

To acquire a high-resolution frame from a plurality of frames sampled from a video image, it is necessary to obtain a high-resolution frame with reduced picture quality degradation regardless of motion of a subject included in the frame. Because of this, between a plurality of contiguous frames Fr_N and Fr_N+1, there is estimated a correspondent relationship. Based on the correspondent relationship, the frames Fr_N+1 and Fr_N are interposed to obtain first and second interpolated frames Fr_H1 and Fr_H2. Based on the correspondent relationship, the coordinates of the frame Fr_N+1 are transformed, and from a correlation value with the frame Fr_N, there is obtained a weighting coefficient α(x°, y°) that makes the weight of the first interpolated frame Fr_H1 greater as a correlation becomes greater. With the weighting coefficient, the first and second interpolated frames are weighted and added to acquire a synthesized frame Fr_G.

Claims

1. A video image synthesis method comprising the steps of:

sampling two contiguous frames from a video image;

placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;

acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;

acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;

acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;

computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;

acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value; and

acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient.

3. The synthesis method as set forth in claim 1, wherein

when said correlation value has been computed for each of the pixels and/or each local region that constitute each of said frames,

said correlation value is filtered to compute a filtered correlation value, and said weighting coefficient is acquired based on said filtered correlation value.

5. The synthesis method as set forth in claim 1, wherein

when said correlation value has been computed for each of the pixels and/or each local region that constitute each of said frames,

said weighting coefficient is interpolated to acquire weighting coefficients for all pixels that constitute said first and second interpolated frames.

7. The synthesis method as set forth in claim 1, wherein said weighting coefficient is acquired by referring to a nonlinear graph in which said correlation value is represented in the horizontal axis and said weighting coefficient in the vertical axis.

9. The synthesis method as set forth in claim 1, wherein the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said coordinate-transformed frame, computation of said correlation value, acquisition of said weighting coefficient, and acquisition of said synthesized frame are performed by employing at least one component that constitutes said frame.

2. A video image synthesis method comprising the steps of:

sampling three or more contiguous frames from a video image;

placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;

acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;

acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;

acquiring a plurality of coordinate-transformed frames by transforming coordinates of the images within the patches of said other frames to a coordinate space of said reference frame, based on said correspondent relationships;

computing correlation values that represent a correlation between the image within the patch of each of said coordinate-transformed frames and the image within said reference patch of said reference frame;

acquiring weighting coefficients that make a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation values; and

acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.

4. The synthesis method as set forth in claim 2, wherein

when said correlation value has been computed for each of the pixels and/or each local region that constitute each of said frames,

said correlation value is filtered to compute a filtered correlation value, and said weighting coefficient is acquired based on said filtered correlation value.

6. The synthesis method as set forth in claim 2, wherein

when said correlation value has been computed for each of the pixels and/or each local region that constitute each of said frames,

said weighting coefficient is interpolated to acquire weighting coefficients for all pixels that constitute said first and second interpolated frames.

8. The synthesis method as set forth in claim 2, wherein said weighting coefficient is acquired by referring to a nonlinear graph in which said correlation value is represented in the horizontal axis and said weighting coefficient in the vertical axis.

10. The synthesis method as set forth in claim 2, wherein the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said coordinate-transformed frame, computation of said correlation value, acquisition of said weighting coefficient, and acquisition of said synthesized frame are performed by employing at least one component that constitutes said frame.

11. A video image synthesizer comprising:

sampling means for sampling two contiguous frames from a video image;

correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;

first interpolation means for acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;

second interpolation means for acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;

coordinate transformation means for acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;

correlation-value computation means for computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;

weighting-coefficient acquisition means for acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value; and

synthesis means for acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient.

13. The video image synthesizer as set forth in claim 11, wherein

when said correlation value has been computed for each of pixels and/or each local region that constitute each of said frames,

said synthesizer further comprises means for filtering said correlation value to compute a filtered correlation value, and said weighting-coefficient acquisition means acquires said weighting coefficient, based on said filtered correlation value.

15. The video image synthesizer as set forth in claim 11, wherein

when said correlation value has been computed for each of the pixels and/or each local region that constitute each of said frames,

said weighting-coefficient acquisition means performs interpolation on said weighting coefficient, thereby acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.

17. The video image synthesizer as set forth in claim 11, wherein said weighting-coefficient acquisition means acquires said weighting coefficient by referring to a nonlinear graph in which said correlation value is represented in the horizontal axis and said weighting coefficient in the vertical axis.

19. The video image synthesizer as set forth in claim 11, wherein said correspondent relationship estimation means, said first interpolation means, said second interpolation means, said coordinate transformation means, said correlation-value computation means, said weighting-coefficient acquisition means, and said synthesis means perform the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said coordinate-transformed frame, computation of said correlation value, acquisition of said weighting coefficient, and acquisition of said synthesized frame, by employing at least one component that constitutes said frame.

12. A video image synthesizer comprising:

sampling means for sampling three or more contiguous frames from a video image;

correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;

first interpolation means for acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;

second interpolation means for acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;

coordinate transformation means for acquiring a plurality of coordinate-transformed frames by transforming coordinates of the images within the patches of said other frames to a coordinate space of said reference frame, based on said correspondent relationships;

correlation-value computation means for computing correlation values that represent a correlation between the image within the patch of each of said coordinate-transformed frames and the image within said reference patch of said reference frame;

weighting-coefficient acquisition means for acquiring weighting coefficients that make a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation values; and

synthesis means for acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.

14. The video image synthesizer as set forth in claim 12, wherein

when said correlation value has been computed for each of pixels and/or each local region that constitute each of said frames,

said synthesizer further comprises means for filtering said correlation value to compute a filtered correlation value, and said weighting-coefficient acquisition means acquires said weighting coefficient, based on said filtered correlation value.

16. The video image synthesizer as set forth in claim 12, wherein

when said correlation value has been computed for each of the pixels and/or each local region that constitute each of said frames,

said weighting-coefficient acquisition means performs interpolation on said weighting coefficient, thereby acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.

18. The video image synthesizer as set forth in claim 12, wherein said weighting-coefficient acquisition means acquires said weighting coefficient by referring to a nonlinear graph in which said correlation value is represented in the horizontal axis and said weighting coefficient in the vertical axis.

20. The video image synthesizer as set forth in claim 12, wherein said correspondent relationship estimation means, said first interpolation means, said second interpolation means, said coordinate transformation means, said correlation-value computation means, said weighting-coefficient acquisition means, and said synthesis means perform the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said coordinate-transformed frame, computation of said correlation value, acquisition of said weighting coefficient, and acquisition of said synthesized frame, by employing at least one component that constitutes said frame.

21. A program for causing a computer to execute a video image synthesis method comprising:

a procedure of sampling two contiguous frames from a video image;

a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;

a procedure of acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;

a procedure of acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;

a procedure of acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;

a procedure of computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;

a procedure of acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value; and

a procedure of acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient.

23. The program as set forth in claim 21, wherein

when said correlation value has been computed for each pixel and/or each local region that constitutes each of said frames,

said method further comprises a procedure of filtering said correlation value to compute a filtered correlation value, and said weighting-coefficient acquisition procedure is a procedure of acquiring said weighting coefficient, based on said filtered correlation value.

25. The program as set forth in claim 21, wherein

when said correlation value has been computed for each of the pixels and/or each local region that constitute each of said frames,

said weighting-coefficient acquisition procedure is a procedure of performing interpolation on said weighting coefficient and acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.

27. The program as set forth in claim 21, wherein said weighting-coefficient acquisition procedure is a procedure of acquiring said weighting coefficient by referring to a nonlinear graph in which said correlation value is represented in the horizontal axis and said weighting coefficient in the vertical axis.

29. The program as set forth in claim 21, wherein said correspondent relationship estimation procedure, said first interpolated frame acquisition procedure, said second interpolated frame acquisition procedure, said coordinate-transformed frame acquisition procedure, said correlation value computation procedure, said weighting-coefficient acquisition procedure, and said synthesized-frame acquisition procedure are procedures of performing the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said coordinate-transformed frame, computation of said correlation value, acquisition of said weighting coefficient, and acquisition of said synthesized frame by employing at least one component that constitutes said frame, respectively.

22. A program for causing a computer to execute a video image synthesis method comprising:

a procedure of sampling three or more contiguous frames from a video image;

a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;

a procedure of acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;

a procedure of acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;

a procedure of acquiring a plurality of coordinate-transformed frames by transforming coordinates of the images within the patches of said other frames to a coordinate space of said reference frame, based on said correspondent relationships;

a procedure of computing correlation values that represent a correlation between the image within the patch of each of said coordinate-transformed frames and the image within said reference patch of said reference frame;

a procedure of acquiring weighting coefficients that make a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation values; and

a procedure of acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.

24. The program as set forth in claim 22, wherein

when said correlation value has been computed for each pixel and/or each local region that constitute each of said frames,

said method further comprises a procedure of filtering said correlation value to compute a filtered correlation value, and said weighting-coefficient acquisition procedure is a procedure of acquiring said weighting coefficients, based on said filtered correlation value.

26. The program as set forth in claim 22, wherein

when said correlation value has been computed for each of the pixels and/or each local region that constitute each of said frames,

said weighting-coefficient acquisition procedure is a procedure of performing interpolation on said weighting coefficients and acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.

28. The program as set forth in claim 22, wherein said weighting-coefficient acquisition procedure is a procedure of acquiring said weighting coefficients by referring to a nonlinear graph in which said correlation value is represented in the horizontal axis and said weighting coefficient in the vertical axis.

30. The program as set forth in claim 22, wherein said correspondent relationship estimation procedure, said first interpolated frame acquisition procedure, said second interpolated frame acquisition procedure, said coordinate-transformed frame acquisition procedure, said correlation value computation procedure, said weighting-coefficient acquisition procedure, and said synthesized-frame acquisition procedure are procedures of performing the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said coordinate-transformed frame, computation of said correlation value, acquisition of said weighting coefficient, and acquisition of said synthesized frame by employing at least one component that constitutes said frame, respectively.

31. A video image synthesis method comprising the steps of:

sampling two contiguous frames from a video image;

placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;

acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;

acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;

acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of said other frame;

acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and

acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient.

33. The synthesis method as set forth in claim 31, wherein

when said edge information has been computed for each of the pixels that constitute each of said frames,

said weighting coefficient is interpolated to acquire weighting coefficients for all pixels that constitute said first and second interpolated frames.

35. The synthesis method as set forth in claim 31, wherein the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame are performed by employing at least one component that constitutes said frame.

32. A video image synthesis method comprising the steps of:

sampling three or more contiguous frames from a video image;

placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;

acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;

acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;

acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of each of said other frames;

acquiring weighting coefficients that make a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and

acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.

34. The synthesis method as set forth in claim 32, wherein

when said edge information has been computed for each of the pixels that constitute each of said frames,

said weighting coefficient is interpolated to acquire weighting coefficients for all pixels that constitute said first and second interpolated frames.

36. The synthesis method as set forth in claim 32, wherein the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame are performed by employing at least one component that constitutes said frame.

37. A video image synthesizer comprising:

sampling means for sampling two contiguous frames from a video image;

correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;

first interpolation means for acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;

second interpolation means for acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;

edge information acquisition means for acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of said other frame;

weighting-coefficient acquisition means for acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and

synthesis means for acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient.

39. The video image synthesizer as set forth in claim 37, wherein

when said edge information has been computed for each of the pixels that constitute each of said frames,

said weighting-coefficient acquisition means performs interpolation on said weighting coefficient, thereby acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.

41. The video image synthesizer as set forth in claim 37, wherein said correspondent relationship estimation means, said first interpolation means, said second interpolation means, said edge information acquisition means, said weighting-coefficient acquisition means, and said synthesis means perform the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame, by employing at least one component that constitutes said frame.

38. A video image synthesizer comprising:

sampling means for sampling three or more contiguous frames from a video image;

correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;

first interpolation means for acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;

second interpolation means for acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;

edge information acquisition means for acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of each of said other frames;

weighting-coefficient acquisition means for acquiring weighting coefficients that make a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and

synthesis means for acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.

40. The video image synthesizer as set forth in claim 38, wherein

when said edge information has been computed for each of the pixels that constitute each of said frames,

said weighting-coefficient acquisition means performs interpolation on said weighting coefficients, thereby acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.

42. The video image synthesizer as set forth in claim 38, wherein said correspondent relationship estimation means, said first interpolation means, said second interpolation means, said edge information acquisition means, said weighting-coefficient acquisition means, and said synthesis means perform the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame, by employing at least one component that constitutes said frame.

43. A program for causing a computer to execute a video image synthesis method comprising:

a procedure of sampling two contiguous frames from a video image;

a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;

a procedure of acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;

a procedure of acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;

a procedure of acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of said other frame;

a procedure of acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and

a procedure of acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient.

45. The program as set forth in claim 43, wherein

when said edge information has been computed for each of the pixels that constitute each of said frames,

said weighting-coefficient acquisition procedure is a procedure of performing interpolation on said weighting coefficient and acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.

47. The program as set forth in claim 43, wherein said correspondent relationship estimation procedure, said first interpolated frame acquisition procedure, said second interpolated frame acquisition procedure, said edge information acquisition procedure, said weighting-coefficient acquisition procedure, and said synthesized-frame acquisition procedure are procedures of performing the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame by employing at least one component that constitutes said frame, respectively.

44. A program for causing a computer to execute a video image synthesis method comprising:

a procedure of sampling three or more contiguous frames from a video image;

a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;

a procedure of acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;

a procedure of acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;

a procedure of acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of each of said other frames;

a procedure of acquiring weighting coefficients that make a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and

a procedure of acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames.

46. The program as set forth in claim 44, wherein

when said edge information has been computed for each of the pixels that constitute each of said frames,

said weighting-coefficient acquisition procedure is a procedure of performing interpolation on said weighting coefficients and acquiring weighting coefficients for all pixels that constitute said first and second interpolated frames.

48. The program as set forth in claim 44, wherein said correspondent relationship estimation procedure, said first interpolated frame acquisition procedure, said second interpolated frame acquisition procedure, said edge information acquisition procedure, said weighting-coefficient acquisition procedure, and said synthesized-frame acquisition procedure are procedures of performing the estimation of said correspondent relationship, acquisition of said first interpolated frame, acquisition of said second interpolated frame, acquisition of said edge information, acquisition of said weighting coefficient, and acquisition of said synthesized frame by employing at least one component that constitutes said frame, respectively.

49. A video image synthesis method comprising the steps of:

sampling a predetermined number of contiguous frames, which include a reference frame and are two or more frames, from said video image;

placing a reference patch comprising one or a plurality of rectangular areas on said reference frame;

respectively placing patches which are the same as said reference patch, on the others of said predetermined number of frames;

moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch;

respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

acquiring a synthesized frame from said predetermined number of frames, based on said correspondent relationships;

wherein said predetermined number of frames are determined based on image characteristics of said video image or synthesized frame, and said predetermined number of frames are sampled.

50. The synthesis method as set forth in claim 49, wherein

said correspondent relationships are acquired in order of other frames closer to said reference frame, and a correlation is acquired between each of said other frames, in which said correspondent relationship is acquired, and said reference frame;

and when said correlation is lower than a predetermined threshold value, acquisition of said correspondent relationships is stopped, and said synthesized frame is obtained based on said correspondent relationship by employing said other frames, in which said correspondent relationship has been acquired, and said reference frame.

51. A video image synthesizer comprising:

sampling means for sampling a predetermined number of contiguous frames, which include a reference frame and are two or more frames, from a video image;

correspondent relationship acquisition means for placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said predetermined number of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

frame synthesis means for acquiring a synthesized frame from said predetermined number of frames, based on said correspondent relationships acquired by said correspondent relationship acquisition means;

wherein said sampling means is equipped with frame-number determination means for determining said predetermined number of frames on the basis of image characteristics of said video image or synthesized frame, and samples said predetermined number of frames determined by said frame-number determination means.

52. The video image synthesizer as set forth in claim 51, wherein

said correspondent relationship acquisition means acquires said correspondent relationships in order of other frames closer to said reference frame;

there is provided stoppage means for acquiring a correlation between each of said other frames, in which said correspondent relationship is acquired by said correspondent relationship acquisition means, and said reference frame, and stopping a process which is being performed in said correspondent relationship acquisition means when said correlation is lower than a predetermined threshold value; and

said frame synthesis means acquires said synthesized frame by employing said other frames, in which said correspondent relationship has been acquired, and said reference frame, based on said correspondent relationship acquired by said correspondent relationship acquisition means.

53. A program for causing a computer to execute:

a determination process of determining the number of frames, based on image characteristics of a video image or image characteristics of a synthesized frame which is obtained from the determined number of frames of said video image;

a sampling process of sampling the determined number of frames which are contiguous and include a reference frame, from said video image;

a correspondent relationship acquisition process of placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said predetermined number of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

a frame synthesis process of acquiring said synthesized frame from said determined number of frames, based on said correspondent relationships.

54. The program as set forth in claim 53, wherein

said correspondent relationship acquisition process acquires said correspondent relationships in order of other frames closer to said reference frame;

and

said computer is further caused to execute a process of acquiring a correlation between each of said other frames, in which said correspondent relationship is acquired, and said reference frame, and stopping said correspondent relationship acquisition process when said correlation is lower than a predetermined threshold value.

55. A video image synthesis method comprising the steps of:

obtaining a contiguous frame group by detecting a plurality of frames that represent contiguous scenes in a video image;

placing a reference patch comprising one or a plurality of rectangular areas on one of said plurality of frames included in said contiguous frame group which is used as a reference frame;

respectively placing patches which are the same as said reference patch, on the others of said plurality of frames;

moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch;

respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.

56. The synthesis method as set forth in claim 55, wherein

a correlation between adjacent frames, which is started from said reference frame, is acquired;

and

said contiguous frame group that is detected comprises frames ranging from said reference frame to a frame, which is closer to said reference frame, between a pair of said adjacent frames in which said correlation is lower than a predetermined first threshold value.

57. The synthesis method as set forth in claim 56, wherein

a histogram is computed for at least one of the Y, Cb, and Cr components of each of said adjacent frames (where the Y component is a luminance component and the Cb and Cr components are color difference components);

a Euclidean distance for each component between said adjacent frames is computed by employing said histogram;

the sum of the Euclidean distances for said three components is computed; and

when said sum is a predetermined second threshold value or greater, the correlation between said adjacent frames is lower than said predetermined first threshold value.

58. The synthesis method as set forth in claim 57, wherein said histogram is computed by dividing each of components, which are used, among said three components by a value greater than 1.

59. The synthesis method as set forth in claim 56, wherein

a difference between pixel values of corresponding pixels of said adjacent frames is computed for all corresponding pixels;

the sum of absolute values of the differences for all corresponding pixels is computed; and

when said sum is a third threshold value or greater, said correlation between adjacent frames is lower than said predetermined first threshold value.

60. The synthesis method as set forth in claim 56, wherein said correlation is computed by employing a reduced image or thinned image of each frame.

61. The synthesis method as set forth in claim 55, wherein the detection of frames that constitute said contiguous frame group is stopped when the number of detected frames reaches a predetermined upper limit value.

62. A video image synthesizer comprising:

contiguous frame group detection means for obtaining a contiguous frame group by detecting a plurality of frames that represent contiguous scenes in a video image;

correspondent relationship acquisition means for placing a reference patch comprising one or a plurality of rectangular areas on one of said plurality of frames included in said contiguous frame group which is used as a reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

frame synthesis means for acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships acquired by said correspondent relationship acquisition means.

63. The video image synthesizer as set forth in claim 62, wherein

said contiguous frame group detection means is equipped with correlation computation means for computing a correlation between adjacent frames which is started from said reference frame;

and

said contiguous frame group, which is detected by said contiguous frame group detection means, comprises frames ranging from said reference frame to a frame, which is closer to said reference frame, between a pair of said adjacent frames in which said correlation is lower than a predetermined first threshold value.

64. The video image synthesizer as set forth in claim 63, wherein

said correlation computation means computes a histogram for at least one of the Y, Cb, and Cr components of each of said adjacent frames (where the Y component is a luminance component and the Cb and Cr components are color difference components), also computes a Euclidean distance for each component between said adjacent frames by employing said histogram, and computes the sum of the Euclidean distances for said three components;

and

when said sum is a predetermined second threshold value or greater, said contiguous frame group detection means judges that said correlation between said adjacent frames is lower than said predetermined first threshold value.

65. The video image synthesizer as set forth in claim 64, wherein said correlation computation means computes said histogram by dividing each of components, which are used, among said three components by a value greater than 1.

66. The video image synthesizer as set forth in claim 63, wherein

said correlation computation means computes a difference between pixel values of corresponding pixels of said adjacent frames and also computes the sum of absolute values of the differences for all corresponding pixels;

and

when said sum is a third threshold value or greater, said contiguous frame group detection means judges that said correlation between adjacent frames is lower than said predetermined first threshold value.

67. The video image synthesizer as set forth in claim 63, wherein said correlation computation means computes said correlation, employing a reduced image or thinned image of each frame.

68. The video image synthesizer as set forth in claim 62, further comprising stoppage means for stopping the detection of frames, which constitute said contiguous frame group, when the number of frames detected by said contiguous frame group detection means reaches a predetermined upper limit value.

69. A program for causing a computer to execute:

a contiguous frame group detection process of obtaining a contiguous frame group by detecting a plurality of frames that represent contiguous scenes in a video image;

a correspondent relationship acquisition process of placing a reference patch comprising one or a plurality of rectangular areas on one of said plurality of frames included in said contiguous frame group which is used as a reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

a frame synthesis process of acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.

70. A video image synthesis method comprising the steps of:

extracting a frame group that constitutes one or more important scenes from a video image;

determining a frame, which is located at approximately a center, among a plurality of frames of said frame group as a reference frame for said important scene;

placing a reference patch comprising one or a plurality of rectangular areas on said reference frame;

respectively placing patches which are the same as said reference patch, on the others of said plurality of frames;

moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch;

respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.

72. The synthesis method as set forth in claim 70, wherein

a correlation between adjacent frames of said video image is computed;

and

a set of contiguous frames where said correlation is high is extracted as said frame group that constitutes one or more important scenes.

74. The synthesis method as set forth in claim 70, wherein

a correlation between adjacent frames of said video image is computed;

a set of contiguous frames where said correlation is high is extracted as a frame group that constitutes temporary important scenes;

correlations between said temporary important scenes not adjacent are respectively computed; and

a frame group, interposed between two temporary important scenes where said correlation is high and which are closest to each other, is extracted as said frame group that constitutes one or more important scenes.

71. A video image synthesis method comprising the steps of:

extracting a frame group that constitutes one or more important scenes from a video image;

extracting high-frequency components of each of a plurality of frames constituting said frame group;

computing the sum of said high-frequency components for each of said frames;

determining a frame, in which said sum is highest, as a reference frame for said important scene;

placing a reference patch comprising one or a plurality of rectangular areas on said reference frame;

respectively placing patches which are the same as said reference patch, on the others of said plurality of frames;

moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch;

respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.

73. The synthesis method as set forth in claim 71, wherein

a correlation between adjacent frames of said video image is computed;

and

a set of contiguous frames where said correlation is high is extracted as said frame group that constitutes one or more important scenes.

75. The synthesis method as set forth in claim 71, wherein

a correlation between adjacent frames of said video image is computed;

a set of contiguous frames where said correlation is high is extracted as a frame group that constitutes temporary important scenes;

correlations between said temporary important scenes not adjacent are respectively computed; and

a frame group, interposed between two temporary important scenes where said correlation is high and which are closest to each other, is extracted as said frame group that constitutes one or more important scenes.

76. A video image synthesizer comprising:

important scene extraction means for extracting a frame group that constitutes one or more important scenes from a video image;

reference-frame determination means for determining a frame, which is located at approximately a center, among a plurality of frames of said frame group as a reference frame for said important scene;

correspondent relationship acquisition means for placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

frame synthesis means for acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.

78. The video image synthesizer as set forth in claim 76, wherein said important scene extraction means is equipped with correlation computation means for computing a correlation between adjacent frames of said video image, and extracts a set of contiguous frames, in which said correlation computed by said correlation computation means is high, as said frame group that constitutes one or more important scenes.

80. The video image synthesizer as set forth in claim 76, wherein

said important-scene extraction means comprises

first correlation computation means for computing a correlation between adjacent frames of said video image,

temporary important scene extraction means for extracting a set of contiguous frames, in which said correlation computed by said first correlation computation means is high, as a frame group that constitutes temporary important scenes, and

second correlation computation means for respectively computing correlations between said temporary important scenes not adjacent;

and

said important-scene extraction means extracts a frame group, interposed between two temporary important scenes where said correlation computed by said second correlation commutation means is high and which are closest to each other, as said frame group that constitutes one or more important scenes.

82. The video image synthesizer as set forth in claim 76, wherein said important-scene extraction means comprises the important-scene extraction means as set forth in claim 78 and the important-scene extraction means as set forth in claim 80, and which further comprises selection means for selecting either the important-scene extraction means as set forth in claim 78 or the important-scene extraction means as set forth in claim 80.

77. A video image synthesizer comprising:

important scene extraction means for extracting a frame group that constitutes one or more important scenes from a video image;

reference-frame determination means for extracting high-frequency components of each of a plurality of frames constituting said frame group, then computing the sum of said high-frequency components for each of said frames, and determining a frame, in which said sum is highest, as a reference frame for said important scene;

correspondent relationship acquisition means for placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

frame synthesis means for acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.

79. The video image synthesizer as set forth in claim 77, wherein said important scene extraction means is equipped with correlation computation means for computing a correlation between adjacent frames of said video image, and extracts a set of contiguous frames, in which said correlation computed by said correlation computation means is high, as said frame group that constitutes one or more important scenes.

81. The video image synthesizer as set forth in claim 77, wherein

said important-scene extraction means comprises

first correlation computation means for computing a correlation between adjacent frames of said video image,

temporary important scene extraction means for extracting a set of contiguous frames, in which said correlation computed by said first correlation computation means is high, as a frame group that constitutes temporary important scenes, and

second correlation computation means for respectively computing correlations between said temporary important scenes not adjacent;

and

said important-scene extraction means extracts a frame group, interposed between two temporary important scenes where said correlation computed by said second correlation commutation means is high and which are closest to each other, as said frame group that constitutes one or more important scenes.

83. The video image synthesizer as set forth in claim 77, wherein said important-scene extraction means comprises the important-scene extraction means as set forth in claim 78 and the important-scene extraction means as set forth in claim 80, and which further comprises selection means for selecting either the important-scene extraction means as set forth in claim 78 or the important-scene extraction means as set forth in claim 80.

84. A program for causing a computer to execute:

an important scene extraction process of extracting a frame group that constitutes one or more important scenes from a video image;

a reference-frame determination process of determining a frame, which is located at approximately a center, among a plurality of frames of said frame group as a reference frame for said important scene;

a correspondent relationship acquisition process of placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

a frame synthesis process of acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.

85. A program for causing a computer to execute:

an important scene extraction process of extracting a frame group that constitutes one or more important scenes from a video image;

a reference-frame determination process of extracting high-frequency components of each of a plurality of frames constituting said frame group, then computing the sum of said high-frequency components for each of said frames, and determining a frame, in which said sum is highest, as a reference frame for said important scene;

a correspondent relationship acquisition process of placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and

a frame synthesis process of acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.

86. A method of acquiring a processed frame by performing image processing on a desired frame sampled from a video image, said method comprising the steps of:

computing a similarity between said desired frame and at least one frame which is temporally before and after said desired frame; and

acquiring said processed frame by obtaining a weighting coefficient that becomes greater if said similarity becomes greater, then weighting said at least one frame with said weighting coefficient, and synthesizing said weighted frame and said desired frame.

87. The synthesis method as set forth in claim 86, wherein

said desired frame is partitioned into a plurality of areas;

said similarity is computed for each of corresponding areas in said at least one frame which correspond to said plurality of areas; and

said processed frame is acquired by obtaining weighting coefficients that become greater if said similarity becomes greater, then weighting said corresponding areas of said at least one frame with said weighting coefficients, and synthesizing said weighted areas and said plurality of areas.

88. The synthesis method as set forth in claim 86, wherein

said desired frame is partitioned into a plurality of subject areas that are included in said desired frame;

said similarity is computed for each of corresponding subject areas in said at least one frame which correspond to said plurality of subject areas; and

said processed frame is acquired by obtaining weighting coefficients that become greater if said similarity becomes greater, then weighting said corresponding subject areas of said at least one frame with said weighting coefficients, and synthesizing said weighted subject areas and said plurality of subject areas.

89. An image processor for acquiring a processed frame by performing image processing on a desired frame sampled from a video image, said image processor comprising:

similarity computation means for computing a similarity between said desired frame and at least one frame which is temporally before and after said desired frame; and

synthesis means for obtaining a weighting coefficient that becomes greater if said similarity becomes greater, then weighting said at least one frame with said weighting coefficient, and synthesizing said weighted frame and said desired frame into said processed frame.

90. The image processor as set forth in claim 89, wherein

said similarity computation means partitions said desired frame into a plurality of areas and computes said similarity for each of corresponding areas in said at least one frame which correspond to said plurality of areas;

and

said synthesis means obtains weighting coefficients that become greater if said similarity becomes greater, then weights said corresponding areas of said at least one frame with said weighting coefficients, and synthesizes said weighted areas and said plurality of areas into said processed frame.

91. The image processor as set forth in claim 89, wherein

said similarity computation means partitions said desired frame into a plurality of subject areas that are included in said desired frame, and computes said similarity for each of corresponding subject areas in said at least one frame which correspond to said plurality of subject areas;

and

said synthesis means obtains weighting coefficients that become greater if said similarity becomes greater, then weights said corresponding subject areas of said at least one frame with said weighting coefficients, and synthesizes said weighted subject areas and said plurality of subject areas into said processed frame.

92. A program for causing a computer to execute an image processing method of acquiring a processed frame by performing image processing on a desired frame sampled from a video image, said program comprising:

a similarity computation procedure of computing a similarity between said desired frame and at least one frame which is temporally before and after said desired frame; and

a synthesis process of obtaining a weighting coefficient that becomes greater if said similarity becomes greater, then weighting said at least one frame with said weighting coefficient, and synthesizing said weighted frame and said desired frame into said processed frame.

93. The program as set forth in claim 92, wherein

said similarity computation procedure comprises a procedure of partitioning said desired frame into a plurality of areas and a procedure of computing said similarity for each of corresponding areas in said at least one frame which correspond to said plurality of areas;

and

said synthesis procedure is a procedure of obtaining weighting coefficients that become greater if said similarity becomes greater, then weighting said corresponding areas of said at least one frame with said weighting coefficients, and synthesizing said weighted areas and said plurality of areas into said processed frame.

94. The program as set forth in claim 92, wherein

said similarity computation procedure comprises a procedure of partitioning said desired frame into a plurality of subject areas that are included in said desired frame, and a procedure of computing said similarity for each of corresponding subject areas in said at least one frame which correspond to said plurality of subject areas;

and

said synthesis procedure is a procedure of obtaining weighting coefficients that become greater if said similarity becomes greater, then weighting said corresponding subject areas of said at least one frame with said weighting coefficients, and synthesizing said weighted subject areas and said plurality of subject areas into said processed frame.