Video image synthesis method, video image synthesizer, image processing method, image processor, and programs for executing the synthesis method and processing method
First Claim
1. A video image synthesis method comprising the steps of:
- sampling two contiguous frames from a video image;
placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value; and
acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient.
2 Assignments
0 Petitions
Accused Products
Abstract
To acquire a high-resolution frame from a plurality of frames sampled from a video image, it is necessary to obtain a high-resolution frame with reduced picture quality degradation regardless of motion of a subject included in the frame. Because of this, between a plurality of contiguous frames FrN and FrN+1, there is estimated a correspondent relationship. Based on the correspondent relationship, the frames FrN+1 and FrN are interposed to obtain first and second interpolated frames FrH1 and FrH2. Based on the correspondent relationship, the coordinates of the frame FrN+1 are transformed, and from a correlation value with the frame FrN, there is obtained a weighting coefficient α(x°, y°) that makes the weight of the first interpolated frame FrH1 greater as a correlation becomes greater. With the weighting coefficient, the first and second interpolated frames are weighted and added to acquire a synthesized frame FrG.
-
Citations
94 Claims
-
1. A video image synthesis method comprising the steps of:
-
sampling two contiguous frames from a video image;
placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value; and
acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient. - View Dependent Claims (3, 5, 7, 9)
-
-
2. A video image synthesis method comprising the steps of:
-
sampling three or more contiguous frames from a video image;
placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;
acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;
acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;
acquiring a plurality of coordinate-transformed frames by transforming coordinates of the images within the patches of said other frames to a coordinate space of said reference frame, based on said correspondent relationships;
computing correlation values that represent a correlation between the image within the patch of each of said coordinate-transformed frames and the image within said reference patch of said reference frame;
acquiring weighting coefficients that make a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation values; and
acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames. - View Dependent Claims (4, 6, 8, 10)
-
-
11. A video image synthesizer comprising:
-
sampling means for sampling two contiguous frames from a video image;
correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
first interpolation means for acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
second interpolation means for acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
coordinate transformation means for acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
correlation-value computation means for computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
weighting-coefficient acquisition means for acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value; and
synthesis means for acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient. - View Dependent Claims (13, 15, 17, 19)
-
-
12. A video image synthesizer comprising:
-
sampling means for sampling three or more contiguous frames from a video image;
correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;
first interpolation means for acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;
second interpolation means for acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;
coordinate transformation means for acquiring a plurality of coordinate-transformed frames by transforming coordinates of the images within the patches of said other frames to a coordinate space of said reference frame, based on said correspondent relationships;
correlation-value computation means for computing correlation values that represent a correlation between the image within the patch of each of said coordinate-transformed frames and the image within said reference patch of said reference frame;
weighting-coefficient acquisition means for acquiring weighting coefficients that make a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation values; and
synthesis means for acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames. - View Dependent Claims (14, 16, 18, 20)
-
-
21. A program for causing a computer to execute a video image synthesis method comprising:
-
a procedure of sampling two contiguous frames from a video image;
a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
a procedure of acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
a procedure of acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
a procedure of acquiring a coordinate-transformed frame by transforming coordinates of the image within said second patch of said other frame to a coordinate space of said reference frame, based on said correspondent relationship;
a procedure of computing a correlation value that represents a correlation between the image within the patch of said coordinate-transformed frame and the image within said reference patch of said reference frame;
a procedure of acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation value; and
a procedure of acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient. - View Dependent Claims (23, 25, 27, 29)
-
-
22. A program for causing a computer to execute a video image synthesis method comprising:
-
a procedure of sampling three or more contiguous frames from a video image;
a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;
a procedure of acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;
a procedure of acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;
a procedure of acquiring a plurality of coordinate-transformed frames by transforming coordinates of the images within the patches of said other frames to a coordinate space of said reference frame, based on said correspondent relationships;
a procedure of computing correlation values that represent a correlation between the image within the patch of each of said coordinate-transformed frames and the image within said reference patch of said reference frame;
a procedure of acquiring weighting coefficients that make a weight of said first interpolated frame greater as said correlation becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said correlation values; and
a procedure of acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames. - View Dependent Claims (24, 26, 28, 30)
-
-
31. A video image synthesis method comprising the steps of:
-
sampling two contiguous frames from a video image;
placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of said other frame;
acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and
acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient. - View Dependent Claims (33, 35)
-
-
32. A video image synthesis method comprising the steps of:
-
sampling three or more contiguous frames from a video image;
placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;
acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;
acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;
acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of each of said other frames;
acquiring weighting coefficients that make a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and
acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames. - View Dependent Claims (34, 36)
-
-
37. A video image synthesizer comprising:
-
sampling means for sampling two contiguous frames from a video image;
correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
first interpolation means for acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
second interpolation means for acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
edge information acquisition means for acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of said other frame;
weighting-coefficient acquisition means for acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and
synthesis means for acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient. - View Dependent Claims (39, 41)
-
-
38. A video image synthesizer comprising:
-
sampling means for sampling three or more contiguous frames from a video image;
correspondent relationship estimation means for placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;
first interpolation means for acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;
second interpolation means for acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;
edge information acquisition means for acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of each of said other frames;
weighting-coefficient acquisition means for acquiring weighting coefficients that make a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and
synthesis means for acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames. - View Dependent Claims (40, 42)
-
-
43. A program for causing a computer to execute a video image synthesis method comprising:
-
a procedure of sampling two contiguous frames from a video image;
a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said two frames which is used as a reference frame, then placing on the other of said two frames a second patch which is the same as said reference patch, then moving and/or deforming said second patch in said other frame so that an image within said second patch coincides with an image within said reference patch, and estimating a correspondent relationship between a pixel within said second patch on said other frame and a pixel within said reference patch on said reference frame, based on said second patch after the movement and/or deformation and on said reference patch;
a procedure of acquiring a first interpolated frame whose resolution is higher than each of said frames, by performing interpolation either on the image within said second patch of said other frame or on the image within said second patch of said other frame and image within said reference patch of said reference frame, based on said correspondent relationship;
a procedure of acquiring a second interpolated frame whose resolution is higher than each of said frames, by performing interpolation on the image within said reference patch of said reference frame;
a procedure of acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of said other frame;
a procedure of acquiring a weighting coefficient that makes a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and
a procedure of acquiring a synthesized frame by weighting and synthesizing said first and second interpolated frames, based on said weighting coefficient. - View Dependent Claims (45, 47)
-
-
44. A program for causing a computer to execute a video image synthesis method comprising:
-
a procedure of sampling three or more contiguous frames from a video image;
a procedure of placing a reference patch comprising one or a plurality of rectangular areas on one of said three or more frames which is used as a reference frame, then respectively placing on the others of said three or more frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames coincides with an image within said reference patch, and respectively estimating correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch;
a procedure of acquiring a plurality of first interpolated frames whose resolution is higher than each of said frames, by performing interpolation either on the image within the patch of each of said other frames or on the image within the patch of each of said other frames and image within said reference patch of said reference frame, based on said correspondent relationships;
a procedure of acquiring one or a plurality of second interpolated frames whose resolution is higher than each of said frames and which are correlated with said plurality of first interpolated frames, by performing interpolation on the image within said reference patch of said reference frame;
a procedure of acquiring edge information that represents an edge intensity of the image within said reference patch of said reference frame and/or image within the patch of each of said other frames;
a procedure of acquiring weighting coefficients that make a weight of said first interpolated frame greater as said edge information becomes greater, when synthesizing said first interpolated frame and second interpolated frame, based on said edge information; and
a procedure of acquiring intermediate synthesized frames by weighting and synthesizing said first and second interpolated frames that correspond to each other on the basis of said weighting coefficients, and acquiring a synthesized frame by synthesizing said intermediate synthesized frames. - View Dependent Claims (46, 48)
-
-
49. A video image synthesis method comprising the steps of:
-
sampling a predetermined number of contiguous frames, which include a reference frame and are two or more frames, from said video image;
placing a reference patch comprising one or a plurality of rectangular areas on said reference frame;
respectively placing patches which are the same as said reference patch, on the others of said predetermined number of frames;
moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch;
respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
acquiring a synthesized frame from said predetermined number of frames, based on said correspondent relationships;
wherein said predetermined number of frames are determined based on image characteristics of said video image or synthesized frame, and said predetermined number of frames are sampled. - View Dependent Claims (50)
-
-
51. A video image synthesizer comprising:
-
sampling means for sampling a predetermined number of contiguous frames, which include a reference frame and are two or more frames, from a video image;
correspondent relationship acquisition means for placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said predetermined number of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
frame synthesis means for acquiring a synthesized frame from said predetermined number of frames, based on said correspondent relationships acquired by said correspondent relationship acquisition means;
wherein said sampling means is equipped with frame-number determination means for determining said predetermined number of frames on the basis of image characteristics of said video image or synthesized frame, and samples said predetermined number of frames determined by said frame-number determination means. - View Dependent Claims (52)
-
-
53. A program for causing a computer to execute:
-
a determination process of determining the number of frames, based on image characteristics of a video image or image characteristics of a synthesized frame which is obtained from the determined number of frames of said video image;
a sampling process of sampling the determined number of frames which are contiguous and include a reference frame, from said video image;
a correspondent relationship acquisition process of placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said predetermined number of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
a frame synthesis process of acquiring said synthesized frame from said determined number of frames, based on said correspondent relationships. - View Dependent Claims (54)
-
-
55. A video image synthesis method comprising the steps of:
-
obtaining a contiguous frame group by detecting a plurality of frames that represent contiguous scenes in a video image;
placing a reference patch comprising one or a plurality of rectangular areas on one of said plurality of frames included in said contiguous frame group which is used as a reference frame;
respectively placing patches which are the same as said reference patch, on the others of said plurality of frames;
moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch;
respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships. - View Dependent Claims (56, 57, 58, 59, 60, 61)
-
-
62. A video image synthesizer comprising:
-
contiguous frame group detection means for obtaining a contiguous frame group by detecting a plurality of frames that represent contiguous scenes in a video image;
correspondent relationship acquisition means for placing a reference patch comprising one or a plurality of rectangular areas on one of said plurality of frames included in said contiguous frame group which is used as a reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
frame synthesis means for acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships acquired by said correspondent relationship acquisition means. - View Dependent Claims (63, 64, 65, 66, 67, 68)
-
-
69. A program for causing a computer to execute:
-
a contiguous frame group detection process of obtaining a contiguous frame group by detecting a plurality of frames that represent contiguous scenes in a video image;
a correspondent relationship acquisition process of placing a reference patch comprising one or a plurality of rectangular areas on one of said plurality of frames included in said contiguous frame group which is used as a reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
a frame synthesis process of acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.
-
-
70. A video image synthesis method comprising the steps of:
-
extracting a frame group that constitutes one or more important scenes from a video image;
determining a frame, which is located at approximately a center, among a plurality of frames of said frame group as a reference frame for said important scene;
placing a reference patch comprising one or a plurality of rectangular areas on said reference frame;
respectively placing patches which are the same as said reference patch, on the others of said plurality of frames;
moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch;
respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships. - View Dependent Claims (72, 74)
-
-
71. A video image synthesis method comprising the steps of:
-
extracting a frame group that constitutes one or more important scenes from a video image;
extracting high-frequency components of each of a plurality of frames constituting said frame group;
computing the sum of said high-frequency components for each of said frames;
determining a frame, in which said sum is highest, as a reference frame for said important scene;
placing a reference patch comprising one or a plurality of rectangular areas on said reference frame;
respectively placing patches which are the same as said reference patch, on the others of said plurality of frames;
moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch;
respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships. - View Dependent Claims (73, 75)
-
-
76. A video image synthesizer comprising:
-
important scene extraction means for extracting a frame group that constitutes one or more important scenes from a video image;
reference-frame determination means for determining a frame, which is located at approximately a center, among a plurality of frames of said frame group as a reference frame for said important scene;
correspondent relationship acquisition means for placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
frame synthesis means for acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships. - View Dependent Claims (78, 80, 82)
-
-
77. A video image synthesizer comprising:
-
important scene extraction means for extracting a frame group that constitutes one or more important scenes from a video image;
reference-frame determination means for extracting high-frequency components of each of a plurality of frames constituting said frame group, then computing the sum of said high-frequency components for each of said frames, and determining a frame, in which said sum is highest, as a reference frame for said important scene;
correspondent relationship acquisition means for placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
frame synthesis means for acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships. - View Dependent Claims (79, 81, 83)
-
-
84. A program for causing a computer to execute:
-
an important scene extraction process of extracting a frame group that constitutes one or more important scenes from a video image;
a reference-frame determination process of determining a frame, which is located at approximately a center, among a plurality of frames of said frame group as a reference frame for said important scene;
a correspondent relationship acquisition process of placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
a frame synthesis process of acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.
-
-
85. A program for causing a computer to execute:
-
an important scene extraction process of extracting a frame group that constitutes one or more important scenes from a video image;
a reference-frame determination process of extracting high-frequency components of each of a plurality of frames constituting said frame group, then computing the sum of said high-frequency components for each of said frames, and determining a frame, in which said sum is highest, as a reference frame for said important scene;
a correspondent relationship acquisition process of placing a reference patch comprising one or a plurality of rectangular areas on said reference frame, then respectively placing on the others of said plurality of frames patches which are the same as said reference patch, then moving and/or deforming said patches in said other frames so that an image within the patch of each of said other frames approximately coincides with an image within said reference patch, and respectively acquiring correspondent relationships between pixels within the patches of said other frames and a pixel within said reference patch of said reference frame, based on the patches of said other frames after the movement and/or deformation and on said reference patch; and
a frame synthesis process of acquiring a synthesized frame from said plurality of frames, based on said correspondent relationships.
-
-
86. A method of acquiring a processed frame by performing image processing on a desired frame sampled from a video image, said method comprising the steps of:
-
computing a similarity between said desired frame and at least one frame which is temporally before and after said desired frame; and
acquiring said processed frame by obtaining a weighting coefficient that becomes greater if said similarity becomes greater, then weighting said at least one frame with said weighting coefficient, and synthesizing said weighted frame and said desired frame. - View Dependent Claims (87, 88)
-
-
89. An image processor for acquiring a processed frame by performing image processing on a desired frame sampled from a video image, said image processor comprising:
-
similarity computation means for computing a similarity between said desired frame and at least one frame which is temporally before and after said desired frame; and
synthesis means for obtaining a weighting coefficient that becomes greater if said similarity becomes greater, then weighting said at least one frame with said weighting coefficient, and synthesizing said weighted frame and said desired frame into said processed frame. - View Dependent Claims (90, 91)
-
-
92. A program for causing a computer to execute an image processing method of acquiring a processed frame by performing image processing on a desired frame sampled from a video image, said program comprising:
-
a similarity computation procedure of computing a similarity between said desired frame and at least one frame which is temporally before and after said desired frame; and
a synthesis process of obtaining a weighting coefficient that becomes greater if said similarity becomes greater, then weighting said at least one frame with said weighting coefficient, and synthesizing said weighted frame and said desired frame into said processed frame. - View Dependent Claims (93, 94)
-
Specification