Context constrained novel view interpolation

• US 8,817,071 B2
• Filed: 06/14/2010
• Issued: 08/26/2014
• Est. Priority Date: 11/17/2009
• Status: Expired due to Fees

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Wang, Xianwang, Xiao, Jing
• Primary Examiner(s): Lee, Y
• Assistant Examiner(s): AMR, SALAME M
• Application Number:
  US12/814,920
• Publication Number:
  US 20110115921A1
• Time in Patent Office: 1,534 Days
• Field of Search: 345/419, 345/420, 348/42, 348/51, 348/222.1, 348/333.11, 348/43, 359/462, 382/149, 382/162, 715/716
• US Class Current: 348/43
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors

Abstract

Aspects of the present invention include systems and methods for generating a novel view interpolation. In embodiments, feature correspondences and geometrical contexts are used to find additional correspondences based on the assumption of the local linear transformation. The accuracy and the number of correspondence matches may be improved by iterative refinement. Having obtained a set of correspondences, a novel view image can be generated.

Claims

1. A computer program product comprising at least one non-transient computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to generate a new view of a scene by performing the steps comprising:

(a) accessing a plurality of images of the scene, each image in said plurality of images having a different view of said scene, one image in said plurality of images being a reference image and the remaining images in said plurality of images constituting a set of images;

(b) identifying feature points in said reference image and in each of the images in said set of images, said feature points being individual pixels characterized to be distinguishable from other pixels;

(c) identifying feature points in said reference image that match feature points in said set of images as determined by the feature points'"'"' pixel characterizations, each pair of matched feature points constituting corresponding feature points;

(d) forming a separate set of corresponding features points between the reference image and each of the images in the set of images, the separate sets of corresponding features points together forming a set of correspondences;

(e) detecting edge pixels in the reference image;

(f) adjusting at least one selected set of corresponding features points by;

(i) for each selected edge pixel taken from the detected edge pixels in the reference image, defining a window around the selected edge pixel in the reference image and identifying a set of local correspondences, said local correspondences including corresponding feature points of the selected set of corresponding features points that are within the window, computing a transformation of the selected edge pixel and its set of local correspondences, using the transformation on the corresponding feature points of each corresponding image in said set of images as defined by selected set of corresponding features points to identify a matching pixel in the corresponding image that matches the selected edge pixel;

(ii) checking the validity of the selected edge pixel and its matching pixel, and responsive to the validity being acceptable, adding the selected edge pixel and its matching pixel to the set of correspondences; and

(g) using the set of correspondences to generate the new view of the scene.

2. The computer program product of claim 1, wherein step (i) includes:

for each selected edge pixel;

identifying from within the window in the reference image, a set of nearest neighbor pixels selected from the set of correspondences; and

for each corresponding image of the set of images;

applying the transformation to the selected edge pixel in the reference image to obtain a transform pixel in the corresponding image from the set of images; and

template matching the window in the reference image that includes the edge pixel with a local window in the corresponding image that includes the transform pixel to identify a corresponding pixel in the corresponding image that corresponds to the edge pixel in the reference image, wherein the corresponding pixel is deemed the matching pixel.

3. The computer program product of claim 1 wherein step (ii) includes:

accessing a three-dimensional point-representation of the scene generated by using the set of correspondences;

projecting the selected edge pixel to the three-dimensional point-representation to identify a surface point on the three-dimensional point-representation, projecting the identified surface point to at least some of the images in the set of images;

computing an error value of the projected identified surface point on said least some of the images in the set of images; and

responsive to the error value being less than a threshold value, adding the selected edge pixel and its matching pixel to the set of correspondences.

4. The computer program product of claim 3 wherein the step of computing an error value of the projected surface point, comprises:

for each of said least some of the images in the set of images, computing an offset value between the projected surface point on said least some of the images in the set of images and its corresponding matching pixel in said least some of the images in the set of images; and

averaging the offset values to obtain the error value.

5. The computer program product of claim 1, wherein the view of each of said a plurality of images is from a different angle to the scene.

6. The computer program product of claim 1 wherein step (g) includes:

accessing a three-dimension point-representation of the scene generated from the set of correspondences;

generate a three-dimensional triangulation mesh of the three-dimensional point-representation;

for each pixel of the new view;

given a projection matrix for the new view of the scene, computing a three-dimensional ray for the pixel between the three-dimensional triangulation mesh and the new view of the scene;

identifying an intersection point on the three-dimensional triangulation mesh at which the three-dimensional ray intersects the three-dimensional triangulation mesh;

for each of the at least some of the images from a group comprising the reference image and the set of images;

projecting a respective ray from the intersection point to said at east some of the image;

identifying a pixel in each of said at least some of the images based upon the intersection of its corresponding, respective ray; and

identifying an angle between each said respective ray and the three-dimensional ray; and

obtaining a color value for the pixel of the new view using the identified angles to blend pixel colors of the identified pixels.

7. The computer program product of claim 6 wherein the step of:

obtaining a color value for the pixel of the new view using the identified angles to blend pixel colors of the identified pixels comprises the steps of;

identifying the maximum angle among said identified angles;

for each image of the at least some of the images, calculating a weight factor using its respective identified angle and the maximum angle;

normalizing the weight factors; and

calculating a color value for the pixel in the new view as a sum of the identified pixels'"'"' colors multiplied by their respective normalized weight factors.

8. The computer program product of claim 6 wherein the intersection point on the three-dimensional triangulation mesh at which the three-dimensional ray intersects the three-dimensional triangulation mesh is a front-most intersection point.

9. The computer program product of claim 1, wherein in step (b), the feature points are identified using an affine scale-invariant feature transform.

10. The computer program product of claim 6, wherein the colors of pixels having a smaller identified angle are given a greater weight in the blending of pixel colors than pixels having a larger identified angle.

11. A system for generating a novel view of a scene, the system comprising:

one or more interfaces for receiving a reference image of the scene and a set of images of the scene, said reference image and said set of images taken at different views of the scene;

a data storage for storing the set of images; and

one or more processors and at least one computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to generate a novel view of the scene by performing the steps comprising;

(b) identifying feature points in said reference image and in each of the images in said set of images, said feature points being individual pixels characterized to be distinguishable from other pixels;

(c) identifying feature points in said reference image that match feature points in said set of images as determined by the feature points'"'"' pixel characterizations, each pair of matched feature points constituting corresponding feature points;

(d) forming a separate set of corresponding feature points between the reference image and each of the images in the set of images;

(e) detecting edge pixels in the reference image;

(f) adjusting at least one selected set of corresponding features points by;

(i) for each selected edge pixel taken from the detected edge pixels in the reference image, defining a window around the selected edge pixel in the reference image and identifying a set of local correspondences, said local correspondences including corresponding feature points of the selected set of corresponding features points that are within the window, computing a transformation of the selected edge pixel and its set of local correspondences, using the transformation on the corresponding feature points of each corresponding image in said set of images as defined by the selected set of corresponding features points to identify a matching pixel, in the corresponding image that matches the selected edge pixel;

(ii) checking the validity of the selected edge pixel and its matching pixel, and responsive to the validity being acceptable, adding the selected edge pixel and its matching pixel to the set of correspondences; and

(g) using the set of correspondences to generate the new view of the scene.

12. The system of claim 11 wherein step (i) includes:

for each selected edge pixel;

identifying from within the window in the reference image, a set of nearest neighbor pixels selected from the set of correspondences; and

for each corresponding image of the set of images;

applying the transformation to the selected edge pixel in the reference image to obtain a transform pixel in the corresponding image from the set of images; and

template matching the window in the reference image that includes the edge pixel with a local window in the corresponding image that includes the transform pixel to identify a corresponding pixel in corresponding image that corresponds to the edge pixel in the reference image, wherein the corresponding pixel is deemed the matching pixel.

13. The system of claim 11 wherein step (ii) includes:

accessing a three-dimensional point-representation of the scene generated by using the set of correspondences;

projecting the selected edge pixel to the three-dimensional point-representation to identify a surface point on the three-dimensional point-representation, projecting the identified surface point to at least some of the images in the set of images;

computing an error value of the identified surface point on said least some of the images in the set of images; and

responsive to the error value being less than a threshold value, adding the selected edge pixel and its matching pixel to the set of correspondences.

14. The system of claim 13 wherein the step of computing an error value of the projected surface point comprises:

for each of said least some of the images in the set of images, computing an offset value between the projected surface point and its corresponding matching pixel in said least some of the images in the set of images; and

averaging the offset values to obtain the error value.

15. The system of claim 11, wherein the view of each of said a plurality of images is from a different angle to the scene.

16. The system of claim 11 wherein step (g) includes:

accessing a three-dimension point-representation of the scene generated form the set of correspondences;

generate a three-dimensional triangulation mesh of the three-dimensional point-representation;

for each pixel of the new view;

given a projection matrix for the new view of the scene, computing a three-dimensional ray for the pixel between the three-dimensional triangulation mesh and the new view of the scene;

identifying an intersection point on the three-dimensional triangulation mesh at which the three-dimensional ray intersects the three-dimensional triangulation mesh;

for each of the at least some of the images from a group comprising the reference image and the set of images;

projecting a respective ray from the intersection point to said at least some of the images;

identifying a pixel in each of said at least some of the images based upon the intersection of its corresponding, respective ray; and

identifying an angle between each of said respective ray and the three-dimensional ray; and

obtaining a color value for the pixel of the new view using the identified angles to blend pixel colors of the identified pixels.

17. The system of claim 16 wherein the step of:

obtaining a color value for the pixel of the new view using the identified angles to blend pixel colors of the identified pixels comprises the steps of;

identifying the maximum angle among said identified angles;

for each image of the at least some of the images, calculating a weight factor using its respective identified angle and the maximum angle;

normalizing the weight factors; and

calculating a color value for the pixel in the new view as a sum of the identified pixels'"'"' colors multiplied by their respective normalized weight factors.

18. The system of claim 11 wherein the interface comprises sensors for capturing the set of images.