Context constrained novel view interpolation

• US 9,330,491 B2
• Filed: 05/21/2014
• Issued: 05/03/2016
• Est. Priority Date: 11/17/2009
• Status: Expired due to Fees

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Wang, Xianwang, Xiao, Jing
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Brown, Jr., Howard D
• Application Number:
  US14/283,489
• Publication Number:
  US 20140253554A1
• Time in Patent Office: 713 Days
• Field of Search: 348/43, 348/46, 382/131, 382/135
• US Class Current: 1/1
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors

Abstract

Aspects of the present invention include systems and methods for generating a novel view interpolation. In embodiments, feature correspondences and geometrical contexts are used to find additional correspondences based on the assumption of the local linear transformation. The accuracy and the number of correspondence matches may be improved by iterative refinement. Having obtained a set of correspondences, a novel view image can be generated.

Claims

1. A computer program product embodying a non-transient computer readable medium containing instructions to generate a new view of a scene by performing the steps comprising:

(a) accessing a plurality of images of the scene, each image in said plurality of images having a different view of said scene, one image in said plurality of images being a reference image and the remaining images in said plurality of images constituting a set of images;

(b) forming a separate set of corresponding features points between the reference image and at least one of the images in the set of images, the separate sets of corresponding features points together forming a set of correspondences;

(c) detecting edge pixels in the reference image;

(d) adjusting at least one selected set of corresponding features points by;

(i) for each selected edge pixel taken from the detected edge pixels in the reference image, defining a window around the selected edge pixel in the reference image and identifying a set of local correspondences, said local correspondences including corresponding feature points of the selected set of corresponding features points that are within the window, computing a transformation of the selected edge pixel and its set of local correspondences, using the transformation on the corresponding feature points of each corresponding image in said set of images as defined by the selected set of corresponding features points to identify a matching pixel in the corresponding image that matches the selected edge pixel;

(ii) checking the validity of the selected edge pixel and its matching pixel, and responsive to the validity being acceptable, adding the selected edge pixel and its matching pixel to the set of correspondences; and

(e) using the set of correspondences to generate the new view of the scene that contains image information not found in the accessed plurality of images of the scene of step (a).

2. The computer program product of claim 1, wherein step (i) includes:

for each selected edge pixel;

identifying from within the window in the reference image, a set of nearest neighbor pixels selected from the set of correspondences; and

for each corresponding image of the set of images;

applying the transformation to the selected edge pixel in the reference image to obtain a transform pixel in the corresponding image from the set of images; and

template matching the window in the reference image that includes the edge pixel with a local window in the corresponding image that includes the transform pixel to identify a corresponding pixel in the corresponding image that corresponds to the edge pixel in the reference image, wherein the corresponding pixel is deemed the matching pixel.

3. The computer program product of claim 1 wherein step (ii) includes:

accessing a three-dimensional point-representation of the scene generated by using the set of correspondences;

projecting the selected edge pixel to the three-dimensional point-representation to identify a surface point on the three-dimensional point-representation, projecting the identified surface point to at least some of the images in the set of images;

computing an error value of the projected identified surface point on said least some of the images in the set of images; and

responsive to the error value being less than a threshold value, adding the selected edge pixel and its matching pixel to the set of correspondences.

4. The computer program product of claim 3 wherein the step of computing an error value of the projected surface point comprises:

for each of said least some of the images in the set of images, computing an offset value between the projected surface point on said least some of the images in the set of images and its corresponding matching pixel in said least some of the images in the set of images; and

averaging the offset values to obtain the error value.

5. The computer program product of claim 1, wherein the view of each of said a plurality of images is from a different angle to the scene.

11. The computer program product of claim 1, wherein in step (b), a separate set of corresponding features points is formed between the reference image and each images in the set of images.

12. The computer program product of claim 1, wherein in step (d), all sets of corresponding feature points are selected for adjusting using steps (i) and (ii).

14. The computer program product of claim 1, wherein in step (c), a plurality of sets of corresponding feature points are selected for adjusting using steps (i) and (ii).

6. A hardware computing system for generating a novel view of a scene, the system comprising:

one or more communication interfaces for receiving a reference image of the scene and a set of images of the scene, said reference image and said set of images being taken at different views of the scene;

one or more data processors, wherein execution of one or more sequences of instructions by one or more processors causes the one or more processors to generate a novel view of the scene by performing the steps comprising;

(a) forming a separate set of corresponding feature points between the reference image and at least one of the images in the set of images, the separate sets of corresponding features points together forming a set of correspondences;

(b) detecting edge pixels in the reference image;

(c) adjusting at least one selected set of corresponding features points by;

(i) for each selected edge pixel taken from the detected edge pixels in the reference image, defining a window around the selected edge pixel in the reference image and identifying a set of local correspondences, said local correspondences including corresponding feature points of the selected set of corresponding features points that are within the window, computing a transformation of the selected edge pixel and its set of local correspondences, using the transformation on the corresponding feature points of each corresponding image in said set of images as defined by the selected set of corresponding features points to identify a matching pixel in the corresponding image that matches the selected edge pixel;

(ii) checking the validity of the selected edge pixel and its matching pixel, and responsive to the validity being acceptable, adding the selected edge pixel and its matching pixel to the set of correspondences; and

(d) using the set of correspondences to generate the new view of the scene that contains image information not found in the set of images of the scene.

7. The system of claim 6 wherein step (i) includes:

for each selected edge pixel;

identifying from within the window in the reference image, a set of nearest neighbor pixels selected from the set of correspondences; and

for each corresponding image of the set of images;

applying the transformation to the selected edge pixel in the reference image to obtain a transform pixel in the corresponding image from the set of images; and

template matching the window in the reference image that includes the edge pixel with a local window in the corresponding image that includes the transform pixel to identify a corresponding pixel in corresponding image that corresponds to the edge pixel in the reference image, wherein the corresponding pixel is deemed the matching pixel.

8. The system of claim 6 wherein step (ii) includes:

accessing a three-dimensional point-representation of the scene generated by using the set of correspondences;

projecting the selected edge pixel to the three-dimensional point-representation to identify a surface point on the three-dimensional point-representation, projecting the identified surface point to at least some of the images in the set of images;

computing an error value of the identified surface point on said least some of the images in the set of images; and

responsive to the error value being less than a threshold value, adding the selected edge pixel and its matching pixel to the set of correspondences.

9. The system of claim 8 wherein the step of computing an error value of the projected surface point comprises:

for each of said least some of the images in the set of images, computing an offset value between the projected surface point and its corresponding matching pixel in said least some of the images in the set of images; and

averaging the offset values to obtain the error value.

10. The system of claim 6, wherein the view of each of said a plurality of images is from a different angle to the scene.

13. The system of claim 6, wherein in step (a), a separate set of corresponding features points is formed between the reference image and a plurality of images in the set of images.