Context Constrained Novel View Interpolation

• US 20110115921A1
• Filed: 06/14/2010
• Published: 05/19/2011
• Est. Priority Date: 11/17/2009
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Wang, Xianwang, Xiao, Jing
• Granted Patent: US 8,817,071 B2
• Time in Patent Office: Days
• US Class Current: 348/187
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors

Abstract

Aspects of the present invention include systems and methods for generating a novel view interpolation. In embodiments, feature correspondences and geometrical contexts are used to find additional correspondences based on the assumption of the local linear transformation. The accuracy and the number of correspondence matches may be improved by iterative refinement. Having obtained a set of correspondences, a novel view image can be generated.

Claims

1. A computer program product comprising at least one computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to generate a new view of a scene by performing the steps comprising:

forming a set of correspondences between a set of features in a reference image of the scene taken from one view and each of at least some a set of images of the scene taken at different views;

detecting edge pixels in the reference image;

for each edge pixel from a set of edge pixels taken from the detected edge pixels in the reference image, using local correspondences selected from the set of correspondences to aid in the selection of a matching pixel in each of at least some of the images of the set of images, wherein the edge pixel and the matching pixels form a matched group of pixels;

checking at least some of the matched groups and responsive to a matched group of pixels being acceptable, adding the matched group to the set of correspondences; and

using the set of correspondences to generate the new view of the scene.

2. The computer program product of claim 1 wherein the step of:

for each edge pixel from a set of edge pixels taken from the detected edge pixels in the reference image, using local correspondences selected from the set of correspondences to select a matching pixel in each of at least some of the images of the set of images, wherein the edge pixel and the matching pixels form a matched group of pixels, comprises the steps of;

for each edge pixel from the set of edge pixels;

identifying in the reference image a set of nearest neighbor pixels selected from the set of correspondences; and

for each image of at least some of the images selected from the set of images;

computing a transformation between the reference image and the image from the set of images using the selected set of nearest neighbors and their corresponding pixels in the image obtained from the set of correspondences;

applying the transformation to the edge pixel in the reference image to obtain a transform pixel in the image from the set of images; and

template matching a local window of the reference image that includes the edge pixel with a local window that includes the transform pixel to identify a corresponding pixel in the image from the set of images that corresponds to the edge pixel in the reference image, wherein the corresponding pixel is deemed the matching pixel.

3. The computer program product of claim 1 wherein the step of:

checking at least some of the matched groups and responsive to a matched group of pixels being acceptable, adding the matched group to the set of correspondences, comprises the steps of;

for each of the at least some of the matched group of pixels;

computing a three-dimensional point using at least some of the pixels from the matched group of pixels;

projecting the three-dimensional point to at least some of the images selected from a group comprising the reference image and the set of images;

computing an error value of the projected three-dimensional point; and

responsive to the error value being less than a threshold value, adding the matched group to the set of correspondences.

4. The computer program product of claim 3 wherein the step of computing an error value of the projected three-dimensional point comprises:

for each of at least some of the images selected from a group comprising the reference image and the set of images, computing an offset value between the projected three-dimensional point and its corresponding pixel in the image from the matched group of pixels; and

averaging the offset values to obtain the error value.

5. The computer program product of claim 1 further comprising the step of:

calibrating a set of camera views, each camera view in the set of camera views having a unique position and generating an image for the set of images; and

rectifying the set of camera views.

6. The computer program product of claim 1 wherein the step of:

using the set of correspondences to generate the view of the scene, comprising the steps of;

obtain three-dimension points from the set of correspondences;

generate a three-dimensional triangulation mesh of the three-dimensional points;

for each pixel of the new view;

given a projection matrix for the new view of the scene, computing a three-dimensional ray for the pixel;

identifying an intersection point on the three-dimensional triangulation mesh at which the three-dimensional ray intersects the three-dimensional triangulation mesh;

for each of the at least some of the images from a group comprising the reference image and the set of images;

projecting a ray from the intersection point to the image;

identifying a pixel in the image based upon an intersection of the ray and the image; and

identifying an angle between the ray and the three-dimensional ray; and

obtaining a color value for the pixel of the new view using the angles to blend pixel colors of the identified pixels.

7. The computer program product of claim 6 wherein the step of:

obtaining a color value for the pixel of the new view using the angles to blend pixel colors of the identified pixels comprises the steps of;

identifying a maximum angle between a ray and the three-dimensional ray;

for each image of the at least some of the images, calculating a weight factor using the angle between the ray to the image and the three-dimensional ray and the maximum angle;

normalizing the weight factors; and

calculating a color value for the pixel in the new view as a sum of the identified pixels colors multiplied by their respective normalized weight factors.

8. The computer program product of claim 6 wherein the intersection point on the three-dimensional triangulation mesh at which the three-dimensional ray intersects the three-dimensional triangulation mesh is a foremost intersection point.

9. A system for generating a novel view of a scene, the system comprising:

one or more interfaces for receiving a set of images of the scene taken at different views;

a data storage for storing the set of images; and

one or more processors and at least one computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to generate a novel view of the scene by performing the steps comprising;

forming a set of correspondences between a set of features in a reference image of the scene taken from one view and each of at least some the set of images of the scene taken at different views;

detecting edge pixels in the reference image;

for each edge pixel from a set of edge pixels taken from the detected edge pixels in the reference image, using local correspondences selected from the set of correspondences to aid in the selection of a matching pixel in each of at least some of the images of the set of images, wherein the edge pixel and the matching pixels form a matched group of pixels;

checking at least some of the matched groups and responsive to a matched group of pixels being acceptable, adding the matched group to the set of correspondences; and

using the set of correspondences to generate the new view of the scene.

10. The system of claim 9 wherein the step of:

for each edge pixel from a set of edge pixels taken from the detected edge pixels in the reference image, using local correspondences selected from the set of correspondences to select a matching pixel in each of at least some of the images of the set of images, wherein the edge pixel and the matching pixels form a matched group of pixels, comprises the steps of;

for each edge pixel from the set of edge pixels;

identifying in the reference image a set of nearest neighbor pixels selected from the set of correspondences; and

for each image of at least some of the images selected from the set of images;

computing a transformation between the reference image and the image from the set of images using the selected set of nearest neighbors and their corresponding pixels in the image obtained from the set of correspondences;

applying the transformation to the edge pixel in the reference image to obtain a transform pixel in the image from the set of images; and

template matching a local window of the reference image that includes the edge pixel with a local window that includes the transform pixel to identify a corresponding pixel in the image from the set of images that corresponds to the edge pixel in the reference image, wherein the corresponding pixel is deemed the matching pixel.

11. The system of claim 9 wherein the step of:

checking at least some of the matched groups and responsive to a matched group of pixels being acceptable, adding the matched group to the set of correspondences, comprises the steps of;

for each of the at least some of the matched group of pixels;

computing a three-dimensional point using at least some of the pixels from the matched group of pixels;

projecting the three-dimensional point to at least some of the images selected from a group comprising the reference image and the set of images;

computing an error value of the projected three-dimensional point; and

responsive to the error value being less than a threshold value, adding the matched group to the set of correspondences.

12. The system of claim 11 wherein the step of computing an error value of the projected three-dimensional point comprises:

for each of at least some of the images selected from a group comprising the reference image and the set of images, computing an offset value between the projected three-dimensional point and its corresponding pixel in the image from the matched group of pixels; and

averaging the offset values to obtain the error value.

13. The system of claim 9 further comprising the step of:

calibrating a set of camera views, each camera view in the set of camera views having a unique position and generating an image for the set of images; and

rectifying the set of camera views.

14. The system of claim 9 wherein the step of:

using the set of correspondences to generate the view of the scene, comprising the steps of;

obtain three-dimension points form the set of correspondences;

generate a three-dimensional triangulation mesh of the three-dimensional points;

for each pixel of the new view;

given a projection matrix for the new view of the scene, computing a three-dimensional ray for the pixel;

identifying an intersection point on the three-dimensional triangulation mesh at which the three-dimensional ray intersects the three-dimensional triangulation mesh;

for each of the at least some of the images from a group comprising the reference image and the set of images;

projecting a ray from the intersection point to the image;

identifying a pixel in the image based upon an intersection of the ray and the image; and

identifying an angle between the ray and the three-dimensional ray; and

obtaining a color value for the pixel of the new view using the angles to blend pixel colors of the identified pixels.

15. The system of claim 14 wherein the step of:

obtaining a color value for the pixel of the new view using the angles to blend pixel colors of the identified pixels comprises the steps of;

identifying a maximum angle between a ray and the three-dimensional ray;

for each image of the at least some of the images, calculating a weight factor using the angle between the ray to the image and the three-dimensional ray and the maximum angle;

normalizing the weight factors; and

calculating a color value for the pixel in the new view as a sum of the identified pixels colors multiplied by their respective normalized weight factors.

16. The system of claim 9 wherein the interface comprises sensors for capturing the set of images.

17. A computer program product comprising at least one computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to generate a novel view of a scene, the method comprising:

obtaining a set of correspondences between a reference image of the scene taken from one view of the scene and a second image taken from another view of the scene;

using the set of correspondences and local context information to identify additional correspondences between the reference image and the second image, the additional correspondences being added to the set of correspondences;

constructing a three-dimensional mesh using three-dimensional coordinates obtained from at least some of the correspondences from the set of correspondences; and

rendering the novel view of the scene using projections from pixels of the novel view to the three-dimensional mesh to identify pixel color values from the reference image and the second image and weight factors for blending the identified pixel color values.

18. The computer program product of claim 17 wherein the step of rendering the novel view of the scene using projections from pixels of the novel view to the three-dimensional mesh to identify pixel color values from the reference image and the second image and angular blend factors for blending the identified pixel color values comprises:

for each pixel of the novel view;

given a projection matrix for the novel view of the scene, computing a three-dimensional ray for the pixel;

identifying an intersection point on the three-dimensional mesh at which the three-dimensional ray intersects the three-dimensional mesh;

for each of the reference image and the second image;

projecting a ray from the intersection point to a camera center of the image;

identifying a pixel in the image that the ray intersects; and

determining an angle between the ray and the three-dimensional ray; and

obtaining a color value for the pixel in the novel view using the angles to generate the weight factors to blend colors of the identified pixels.

19. The computer program product of claim 18 wherein the step of:

obtaining a color value for the pixel of the novel view using the angles to generate the weight factors to blend colors of the identified pixels comprises the steps of;

generating weigh factors for each of the reference image and the second image using their respective angles such that a smaller angle is given a higher weight factor than a larger angle; and

calculating a color value for the pixel in the novel view as a sum of the identified pixels colors multiplied by their respective weight factors.

20. The computer program product of claim 17 wherein the step of:

using the set of correspondences and local context information to identify additional correspondences between the reference image and the second image, the additional correspondences being added to the set of correspondences comprises the steps of;

detecting edge pixels in the reference image;

for each edge pixel from the set of edge pixels;

identifying in the reference image a set of nearest neighbor pixels selected from the set of correspondences;

computing a transformation between the reference image and the second image using the selected set of nearest neighbors and their corresponding pixels in the second image obtained from the set of correspondences;

applying the transformation to the edge pixel in the reference image to obtain a transform pixel in the second image; and

matching a local window of the reference image that includes the edge pixel with a local window in the second image that includes the transform pixel to identify a corresponding pixel in the second image that corresponds to the edge pixel in the reference image.