METHOD FOR GENERATING A HIGH-RESOLUTION VIRTUAL-FOCAL-PLANE IMAGE

• US 20100103175A1
• Filed: 10/25/2007
• Published: 04/29/2010
• Est. Priority Date: 10/25/2006
• Status: Abandoned Application

• Current Assignee: Tokyo Institute of Technology (National University Corporation Tokyo Institute of Technology)
• Original Assignee: Tokyo Institute of Technology (National University Corporation Tokyo Institute of Technology)
• Inventors: Ikeda, Kaoru, Shimizu, Masao, Okutomi, Masatoshi
• Application Number:
  US12/443,844
• Publication Number:
  US 20100103175A1
• Time in Patent Office: Days
• US Class Current: 345/428
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors

Abstract

The present invention provides a method for generating a high-resolution virtual-focal-plane image which is capable of generating a virtual-focal-plane image with an arbitrary desired resolution simply and rapidly by using multi-view images.

The method of the present invention comprises a disparity estimating process step for estimating disparities by performing the stereo matching for multi-view images consisting of multiple images with different capturing positions and obtaining a disparity image; a region selecting process step for selecting an image among multi-view images as a basis image, setting all remaining images as reference images, and selecting a predetermined region on the basis image as a region of interest; a virtual-focal-plane estimating process step for estimating a plane in the disparity space for the region of interest based on the disparity image, and setting the estimated plane as a virtual-focal-plane; and an image integrating process step for obtaining image deformation parameters used for deforming each reference image to the basis image for the virtual-focal-plane, and generating the virtual-focal-plane image by deforming multi-view images with obtained image deformation parameters.

Claims

1. A method for generating a high-resolution virtual-focal-plane image which generates a virtual-focal-plane image by using one set of multi-view images consisting of multiple images obtained from multiple different view points, said method characterized in that

with respect to an arbitrary predetermined region, said virtual-focal-plane image is generated by performing a deformation so that each image of said multi-view images overlaps.

2. The method for generating a high-resolution virtual-focal-plane image according to claim 1, wherein disparities are obtained by performing the stereo matching for said multi-view images, and said deformation are obtained by using said obtained disparities.

3. The method for generating a high-resolution virtual-focal-plane image according to claim 2, wherein said deformation utilizes a two-dimensional homography for overlapping two images.

4. The method for generating a high-resolution virtual-focal-plane image according to claim 3, wherein said deformation is performed for said multiple images. Consisting of said multi-view images, said deformed multiple images are integrated, an integrated pixel group is sectioned with a lattice having an arbitrary size, said virtual-focal-plane image with an arbitrary resolution is generated by setting said lattice as a pixel.

5. A method for generating a high-resolution virtual-focal-plane image which generates a virtual-focal-plane image by using one set of multi-view images consisting of multiple images captured from multiple different view points for a capturing object, said method comprising.

A disparity estimating process step for estimating disparities by performing the stereo matching for said multi-view images and obtaining a disparity image;

a region selecting process step for selecting an image among said multiple images consisting of said multi-view images as a basis image, setting all remaining images except said basis image as reference images, and selecting a predetermined region on said basis image as a region of interest;

a virtual-focal-plane estimating process step for estimating a plane in the disparity space for said region of interest based on said disparity image, and setting said estimated plane as a virtual-focal-plane; and

an image integrating process step for obtaining image deformation parameters that are used for deforming said each reference image to said basis image for said virtual-focal-plane, and generating said virtual-focal-plane image by deforming said multi-view images with said obtained image deformation parameters.

6. The method for generating a high-resolution virtual-focal-plane image according to claim 5, wherein said multi-view images are obtained by a camera group that consists of multiple cameras and has a two-dimensional arrangement.

7. The method for generating a high-resolution virtual-focal-plane image according to claim 5, wherein an image capture device is fixed on a moving means, said multi-view images are images captured by moving said image capture device after assuming a camera group that consists of multiple cameras and has a two-dimensional arrangement.

8. The method for generating a high-resolution virtual-focal-plane image according to claim 5, wherein in said virtual-focal-plane estimating process step, edges in the image belonging to said region of interest of said basis image are extracted, a plane in the disparity space for said region of interest is estimated by only using disparities obtained in parts existing at said edges, and said estimated plane is set as said virtual-focal-plane.

9. The method for generating a high-resolution virtual-focal-plane image according to claim 5, wherein said image integrating process step comprises

a first step for obtaining the disparity corresponding to each vertex of said region of interest on said basis image;

a second step for obtaining coordinate positions of corresponding points of said reference image that correspond to each vertex of said region of interest on said basis image;

a third step for obtaining a homography matrix that overlaps these coordinate pairs from the correspondence relation of two vertices;

a fourth step for obtaining said homography matrix that gives the transformation for overlapping two planes by performing processes of said second step and said third step with respect to all reference images; and

a fifth step for performing the image integrating process by deforming each reference image with the obtained homography matrix, sectioning the integrated pixel group with a lattice having a predetermined size, and generating said virtual-focal-plane image with a resolution determined by said predetermined size of said lattice by setting said lattice as a pixel.

10. The method for generating a high-resolution virtual-focal-plane image according to claim 6, wherein in said virtual-focal-plane estimating process step, edges in the image belonging to said region of interest of said basis image are extracted, a plane in the disparity space for said region of interest is estimated by only using disparities obtained in parts existing at said edges, and said estimated plane is set as said virtual-focal-plane.

11. The method for generating a high-resolution virtual-focal-plane image according to claim 7, wherein in said virtual-focal-plane estimating process step, edges in the image belonging to said region of interest of said basis image are extracted, a plane in the disparity space for said region of interest is estimated by only using disparities obtained in parts existing at said edges, and said estimated plane is set as said virtual-focal-plane.

12. The method for generating a high-resolution virtual-focal-plane image according to claim 5, wherein said image integrating process step comprises

a first step for obtaining the disparity corresponding to each vertex of said region of interest on said basis image;

a second step for obtaining coordinate positions of corresponding points of said reference image that correspond to each vertex of said region of interest on said basis image;

a third step for obtaining a homography matrix that overlaps these coordinate pairs from the correspondence relation of two vertices;

a fourth step for obtaining said homography matrix that gives the transformation for overlapping two planes by performing processes of said second step and said third step with respect to all reference images; and

a fifth step for performing the image integrating process by deforming each reference image with the obtained homography matrix, sectioning the integrated pixel group with a lattice having a predetermined size, and generating said virtual-focal-plane image with a resolution determined by said predetermined size of said lattice by setting said lattice as a pixel.

13. The method for generating a high-resolution virtual-focal-plane image according to claim 6, wherein said image integrating process step comprises

a first step for obtaining the disparity corresponding to each vertex of said region of interest on said basis image;

a second step for obtaining coordinate positions of corresponding points of said reference image that correspond to each vertex of said region of interest on said basis image;

a third step for obtaining a homography matrix that overlaps these coordinate pairs from the correspondence relation of two vertices;

a fourth step for obtaining said homography matrix that gives the transformation for overlapping two planes by performing processes of said second step and said third step with respect to all reference images; and

a fifth step for performing the image integrating process by deforming each reference image with the obtained homography matrix, sectioning the integrated pixel group with a lattice having a predetermined size, and generating said virtual-focal-plane image with a resolution determined by said predetermined size of said lattice by setting said lattice as a pixel.

14. The method for generating a high-resolution virtual-focal-plane image according to claim 7, wherein said image integrating process step comprises

a first step for obtaining the disparity corresponding to each vertex of said region of interest on said basis image;

a second step for obtaining coordinate positions of corresponding points of said reference image that correspond to each vertex of said region of interest on said basis image;

a third step for obtaining a homography matrix that overlaps these coordinate pairs from the correspondence relation of two vertices;

a fourth step for obtaining said homography matrix that gives the transformation for overlapping two planes by performing processes of said second step and said third step with respect to all reference images; and

a fifth step for performing the image integrating process by deforming each reference image with the obtained homography matrix, sectioning the integrated pixel group with a lattice having a predetermined size, and generating said virtual-focal-plane image with a resolution determined by said predetermined size of said lattice by setting said lattice as a pixel.

15. The method for generating a high-resolution virtual-focal-plane image according to claim 8, wherein said image integrating process step comprises

a first step for obtaining the disparity corresponding to each vertex of said region of interest on said basis image;

a second step for obtaining coordinate positions of corresponding points of said reference image that correspond to each vertex of said region of interest on said basis image;

a third step for obtaining a homography matrix that overlaps these coordinate pairs from the correspondence relation of two vertices;

a fourth step for obtaining said homography matrix that gives the transformation for overlapping two planes by performing processes of said second step and said third step with respect to all reference images; and

a fifth step for performing the image integrating process by deforming each reference image with the obtained homography matrix, sectioning the integrated pixel group with a lattice having a predetermined size, and generating said virtual-focal-plane image with a resolution determined by said predetermined size of said lattice by setting said lattice as a pixel.