Image transformation and synthesis methods

• US 6,327,381 B1
• Filed: 01/09/1998
• Issued: 12/04/2001
• Est. Priority Date: 12/29/1994
• Status: Expired due to Term

• Current Assignee: Worldscape, Inc.
• Original Assignee: Worldscape, LLC
• Inventors: Ehrlich, Michael, Macintosh, David, Rogina, Peter R.
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/005,357
• Time in Patent Office: 1,425 Days
• Field of Search: 382/154, 382/153, 382/152, 388/155, 388/156, 388/157, 345/419, 345/421, 345/425, 345/427
• US Class Current: 382/154
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

In a telepresence system, a scene is captured by recording pixel data elements, each associated with a pixel ray vector having a direction and an intercept on an known locus in the frame of reference of the scene. Each pixel data element includes data representing the illumination along the pixel ray vector. For example, the pixel data elements may be captured by operating numerous video cameras pointing in different directions on a spherical locus. A virtual viewpoint image representing the image which would be seen from an arbitrary viewpoint, looking in an arbitrary direction, can be synthesized by determining the directions of synthetic pixel ray vectors from each pixel of the virtual viewpoint image through the virtual viewpoint and the intercepts of these vectors on the locus. Recorded pixel data elements having pixel ray vector directions and intercepts close to those of the synthetic pixel ray vector can be copied or interpolated to provide data representing illumination in the synthetic pixel.

Claims

1. A method of synthesizing an image of a scene corresponding to the image of said scene which would be observed from a virtual viewpoint location, the method comprising the steps of:

(a) providing a plurality of starting pixel data elements, each said starting pixel data element incorporating data corresponding to illumination seen along a starting pixel ray vector associated with that starting pixel data element, each said starting pixel ray vector having a direction and an intercept on a locus;

(b) for each pixel of said synthesized image;

(1) determining the direction and intercept on said locus of a synthetic pixel ray vector from such pixel through said virtual viewpoint location;

(2) selecting one or more starting pixel data elements associated with starting pixel ray vectors having directions and intercepts close to the direction and intercept of the synthetic pixel ray vector; and

(3) deriving synthetic pixel data for the pixel of the synthesized image from the starting pixel data of the selected starting pixel data elements.

2. A method as claimed in claim 1 wherein said step of deriving said synthetic pixel data includes the step of interpolating between the starting pixel data of the selected starting pixel data elements.

3. A method as claimed in claim 1 wherein said step of providing said starting pixel data elements includes the step of providing a plurality of direction matrices, each said direction matrix including pixel data elements associated with pixel ray vectors having directions parallel within a preselected direction tolerance range to a common direction, said step of selecting one or more starting pixel data elements for each synthetic pixel including the steps of selecting one or more of said direction matrices having a common direction close to the direction of the synthetic pixel ray vector for the synthetic pixel and selecting pixel data elements from the selected direction matrices associated with starting pixel ray vectors having intercepts within a preselected intercept tolerance of the intercept of the synthetic pixel ray vector on the locus.

4. A method as claimed in claim 1 wherein said step of providing said starting pixel data elements includes the step of ordering said starting pixel data elements in a multidimensional matrix having at least one dimension correlated with direction of the starting pixel ray vectors and at least one dimension correlated with the intercepts of said starting pixel ray vectors.

5. A method as claimed in claim 4 wherein said locus is two-dimensional and said matrix has two dimensions corresponding to the locations of the intercepts in the two dimensions of said locus.

6. A method as claimed in claim 1 wherein the step of providing said starting pixel data elements includes the step of providing a plurality of discrete two dimensional images corresponding to the image of a scene observed from a plurality of discrete viewpoints lying on said locus, each said discrete image including pixel data elements associated with pixels offset from one another in horizontal and vertical directions within said discrete image, the pixel data element associated with each pixel in a discrete image representing illumination seen along a starting pixel ray vector from that pixel through the viewpoint of that discrete image.

7. A method as claimed in claim 6 wherein said locus has azimuth and elevation directions wherein said discrete viewpoints are offset from one another in azimuth and elevation directions on said locus, such differences in azimuth and elevation corresponding to representing components of differences in direction between viewpoint location vectors from an origin to said viewpoints, the horizontal and vertical directions within each discrete image being parallel to said azimuth and elevation directions along said locus.

8. A method as claimed in claim 7 wherein the azimuths of pixel ray vectors associated with different pixels of each discrete image differ from one another by equal inter-pixel azimuth differences and wherein said offsets between said viewpoints in azimuth have magnitudes which are integral multiples of the inter-pixel azimuth differences in each said discrete image.

9. A method as claimed in claim 7 wherein the elevations of pixel ray vectors associated with the pixels of each discrete image differ from one another by equal inter-pixel elevation differences and wherein said offsets between said viewpoints in elevation have magnitudes which are integral multiples of the inter-pixel elevation differences in each said discrete image.

10. A method as claimed in claim 9 wherein said step of providing said starting pixel data elements includes the step of providing a plurality of direction matrices, each said direction matrix including pixel data elements associated with pixel ray vectors extending parallel to one another in a common direction but taken from a plurality of said discrete images, said step of selecting one or more starting pixel data elements for each synthetic pixel including the steps of selecting one or more of said direction matrices having a common direction close to the direction of the synthetic pixel ray vector for the synthetic pixel and selecting pixel data elements from the selected direction matrices taken from discrete images having viewpoints close to the intercept of the synthetic pixel ray vector on the locus.

11. A method as claimed in claim 6 wherein said locus is all or a portion of a sphere, said origin is the center of the sphere, each said discrete image has a viewpoint direction perpendicular to the horizontal and vertical directions of such image and extending through the viewpoint of such image on said locus, said viewpoint directions being substantially radial with respect to said sphere or portion of a sphere.