System and method of image reconstruction

• US 9,465,436 B2
• Filed: 11/06/2013
• Issued: 10/11/2016
• Est. Priority Date: 11/09/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Raghoebardayal, Sharwin Winesh, Benson, Simon Mark, Bickerstaff, Ian Henry
• Primary Examiner(s): ZALALEE, SULTANA MARCIA
• Application Number:
  US14/073,087
• Publication Number:
  US 20140132603A1
• Time in Patent Office: 1,070 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of image reconstruction is provided. The method includes capturing a stereoscopic pair of images of a scene, generating a disparity map for the pair of images, and defining a series of value ranges corresponding to disparity with each value range in the series having an end point corresponding to a greater disparity than an end point of preceding value ranges in the series. For each value range in the series, and progressing in order through the series from the first value range in the series, the method includes selecting points in the disparity map falling within the respective value range and generating a mesh responsive to those selected points. The method also includes subsequently merging the series of generated meshes to form a 3D model of the scene.

Claims

1. A method of image reconstruction, comprising the steps of:

capturing a stereoscopic pair of images of a scene;

generating, by one or more processors, a disparity map for the pair of images;

defining a series of value ranges corresponding to disparity, each value range in the series having an end point corresponding to a greater disparity than an end point of preceding value ranges in the series;

for each value range in the series, and progressing in order through the series from the first value range in the series,

selecting points in the disparity map falling within the respective value range; and

generating, by the one or more processors, a mesh responsive to those selected points,

and subsequently merging the generated meshes to form a 3D model of the scene.

2. A method of image reconstruction according to claim 1, comprising the steps of:

detecting a minimum disparity ‘

dispMin’

in the disparity map;

detecting a maximum allowed disparity ‘

dispMax’

in the disparity map; and

defining a plurality of N value ranges between dispMin and dispMax.

3. A method of image reconstruction according to claim 2, in which an extent of each value range is equal to one selected from the list consisting of:

i. (dispMax−

dispMin) / N; and

ii. P times (dispMax−

dispMin) / N, where P is the series number of the respective value range, 1 ≦

P≦

N.

4. A method of image reconstruction according to claim 2, comprising the steps of:

detecting one or more faces in at least one of the stereoscopic pair of images;

detecting a corresponding disparity value of one or more of the faces; and

selecting N so that the disparity value of the one or more faces is in a value range other than that comprising dispMin.

5. A method of image reconstruction according to claim 1, in which the step of generating a mesh responsive to the selected points in the disparity map falling within a respective value range in turn comprises the step of:

interpolating the mesh between the selected points within the respective value range.

6. A method of image reconstruction according to claim 5, in which the step of interpolating the mesh is restricted to a predetermined value range or ranges in the series of value ranges.

7. A method of image reconstruction according to claim 5, in which the step of interpolating the mesh comprises a limit on the size of gap to be interpolated, as a function of the position of the value range in the series of value ranges.

8. A method of image reconstruction according to claim 1, in which the step of merging the generated meshes comprises one or more selected from the list consisting of:

i. joining meshes that terminate within a predetermined distance of each other;

ii. deleting a duplicate mesh polygon that is substantially collocated with a mesh polygon of a more distant layer;

iii. deleting mesh polygons that are substantially normal to an image plane; and

iv. closing meshes that have similar planar positions to more distant meshes but are separated by a predetermined depth from those more distant meshes.

9. A method of image reconstruction according to claim 1, comprising the step of:

generating one or more textures to apply to respective meshes in the 3D model of the scene,

and in which this step in turn comprises the steps of;

detecting a contiguous line of one or more undefined texture pixels for which no corresponding colour data is available in the stereoscopic pair of images due to occlusion by foreground objects;

detecting first defined pixels adjacent to each side of the line of undefined texture pixels;

detecting further defined pixels adjacent to the first pixels; and

updating the undefined texture pixels with interpolated values responsive to the further defined pixels.

10. A method of image reconstruction according to claim 9, in which the further defined pixels on each side of the line of undefined texture pixels comprise a sequence of further pixels, and successive corresponding pixels in the sequence on each side are interpolated to update the undefined texture pixels.

11. A method of image reconstruction according to claim 9, comprising the step of rendering the 3D model of the scene using one or more of the generated textures respectively applied to the or each mesh in the 3D model.

12. A non-transitory computer program product comprising computer readable instructions that when implemented by a computer cause it to perform a method comprising the steps of:

capturing a stereoscopic pair of images of a scene;

generating a disparity map for the pair of images;

defining a series of value ranges corresponding to disparity, each value range in the series having an end point corresponding to a greater disparity than an end point of preceding value ranges in the series;

for each value range in the series, and progressing in order through the series from the first value range in the series,

selecting points in the disparity map falling within the respective value range; and

generating a mesh responsive to those selected points,

and subsequently merging the generated meshes to form a 3D model of the scene.

13. An entertainment device for image reconstruction, comprising:

an input operable to receive a stereoscopic pair of captured images of a scene;

a processor operable to generate a disparity map for the pair of images and to define a series of value ranges corresponding to disparity, each value range in the series having an end point corresponding to a greater disparity than an end point of preceding value ranges in the series;

the processor being operable, for each value range in the series, and progressing in order through the series from the first value range in the series, to

select points in the disparity map falling within the respective value range; and

generate a mesh responsive to those selected points,

the processor being operable to subsequently merge the generated meshes to form a 3D model of the scene.

14. An entertainment device according to claim 13, comprising:

a second processor operable to generate one or more textures to apply to respective meshes in the 3D model of the scene, the second processor being operable to;

detect a contiguous line of one or more undefined texture pixels for which no corresponding colour data is available in the stereoscopic pair of images due to occlusion by foreground objects;

detect first defined pixels adjacent to each side of the line of undefined texture pixels;

detect further defined pixels adjacent to the first pixels; and

update the undefined texture pixels with interpolated values responsive to the further defined pixels.

15. An entertainment device according to claim 14, comprising:

rendering means for rendering the 3D model of the scene using the or each generated texture respectively applied to the or each mesh in the 3D model.