System and method of image rendering

• US 9,529,427 B2
• Filed: 11/06/2013
• Issued: 12/27/2016
• Est. Priority Date: 11/09/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Raghoebardayal, Sharwin Winesh, Benson, Simon Mark, Bickerstaff, Ian Henry
• Primary Examiner(s): ZALALEE, SULTANA MARCIA
• Application Number:
  US14/072,922
• Publication Number:
  US 20140132601A1
• Time in Patent Office: 1,147 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of rendering an image based upon a first stereoscopic image comprising a pair of images is provided. The method includes generating a virtual three-dimensional model of the scene depicted in the first stereoscopic image responsive to distances derived from the first stereoscopic image, detecting one or more free edges in the three dimensional model, and generating one or more textures for the virtual three-dimensional model from at least one of the pair of images of the first stereoscopic image. The method also includes applying at least one texture to a respective part of the three dimensional model, and rendering the virtual three dimensional model from a different viewpoint to that of the first stereoscopic image. Rendering the virtual three dimensional model comprises modifying a transparency of rendered pixels of an applied texture as a function of the pixel'"'"'s distance from that free edge.

Claims

1. A method of rendering an image based upon a first stereoscopic image comprising a pair of images, the method comprising the steps of:

generating a virtual three-dimensional model of a scene depicted in the first stereoscopic image responsive to distances derived from the first stereoscopic image;

detecting one or more free edges in the virtual three dimensional model;

generating, by one or more processors, one or more textures for the virtual three-dimensional model from at least one of the pair of images of the first stereoscopic image;

applying, by the one or more processors, at least one of the one or more textures to a respective part of the virtual three dimensional model; and

rendering, by the one or more processors, the virtual three dimensional model from a different viewpoint to that of the first stereoscopic image;

wherein rendering the virtual three dimensional model comprises modifying a transparency of rendered pixels of an applied texture as a function of each pixel'"'"'s distance from a given one of the free edges; and

wherein the rendering further includes generation of a texture for at least one polygon comprising color information from foreground and background elements of the scene;

in which generating a virtual three-dimensional model of the scene depicted in the first stereoscopic image comprises;

generating a disparity map from the pair of images of the first stereoscopic image, the disparity map being indicative of distances in the first stereoscopic image;

defining a series of value ranges corresponding to disparity values of the disparity map, each value range in the series having an end point corresponding to a greater disparity than an end point of preceding value ranges in the series;

selecting points in the disparity map falling within the respective value range;

generating a respective mesh responsive to those selected points; and

merging generated meshes to form the 3D model of the scene.

2. A method according to claim 1, in which the step of detecting one or more free edges in the virtual three dimensional model comprises detecting at least a first polygon with a side that is not shared with another polygon.

3. A method according to claim 1, in which the step of detecting one or more free edges in the virtual three dimensional model comprises detecting at least a first polygon with a vertex that is not shared with another polygon.

4. A method according to claim 1, in which the step of modifying the transparency comprises:

generating a gradient of pixel transparency values over a predetermined distance from the given one of the free edges of the virtual three dimensional model such that pixels are more transparent at the edge of the virtual three dimensional model.

5. A method according to claim 1, in which the step of modifying the transparency comprises:

generating a spline fit to the given one of the free edges of the virtual three dimensional model; and

generating a gradient of pixel transparency values over a predetermined distance from that spline, such that pixels are more transparent at the spline.

6. A method according to claim 1, comprising the step of modelling an interaction of a virtual object with the virtual three dimensional model;

and in which the step of rendering the virtual three dimensional model comprises rendering a visible effect of the interaction of the virtual object with the three dimensional model.

7. A method according to claim 1, in which the step of rendering the virtual three dimensional model from a different viewpoint comprises the steps of:

tracking a position of a user'"'"'s head with respect to a display; and

calculating one or more viewpoints for rendering responsive to a deviation of the user'"'"'s head from a default viewpoint.

8. A non-transitory computer program product comprising computer implementable instructions that when run cause a computer to implement a method of rendering an image based upon a first stereoscopic image comprising a pair of images, the method comprising the steps of:

generating a virtual three-dimensional model of a scene depicted in the first stereoscopic image responsive to distances derived from the first stereoscopic image;

detecting one or more free edges in the virtual three dimensional model;

generating one or more textures for the virtual three-dimensional model from at least one of the pair of images of the first stereoscopic image;

applying at least one of the one or more textures to a respective part of the virtual three dimensional model; and

rendering the virtual three dimensional model from a different viewpoint to that of the first stereoscopic image;

wherein rendering the virtual three dimensional model comprises modifying a transparency of rendered pixels of an applied texture as a function of each pixel'"'"'s distance from a given one of the free edges; and

wherein the rendering further includes generation of a texture for at least one polygon comprising color information from foreground and background elements of the scene;

in which generating a virtual three-dimensional model of the scene depicted in the first stereoscopic image comprises;

generating a disparity map from the pair of images of the first stereoscopic image, the disparity map being indicative of distances in the first stereoscopic image;

defining a series of value ranges corresponding to disparity values of the disparity map, each value range in the series having an end point corresponding to a greater disparity than an end point of preceding value ranges in the series;

selecting points in the disparity map falling within the respective value range;

generating a respective mesh responsive to those selected points; and

merging generated meshes to form the 3D model of the scene.

9. An entertainment device for rendering an image based upon a first stereoscopic image comprising a pair of images, the entertainment device comprising:

virtual modelling means for generating a virtual three-dimensional model of a scene depicted in the first stereoscopic image, responsive to distances derived from the first stereoscopic image;

model edge detection means for detecting one or more free edges in the virtual three dimensional model;

texture generation means for generating one or more textures for the virtual three-dimensional model from at least one of the pair of images of the first stereoscopic image;

texture application means for applying at least one of the textures to a respective part of the virtual three dimensional model; and

rendering means for rendering the virtual three dimensional model from a different viewpoint to that of the first stereoscopic image;

wherein the rendering means is operable to modify a transparency of rendered pixels of an applied texture as a function of each pixel'"'"'s distance from a given one of the free edges; and

wherein the rendering means is further operable to generate a texture for at least one polygon comprising color information from foreground and background elements of the scene

in which the virtual modelling means comprises;

disparity map generating means for generating a disparity map from the pair of images of the first stereoscopic image, the disparity map being indicative of distances in the first stereoscopic image;

range setting means for defining a series of value ranges corresponding to disparity values of the disparity map, each value range in the series having an end point corresponding to a greater disparity than an end point of preceding value ranges in the series;

selection means for selecting points in the disparity map falling within the respective value range;

mesh generating means for generating a respective mesh responsive to those selected points; and

mesh merging means for merging generated meshes to form the 3D model of the scene.

10. An entertainment device according to claim 9, in which the rendering means is operable to generate a gradient of pixel transparency values over a predetermined distance from the given one of the free edges of the virtual three dimensional model such that pixels are more transparent at the edge of the virtual three dimensional model.

11. An entertainment device according to claim 9, in which the rendering means is operable to generate a spline fit to the given one of the free edges of the virtual three dimensional model, and to generate a gradient of pixel transparency values over a predetermined distance from that spline, such that pixels are more transparent at the spline.

12. An entertainment device according to claim 9, comprising:

virtual object interaction modelling means for modeling an interaction of a virtual object with the virtual three dimensional model; and

in which

the rendering means is operable to render a visible effect of the interaction of the virtual object with the three dimensional model.

13. An entertainment device according to claim 9, comprising:

input means for head tracking information for a user'"'"'s head; and

calculating means for calculating one or more viewpoints for rendering responsive to a deviation of the user'"'"'s head from a default viewpoint; and

in which

the one or more viewpoints from which the virtual three dimensional model is rendered is based upon respective calculated viewpoints.