System and method of image augmentation

• US 9,310,885 B2
• Filed: 11/06/2013
• Issued: 04/12/2016
• Est. Priority Date: 11/09/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Raghoebardayal, Sharwin Winesh, Benson, Simon Mark, Bickerstaff, Ian Henry
• Primary Examiner(s): Hajnik, Daniel
• Assistant Examiner(s): Zalalee, Sultana M
• Application Number:
  US14/072,929
• Publication Number:
  US 20140132602A1
• Time in Patent Office: 888 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of augmenting a first stereoscopic image, comprising a pair of images, is provided. The method includes generating a disparity map from the pair of images of the first stereoscopic image. The disparity map is indicative of distances in the first stereoscopic image. The method further includes generating a virtual three-dimensional model responsive to the distances indicated by the disparity map, modeling an interaction of a virtual object with that three dimensional model, and outputting, for display, an image corresponding to the first stereoscopic image that comprises a visible effect of the interaction of the virtual object with the three dimensional model.

Claims

1. A method of augmenting a first stereoscopic image comprising a pair of images, the method comprising the steps of:

generating, by one or more processors, a disparity map from the pair of images of the first stereoscopic image, the disparity map being indicative of distances in the first stereoscopic image;

generating a virtual three-dimensional model comprising a mesh based upon the distances indicated by the disparity map;

modelling, by the one or more processors, an interaction of a virtual object with the three dimensional model, in which the virtual object has a physical attribute associated with it, and the interaction of the virtual object with the three dimensional model utilises the physical attribute; and

outputting, for display, an image corresponding to the first stereoscopic image that comprises a visible effect of the interaction of the virtual object with the three dimensional model;

in which the step of generating the virtual three-dimensional model comprises in turn the steps of;

defining a series of value ranges corresponding to disparity values of the disparity map, each value range in the series having an end point corresponding to a greater disparity than an end point of preceding value ranges in the series;

selecting points in the disparity map falling within the respective value range;

generating a respective mesh responsive to those selected points; and

merging the resulting series of generated meshes to form the 3D model of the scene;

wherein;

each respective mesh is within a depth range; and

merging the resulting series of generated meshes to form the 3D model of the scene is done across a depth spanned by the depth ranges of the respective meshes wherein the merged 3D model of the scene is made up of a continuous geometry.

2. A method according to claim 1, in which the image for display is an augmented version of at least one of the pair of images of the first stereoscopic image, and the method comprises the step of:

augmenting one or both of the pair of images of the first stereoscopic image with the virtual object at a position responsive to its interaction with the three dimensional model.

3. A method according to claim 1, comprising the steps of:

generating at least a first texture from one or both of the pair of images of the stereoscopic image;

applying the first texture to at least a respective part of the three dimensional model to obtain a textured three dimensional model; and

rendering the textured three dimensional model together with the virtual object.

4. A method according to claim 3, in which:

the virtual object has one or more physical attributes associated with it, and

the interaction of the virtual object with the three dimensional model is responsive to the or each physical attribute; and

in which

the image for display comprises the rendered textured three dimensional model with the virtual object at a position responsive to its interaction with the three dimensional model.

5. A method according to claim 3, in which the virtual object comprises a light source, and the rendered textured three dimensional model is illuminated responsive to that light source.

6. A method according to claim 5, in which the image for display is an augmented version of at least one of the pair of images of the first stereoscopic image, and the method comprises the steps of:

calculating a difference map indicating differences in rendered pixel values between rendering the textured three dimensional model with and without the light source of the virtual object; and

applying that difference map to the at least one of the pair of images of the first stereoscopic image to generate the displayed image.

7. A method according to claim 5, in which the image for display comprises the rendered textured three dimensional model illuminated responsive to the light source of the virtual object.

8. A method according to claim 3, in which the rendering of the textured three dimensional model with the virtual object is performed for one or more viewpoints other than those of the pair of images of the first stereoscopic image, so as to generate a new view of the scene depicted in the first stereoscopic image.

9. A method according to claim 8, comprising the steps of:

tracking a position of a user'"'"'s head with respect to a display; and

calculating at least one of the one or more viewpoints for rendering responsive to a deviation of the user'"'"'s head from a default viewpoint.

10. A non-transitory computer program product comprising computer readable instructions that when implemented by a computer cause it to perform a method comprising the steps of:

generating a disparity map from the pair of images of the first stereoscopic image, the disparity map being indicative of distances in the first stereoscopic image;

generating a virtual three-dimensional model comprising a mesh based upon the distances indicated by the disparity map;

modelling an interaction of a virtual object with the three dimensional model, in which the virtual object has a physical attribute associated with it, and the interaction of the virtual object with the three dimensional model utilises the physical attribute; and

outputting, for display, an image corresponding to the first stereoscopic image that comprises a visible effect of the interaction of the virtual object with the three dimensional model;

in which the step of generating the virtual three-dimensional model comprises in turn the steps of;

defining a series of value ranges corresponding to disparity values of the disparity map, each value range in the series having an end point corresponding to a greater disparity than an end point of preceding value ranges in the series;

selecting points in the disparity map falling within the respective value range;

generating a respective mesh responsive to those selected points; and

merging the resulting series of generated meshes to form the 3D model of the scene;

wherein;

each respective mesh is within a depth range; and

merging the resulting series of generated meshes to form the 3D model of the scene is done across a depth spanned by the depth ranges of the respective meshes wherein the merged 3D model of the scene is made up of a continuous geometry.

11. An entertainment device for augmenting a first stereoscopic image comprising a pair of images, the entertainment device comprising:

input means for receiving the first stereoscopic image data;

disparity processing means for generating a disparity map from the pair of images of the first stereoscopic image, the disparity map being indicative of distances in the first stereoscopic image;

virtual modelling means for generating a virtual three-dimensional model comprising a mesh based upon the distances indicated by the disparity map;

interaction modelling means for modeling an interaction of a virtual object with the three dimensional model, in which the virtual object has a physical attribute associated with it, and the interaction of the virtual object with the three dimensional model is utilises to the physical attribute; and

output means operable for outputting, for display, an image corresponding to the first stereoscopic image that comprises a visible effect of the interaction of the virtual object with the three dimensional model;

in which the virtual modelling means is configured to generate the virtual three-dimensional model by;

defining a series of value ranges corresponding to disparity values of the disparity map, each value range in the series having an end point corresponding to a greater disparity than an end point of preceding value ranges in the series;

selecting points in the disparity map falling within the respective value range;

generating a respective mesh responsive to those selected points; and

merging the resulting series of generated meshes to form the 3D model of the scene;

wherein;

each respective mesh is within a depth range; and

merging the resulting series of generated meshes to form the 3D model of the scene is done across a depth spanned by the depth ranges of the respective meshes wherein the merged 3D model of the scene is made up of a continuous geometry.

12. An entertainment device according to claim 11, comprising:

texture generation means for generating at least a first texture from one or both of the pair of images of the stereoscopic image;

texturing means for applying the first texture to at least a respective part of the three dimensional model to obtain a textured three dimensional model; and

rendering means for rendering the textured three dimensional model together with the virtual object.

13. An entertainment device according to claim 12, in which the rendering means is operable to perform rendering of the textured three dimensional model with the virtual object for one or more viewpoints other than those of the pair of images of the first stereoscopic image, so as to generate a new view of the scene depicted in the first stereoscopic image.