Glancing angle exclusion

• US 9,251,623 B2
• Filed: 08/30/2012
• Issued: 02/02/2016
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kirk, Adam G.
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US13/599,436
• Publication Number:
  US 20130321590A1
• Time in Patent Office: 1,251 Days
• Field of Search: 345/419, 382/154, 348/48
• US Class Current: 1/1
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04R 2227/005 Audio distribution systems for home, i.e. multi-room use
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction

Abstract

The glancing angle exclusion technique described herein selectively limits projective texturing near depth map discontinuities. A depth discontinuity is defined by a jump between a near-depth surface and a far-depth surface. The claimed technique can limit projective texturing on near and far surfaces to a different degree—for example, the technique can limit far-depth projective texturing within a certain distance to a depth discontinuity but not near-depth projective texturing.

Claims

1. A computer-implemented process for creating a synthetic video from images captured from an array of cameras, comprising the process actions of:

(a) capturing images of a scene using the array of cameras arranged in three dimensional (3D) space relative to the scene;

(b) estimating camera data and 3D geometric information that describes objects in the captured scene both spatially and temporally;

(c) generating a set of geometric proxies which describe objects in the scene as a function of time using the extracted camera and 3D geometric data

(d) determining silhouette boundaries of the geometric proxies in the captured images of the scene;

(e) applying projective texture from the captured images to the geometric proxies while masking the projective texture which exceeds the boundaries of the silhouette by using depth map discontinuities comprising;

for each pixel of the projective texture within a first number of pixels from a depth map discontinuity, determining if the pixel is on the near side or the far side of the discontinuity,

if the pixel is on the near side, not using the pixel for rendering if the pixel is within a second number of pixels from the discontinuity, and

if the pixel is on the far side, not using the pixel for rendering if the pixel is within a third number of pixels from the discontinuity.

2. The computer-implemented process of claim 1 wherein the silhouette boundaries are determined by using an edge detector on a depth map associated with each captured image of the object.

3. The computer-implemented process of claim 1 wherein the projective texture is masked by a blend mask.

4. The computer-implemented process of claim 3 wherein any large depth pixels that are within a variable number of pixels from an edge shared with a small depth pixel are turned off in the blend mask.

5. The computer-implemented process of claim 3 wherein the blend mask is checked before an object is rendered.

6. The computer-implemented process of claim 1, wherein the projective texture is masked using a floating point map.

7. The computer-implemented process of claim 6, wherein the floating point map is a product of a blend mask times the projection of a normal onto a given view direction used in computing a projective texturing weight.

8. The computer-implemented process of claim 1, wherein a weighted joint bilateral filter is used to reduce the number and size of holes in the depth map.

9. The computer-implemented process of claim 1 wherein the geometric proxies are generated using 3D rendering algorithms.

10. A computer-implemented process for generating a 3D spatial video, comprising:

capturing images of a scene using an array of sensors arranged in three dimensional (3D) space relative to the scene, wherein the sensors capture intensity data and depth data of the scene;

synthesizing a three dimensional video frame of the scene comprising;

creating a geometric proxy of at least one object in the scene using estimated sensor geometry and the intensity data and depth data;

applying projective texturing to the geometric proxy of the at least one object by using one or more depth map discontinuities comprising examining pixels of texture data within a first prescribed number of pixels from a depth map discontinuity, and not using pixels within a second prescribed number of pixels from a near side of the discontinuity and a third prescribed number of pixels from a far side of the discontinuity for applying the projective texturing.

11. The computer-implemented process of claim 10 wherein a depth map discontinuity further comprises a jump between a near-depth surface and a far-depth surface.

12. The computer-implemented process of claim 10 wherein projective texturing can be applied to near depth surfaces and far depth surfaces to a different degree.

13. The computer-implemented process of claim 12 wherein projective texturing of a far depth surface can be not rendered within a certain distance to a depth discontinuity.

14. The computer-implemented process of claim 10, wherein a weighted joint bilateral filter is used to reduce the size and number of holes in the depth map.

15. A system for generating a 3D spatial video, comprising:

a computing device;

a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,

(a) input captured depth images and corresponding RGB images of a scene that were captured using an array of cameras arranged in three dimensional (3D) space relative to the scene;

(b) generating a set of geometric proxies which describe objects in the scene;

(c) projecting the geometric proxies onto a depth map corresponding to each RGB image;

(d) running an edge filter on the depth map to find edges of the geometric proxies in the depth map;

(e) computing a projective texture mask to use when applying a projective texture to the geometric proxies by locating depth map discontinuities and determining a distance in pixels from the discontinuities as the edges of the mask, wherein in the mask any large depth pixels of the projective texture that are within a variable number of pixels from an edge shared with a small depth pixel are not used when applying the projective textures to the geometric proxies;

(f) applying projective texture to the geometric proxies while avoiding applying projective texture to the boundaries of the geometric proxies by using the projective texture mask.

16. The system of claim 15 wherein the edge filter is a high pass filter.

17. The system of claim 15 wherein the projective texture mask is determined by:

setting the pixels in the mask to 1 if the pixel correspond to a valid depth estimate;

examining each pixel to see if it is within alpha pixels of a depth discontinuity;

if the pixel is within alpha pixels a depth discontinuity, determining if the pixel is on the near side or the far side of the depth discontinuity, where alpha is a prescribed value;

if the pixel is on the near side, not using the pixel for rendering if the pixel is beta or more pixels from the discontinuity, where beta is a prescribed value;

if the pixel is on the far side, not using the pixel for rendering if the pixel is gamma or more pixels from the discontinuity, where gamma is a prescribed value.

18. The system of claim 17 wherein alpha=5, beta is infinity and gamma=5.

19. The system of claim 17 wherein alpha, beta and gamma are user-defined variables.

20. The system of claim 16 wherein the projective texture mask is a blend mask and wherein any large depth pixels that are within a variable number of pixels from an edge shared with a small depth pixel are turned off in the blend mask.