Holocam Systems and Methods

• US 20150261184A1
• Filed: 03/13/2014
• Published: 09/17/2015
• Est. Priority Date: 03/13/2014
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Moravec, Ivo, Abadpour, Arash, Mannion, Michael, Sukumaran, Sujay, Huda, Syed Alimul, Matei, Bogdan, Kezele, Irina
• Granted Patent: US 9,438,891 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

Aspects of the present invention comprise holocam systems and methods that enable the capture and streaming of scenes. In embodiments, multiple image capture devices, which may be referred to as “orbs,” are used to capture images of a scene from different vantage points or frames of reference. In embodiments, each orb captures three-dimensional (3D) information, which is preferably in the form of a depth map and visible images (such as stereo image pairs and regular images). Aspects of the present invention also include mechanisms by which data captured by two or more orbs may be combined to create one composite 3D model of the scene. A viewer may then, in embodiments, use the 3D model to generate a view from a different frame of reference than was originally created by any single orb.

Claims

1. A holocam system comprising:

an acquisition module that receives visible image data and range data of a scene from two or more vantage points via one or more range data acquisition devices and one or more visible image acquisition devices;

a model creation module that produces, for a vantage point, a three-dimensional (3D) model by combining range data and visible image data from that vantage point; and

a scene registration module, communicatively coupled to the model creation module, that performs scene registration to acquire geometric transformations for merging two or more 3D models of the scene from different vantage points into a source-independent canonical space.

2. The holocam system of claim 1 further comprising:

a runtime processor, communicatively coupled to the acquisition module, that filters the range data according to one or more filtering operations;

an audio module, communicatively coupled to the model creation module, that receive audio data about the scene from one or more audio input devices; and

a transmission module, communicatively coupled to the model creation module, that encodes the scene model for transmission over a network.

3. The holocam system of claim 2 further comprising:

a rendering module that receives an encoded scene model and renders a view of the scene using the encoded scene model for display using one or more display devices.

4. The holocam system of claim 1 wherein the scene registration module is configured to perform the steps comprising:

responsive to an input device having part of its sensors affected by one or more conditions in its field of view, generating an image mask for the input device to mask the affected part of its sensors;

positioning a point acquisition device within the field of view of at least two input devices;

using one or more signals from the point acquisition device to generate points which are used to find correspondences of points between data captured by at least two input devices;

using the correspondences of points to identify three-dimensional points in each of a plurality of image data of a scene; and

using at least some of the identified three-dimensional points to perform a bundle adjustment to obtain a scene registration matrix.

5. The holocam system of claim 1 wherein the scene registration module is configured to perform the steps comprising:

performing point cloud generation from data about the scene taken from a vantage point;

detecting planar surfaces in the point cloud;

identifying a floor plane from among the detected planar surfaces;

estimating a canonical transformation that transforms the point cloud so that points corresponding to the floor plane are mapped to a z=0 plane and vertical direction is correct;

performing voxel segmentation by projecting contents in the scene to the floor plane;

identifying intersections of the floor plane with potential wall planes selected from the detected planar surfaces;

using at least some of the intersections as guideline intersections to identify one or more in-plane rotation estimates between point clouds;

obtaining an in-plane translation estimate using the voxel map for reference; and

for a point cloud, combining the canonical transformation, in-plane rotation, and in-plane translation to form a scene registration matrix.

6. The holocam system of claim 1 wherein the scene registration model further configured to perform the steps comprising:

performing scene reregistration to improve one or more geometric transformations.

7. The holocam system of claim 6 wherein the scene registration module is configured to perform scene reregistration by causing the following steps to be performed:

[a] selecting as a basis a point cloud generated from data of the scene taken from a vantage point;

[b] reprojecting point clouds generated from data of the scene taken from different vantage points onto the basis and treating the aggregation of the resulting combined points clouds as one virtual point cloud;

[c] solving for rigid body movement and producing rotation (R*) and translation (T*) values;

[d] using the produced rotation (R*) and translation (T*) values to update scene registration information;

[e] responsive to a stop condition being reached, selecting a best set of scene registration information based on one or more evaluation metrics; and

[f] responsive to a stop condition not being reached, iterating by returning to step [a].

8. The holocam system of claim 7 wherein the step of reprojecting comprises:

producing a first depth buffer filled with an identifier representing an unknown value;

for points in a point cloud obtained from one of the input devices, calculating projected pixels values using an initial transform and camera intrinsic values;

comparing the values of the projected pixels with values of corresponding pixels in the first depth buffer and setting the depth value in the first depth buffer to a projected pixel'"'"'s depth value responsive to the value in the first depth buffer being (1) the unknown value or (2) larger than the corresponding projected pixel'"'"'s depth value;

producing a second properly-sized depth buffer and filling it with an identifier representing an unknown value;

looping through pixel values in the first depth buffer to;

set a pixel'"'"'s depth value in the second depth buffer to the value of the corresponding pixel in the first depth buffer if the value in the first depth buffer is not an unknown value; and

otherwise setting the pixel'"'"'s depth value to an average value of a neighborhood of values in the first depth buffer that are not unknown values, if any exist;

generating real world X and Y values using values from the second depth buffer; and

combining X, Y, and Z values to produce projection points.

9. A computer-implemented method comprising:

receiving color and depth of a scene from a plurality of vantage points from one or more input devices;

producing, for each vantage point, a three-dimensional (3D) model of data by combining depth data information and visible image data;

defining a source-independent canonical space which is used to register two or more 3D models into that space; and

performing scene registration to acquire geometrical transformations to merge two or more 3D models of the same scene from their corresponding different vantage points into a combined 3D model.

10. The computer-implemented method of claim 9 wherein the scene registration comprises the steps of:

performing point cloud generation from data about the scene taken from a vantage point;

detecting planar surfaces in the point cloud;

identifying a floor plane from among the detected planar surfaces;

estimating a canonical transformation that transforms the point cloud so that points corresponding to the floor plane are mapped to a z=0 plane and vertical direction is correct;

performing voxel segmentation by projecting contents in the scene to the floor plane;

identifying intersections of the floor plane with potential wall planes selected from the detected planar surfaces;

using at least some of the intersections as guideline intersections to identify one or more in-plane rotation estimates between point clouds;

obtaining an in-plane translation estimate using the voxel map for reference; and

for a point cloud, combining the canonical transformation, in-plane rotation, and in-plane translation to form a scene registration matrix.

11. The computer-implemented method of claim 9 wherein the scene registration is user-assisted and comprises the steps of:

responsive to an input device having part of its sensors affected by one or more conditions in its field of view, generating an image mask for the input device to mask the affected part of its sensors;

positioning a point acquisition device within the field of view of at least two input devices;

using one or more signals from the point acquisition device to generate points which are used to find correspondences of points between data captured by at least two input devices;

using the correspondences of points to identify three-dimensional points in each of a plurality of image data of a scene; and

using at least some of the identified three-dimensional points to perform a bundle adjustment to obtain a scene registration matrix.

12. The computer-implemented method of claim 9 wherein scene registration further comprises the step of:

performing scene reregistration to improve one or more geometric transformations.

13. The computer-implemented method of claim 12 wherein scene reregistration comprises the step of:

[a] selecting as a basis a point cloud generated from data of the scene taken from a vantage point;

[b] reprojecting point clouds generated from data of the scene taken from different vantage points onto the basis and treating the aggregation of the resulting combined points clouds as one virtual point cloud;

[c] solving for rigid body movement and producing rotation (R*) and translation (T*) values;

[d] using the produced rotation (R*) and translation (T*) values to update scene registration information;

[e] responsive to a stop condition being reached, selecting a best set of scene registration information based on one or more evaluation metrics; and

[f] responsive to a stop condition not being reached, iterating by returning to step [a].

14. The computer-implemented method of claim 13 wherein the step of reprojecting comprises:

producing a first depth buffer filled with an identifier representing an unknown value;

for points in a point cloud obtained from one of the input devices, calculating projected pixels values using an initial transform and camera intrinsic values;

comparing the values of the projected pixels with values of corresponding pixels in the first depth buffer and setting the depth value in the first depth buffer to a projected pixel'"'"'s depth value responsive to the value in the first depth buffer being (1) the unknown value or (2) larger than the corresponding projected pixel'"'"'s depth value;

producing a second properly-sized depth buffer and filling it with an identifier representing an unknown value;

looping through pixel values in the first depth buffer to;

set a pixel'"'"'s depth value in the second depth buffer to the value of the corresponding pixel in the first depth buffer if the value in the first depth buffer is not an unknown value; and

otherwise setting the pixel'"'"'s depth value to an average value of a neighborhood of values in the first depth buffer that are not unknown values, if any exist;

generating real world X and Y values using values from the second depth buffer; and

combining X, Y, and Z values to produce projection points.

15. The computer-implemented method of claim 13 wherein the step of solving for rigid body movement and producing rotation (R*) and translation (T*) values comprises the steps of:

given two point clouds, P₁ and P₂, and scene registration information comprising rotation and translation values, projecting P₁ and P₂ to produce P₁*;

noting corresponding pair of points between P₁* and P₂ which are less than a threshold distance apart;

utilizing a rigid body movement transformation to produce an estimate of the rotation R* and translation T* values; and

update the rotation and translation values.

16. A system comprising one or more processors;

and

a non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by the one or more processors, causes steps to be performed comprising;

receiving color and depth of a scene from a plurality of vantage points from one or more input devices;

producing, for each vantage point, a three-dimensional (3D) model of data by combining depth data information and visible image data;

defining a source-independent canonical space which is used to register two or more 3D models into that space; and

performing scene registration to acquire geometrical transformations to merge two or more 3D models of the same scene from their corresponding different vantage points into a combined 3D model.

17. The system of claim 16 wherein scene registration comprises the steps of:

performing point cloud generation from data about the scene taken from a vantage point;

detecting planar surfaces in the point cloud;

identifying a floor plane from among the detected planar surfaces;

estimating a canonical transformation that transforms the point cloud so that points corresponding to the floor plane are mapped to a z=0 plane and vertical direction is correct;

performing voxel segmentation by projecting contents in the scene to the floor plane;

identifying intersections of the floor plane with potential wall planes selected from the detected planar surfaces;

using at least some of the intersections as guideline intersections to identify one or more in-plane rotation estimates between point clouds;

obtaining an in-plane translation estimate using the voxel map for reference; and

for a point cloud, combining the canonical transformation, in-plane rotation, and in-plane translation to form a scene registration matrix.

18. The system of claim 16 wherein the scene registration further comprises the step of:

performing scene reregistration to improve one or more geometric transformations.

19. The system of claim 18 wherein scene reregistration comprises the step of:

[a] selecting as a basis a point cloud generated from data of the scene taken from a vantage point;

[b] reprojecting point clouds generated from data of the scene taken from different vantage points onto the basis and treating the aggregation of the resulting combined points clouds as one virtual point cloud;

[c] solving for rigid body movement and producing rotation (R*) and translation (T*) values;

[d] using the produced rotation (R*) and translation (T*) values to update scene registration information;

[e] responsive to a stop condition being reached, selecting a best set of scene registration information based on one or more evaluation metrics; and

[f] responsive to a stop condition not being reached, iterating by returning to step [a].

20. The system of claim 19 wherein the step of reprojecting comprises:

producing a first depth buffer filled with an identifier representing an unknown value;

for points in a point cloud obtained from one of the input devices, calculating projected pixels values using an initial transform and camera intrinsic values;

comparing the values of the projected pixels with values of corresponding pixels in the first depth buffer and setting the depth value in the first depth buffer to a projected pixel'"'"'s depth value responsive to the value in the first depth buffer being (1) the unknown value or (2) larger than the corresponding projected pixel'"'"'s depth value;

producing a second properly-sized depth buffer and filling it with an identifier representing an unknown value;

looping through pixel values in the first depth buffer to;

set a pixel'"'"'s depth value in the second depth buffer to the value of the corresponding pixel in the first depth buffer if the value in the first depth buffer is not an unknown value; and

otherwise setting the pixel'"'"'s depth value to an average value of a neighborhood of values in the first depth buffer that are not unknown values, if any exist;

generating real world X and Y values using values from the second depth buffer; and

combining X, Y, and Z values to produce projection points.