Systems and methods for 3D capturing of objects and motion sequences using multiple range and RGB cameras

• US 10,154,246 B2
• Filed: 11/19/2015
• Issued: 12/11/2018
• Est. Priority Date: 11/20/2014
• Status: Active Grant

• Current Assignee: Cappasity Inc.
• Original Assignee: Cappasity Inc.
• Inventors: Popov, Konstantin
• Primary Examiner(s): Daniels, Anthony J
• Application Number:
  US14/946,133
• Publication Number:
  US 20160150217A1
• Time in Patent Office: 1,118 Days
• Field of Search: 34820799, 348E13002, 348E13006, 348E13015
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

A system and method for electronic modeling of physical objects in a three-dimensional (3D) representation is described. The system generally comprises a plurality of range and RGB cameras which are controlled to provide images of different portions of a 3D person or object. The method operates by taking the separate images produced by the plurality of range and RGB cameras, and combining them into a 3D representation.

Claims

1. A computer system comprising:

at least one server computer; and

,

at least one client computer coupled to the at least one server computer through a network;

a plurality of range cameras, each coupled to the at least one client computer;

a plurality of RGB cameras each coupled to the at least one client computer;

wherein the at least one server computer includes at least one program stored thereon, said at least one program being capable of performing the following steps;

intrinsically calibrating each of the plurality of range cameras with respect to a calibration object prior to capturing a first set of 2D images;

extrinsically calibrating each of the plurality of range cameras prior to capturing the first set of 2D images by determining the relative position of a first one of the plurality of range cameras with respect to a second one of the plurality of range cameras;

extrinsically calibrating each of the plurality of RGB cameras by determining the relative position of a first one of the plurality of range cameras with respect to a first one of the plurality of RGB cameras;

determining the coordinates of a three-dimensional (3D) object through the extrinsic calibrations of the range and RGB cameras;

capturing a first set of two-dimensional (2D) images of the 3D object using the plurality of range cameras;

generating a point cloud from the first set of 2D images; and

,

generating a 3D reconstruction of the 3D object using the first set of 2D images.

2. The computer system of claim 1, wherein said at least one program is capable of performing the further steps of:

capturing a second set of 2D images of the 3D object using the plurality of RGB cameras; and

performing texture mapping of the 3D reconstruction using the second set of 2D images.

3. The computer system of claim 1, wherein said at least one program is capable of performing the further steps of:

calibrating each of the plurality of RGB cameras with respect to a calibration object prior to capturing the second set of 2D images.

4. The computer system of claim 1, wherein said step of capturing a first set of 2D images using the plurality of range cameras comprises:

capturing a first 2D image from a first range camera of the plurality of range cameras which is positioned at a first position with respect to the scene or object;

capturing a second 2D image from a second range camera of the plurality of range cameras which is positioned at a second position with respect to the scene or object.

5. The computer system of claim 1, wherein said step of capturing a first set of 2D images using the plurality of range cameras comprises:

capturing a first 2D image from a first range camera of the plurality of range cameras which is positioned at a first position in front of the scene or object;

capturing a second 2D image from a second range camera of the plurality of range cameras which is positioned at a second position in front of the scene or object;

capturing a third 2D image from a third range camera of the plurality of range cameras which is positioned at a third position behind the scene or object; and

capturing a fourth 2D image from a fourth range camera of the plurality of range cameras which is positioned at a fourth position behind the scene or object,

wherein each of the first, second, third and fourth range cameras capture a different portion of the scene or object.

6. The computer system of claim 5, wherein said step of generating a point cloud from the first set of 2D images comprises:

combining the first, second, third and fourth images captured by the first, second, third and fourth range cameras of different portions of the scene or object to create a single point cloud of the entire scene or object.

7. The computer system of claim 2, wherein said step of capturing a second set of 2D images using the plurality of RGB cameras comprises:

capturing a first 2D image from a first RGB camera of the plurality of RGB cameras which is positioned at a first position with respect to the scene or object;

capturing a second 2D image from a second RGB camera of the plurality of RGB cameras which is positioned at a second position with respect to the scene or object.

8. The computer system of claim 2, wherein said step of capturing a second set of 2D images using the plurality of RGB cameras comprises:

capturing a first 2D image from a first RGB camera of the plurality of RGB cameras which is positioned at a first position in front of the scene or object;

capturing a second 2D image from a second RGB camera of the plurality of RGB cameras which is positioned at a second position in front of the scene or object;

capturing a third 2D image from a third RGB camera of the plurality of RGB cameras which is positioned at a third position behind the scene or object; and

capturing a fourth 2D image from a fourth RGB camera of the plurality of RGB cameras which is positioned at a fourth position behind the scene or object,

wherein each of the first, second, third and fourth RGB cameras capture a different portion of the scene or object.

9. The computer system of claim 5, wherein said step of capturing a first set of 2D images using the plurality of range cameras further comprises:

capturing a fifth 2D image from a fifth range camera of the plurality of range cameras which is positioned at a fifth position above the scene or object.

10. A non-transitory computer readable medium having embodied therein a computer program for processing by a machine, the computer program comprising:

a first code segment for intrinsically calibrating each of the plurality of range cameras with respect to a calibration object, for extrinsically calibrating each of the plurality of range cameras by determining the relative position of a first one of the plurality of range cameras with respect to a second one of the plurality of range cameras, and for extrinsically calibrating each of a plurality of RGB cameras by determining the relative position of a first one of the plurality of range cameras with respect to a first one of the plurality of RGB cameras, prior to capturing a first set of two-dimensional (2D) images of a three-dimensional (3D) object using the plurality of range cameras;

a second code segment for generating a point cloud from the first set of 2D images; and

,

a third code segment for generating a 3D reconstruction of the 3D object using the first set of 2D images.

11. The non-transitory computer readable medium of claim 10, wherein the computer program further comprises:

a fourth code segment for capturing a second set of 2D images of the 3D object using the plurality of RGB cameras;

a fifth code segment for performing texture mapping of the 3D reconstruction using the second set of 2D images.

12. The non-transitory computer readable medium of claim 11, wherein the computer program further comprises:

a sixth code segment for calibrating each of the plurality of RGB cameras with respect to a calibration object prior to capturing the second set of 2D images.

13. A computer-implemented method for three-dimensional (3D) reconstruction of an object or scene, the method comprising the steps of:

(a) calibrating each of a plurality of range cameras with respect to a calibration object prior to capturing a first set of 2D images, said calibrating step involving an intrinsic calibration of each of the plurality of range cameras with respect to a calibration object, an extrinsic calibration of each of the plurality of range cameras by determining the relative position of a first one of the plurality of range cameras with respect to a second one of the plurality of range cameras, and an extrinsic calibration of each of the plurality of RGB cameras by determining the relative position of a first one of the plurality of range cameras with respect to a first one of the plurality of RGB cameras;

(b) imaging the object or scene with the plurality of range cameras and producing at least one first image set;

(c) generating a point cloud using the at least one first image set;

(d) generating a 3D reconstruction of the object or scene using the point cloud; and

,

(e) displaying said 3D reconstruction of the object or scene.

14. The computer-implemented method of claim 13, comprising the further steps of:

(f) capturing a second set of 2D images of the 3D object using the plurality of RGB cameras; and

(g) performing texture mapping of the 3D reconstruction using the second set of 2D images.

15. The computer-implemented method of claim 13, comprising the further steps of:

calibrating each of the plurality of RGB cameras with respect to a calibration object prior to capturing the second set of 2D images.

16. A system for three-dimensional (3D) reconstruction of an object or scene, the system comprising:

a plurality of range cameras for imaging the object or scene;

a plurality of RGB cameras for imaging the object or scene; and

,

a controller comprising a first program module for generating at least one first image set from the plurality of range cameras, a second program module for generating at least one second image set from the plurality of RGB cameras, a third program module for generating a point cloud from the at least one first image set, a fourth program module for generating a 3D mesh reconstruction of the scene or object from the point cloud, and a fifth program module for performing texture mapping of the mesh reconstruction,

wherein the plurality of range cameras and plurality of RGB cameras are disposed at different positions with respect to the object or scene, such that each range or RGB camera captures a different perspective of the object or scene and wherein such different perspectives are combined when generating the point cloud so as to completely reconstruct the 3D scene or object,

wherein the plurality of range cameras and the plurality of RGB cameras are calibrated prior to generating the at least one first and at least one second image sets, said calibration comprising an intrinsic calibration of each of the plurality of range cameras with respect to a calibration object, an extrinsic calibration of each of the plurality of range cameras by determining the relative position of a first one of the plurality of range cameras with respect to a second one of the plurality of range cameras, and an extrinsic calibration of each of the plurality of RGB cameras by determining the relative position of a first one of the plurality of range cameras with respect to a first one of the plurality of RGB cameras.